Find noisy annotators and retrain a classifier without them

A classifier is trained on labels from several human annotators, and some of them label carelessly. Find which annotators are unreliable, drop their labels, retrain, and show whether the classifier actually improved. Explain how you would tell a careless annotator from one who simply got harder examples.