Hand Labeling Considered Harmful

One activity that is a constant of almost all research into education and ed tech is labeling. Labels "are used to identify characteristics operationally defined by researchers and are primarily used for classification and categorization of individuals in survey efforts." And while this article is more focused on the use of labels in AI, the applications are apparent in most any research. And here's the problem, according to this article: if you're labeling by hand, you are introducing bias and error into your research. Interestingly, the article recommends an educational game Survival of the Best Fit "where you’re a CEO who uses a machine learning model to scale their hiring decisions and see how the model replicates the bias inherent in the training data." I played it; it made the case quite well. Now there are many more factors to consider, but it's worth asking, even, what bias the limitation of research subjects to 'students' adds to your study. More (13 page PDF).