Is natural language processing the cheap charlie of analyzing cheap talk? A horse race between classifiers on experimental communication data

We conduct a horse race between various classification algorithms in order to assess whether some algorithms might be more appropriate for classifying experimental communication data than others. We use data reported by various experimental studies involving digital written communication. The effectiveness criterion for comparing the algorithms is based on the agreement between human message-codings and classifications generated by the respective text classification approaches. Our results show that Gradient Boosting Machines are a good choice for separating empty talk from relevant talk messages. This holds (1) independent of the training set size, (2) when the ratio of empty talk to relevant talk is low, and (3) when there are not only two, but three message classes.