User comments are a broadly disputed feature of online news outlets. Audience comments on news sites generate traffic, but also praise for their (idealized) deliberative potential, such as providing public debate forums. On the other hand, user commentary has been criticized for not releasing this potential: Empirical findings suggest that user contributions are often negative in tone, rarely provide justifications for their criticism, spread incivility, and infringements. While many major Swedish news sites disabled commenting features, other organizations reflected upon adjustments. Outlets (still) with user commentary often moderate, mostly focusing on filtering problematic comments. Larger outlets experiment with more active comments engagement. Studies indicate that active moderation, with an encouraging mindset, can increase participation and decrease uncivil commentary. With Mario Haim (Uni of Stavanger) and Austrian news site Der Standard, we developed, tested, and validated a classification grid to automatically identify problematic comments but also quality comments. Such quality comments can be promoted (like New York Times with "NYT Picks") or used for editorial processes, e.g. if containing reasoned argument or personal experiences for a follow-up story. Automated classification can be an important, resource-saving support in finding contributions in the daily flood of user commentary.
The classification grid consulted deliberation theory and prior work that dealt with practical machine learning applications in comment moderation. 980 manually coded comments from Der Standard were used as a training data set. Our results show that while text classification remains to be delicate, machine learning yields better results for categories indicating “good” commentary. We validated our classification grid through guided interviews with two community management members of Der Standard.
The talk will present key findings of the project but also addresses methodological, practical, and ethical questions.
Before joining the Department of Journalism at Södertörn University in May last year, Nina Springer studied and worked at the Department of Media and Communication at LMU Munich and spent the winter term 2017/18 as a visiting researcher at the Berkman Klein Center for Internet and Society in Cambridge, MA. Nina works at the intersection of journalism and audience studies.