From Disagreement to Data: Tracking Collaboration in Text Annotation for Reproducible Science and Collaborative AI

We are developing an open-source web-based annotation tool that makes the collaborative process behind text classification transparent and reproducible. The tool addresses a critical challenge in computational social science and AI research: current annotation practices often lack transparency about how labels are produced, making it difficult for other researchers to replicate findings.

Our platform implements a three-step iterative annotation process conducted in small batches. Annotators begin by independently labeling the same texts in pairs. When they disagree, they engage in structured discussions through an integrated chat interface, exchanging reasoning and clarifying interpretations before optionally revising their initial judgments. Cases that remain unresolved after discussion are escalated to experts, who review all annotations and discussion histories and can modify annotation guidelines when existing rules prove insufficient for reaching agreement.

The tool systematically tracks the entire collaborative process through version control, creating a complete audit trail from raw text to final labels. This documentation captures all annotator discussions and reasoning, every revision to annotation guidelines, including justifications, the evolution of annotation guidelines across annotation rounds, and patterns of disagreement and how they were resolved.

By making collaborative sense-making visible, the tool serves two key research communities. For computational social scientists, it improves interpretive alignment and methodological transparency, enabling better reproducibility across research teams. For collaborative AI researchers, it generates rich metadata about how humans jointly reason about annotation tasks, which is essential information for developing AI systems that can effectively support human annotators.

The project is a collaboration between the Computational Social Science and Collaborative Artificial Intelligence groups at the University of Stuttgart.

Project Members

• Prof. Andreas Bulling | Institute for Visualization and Interactive Systems (Fac. 05)
• Prof. Raphael Heiberger | Institute for Social Sciences (Fac. 10)
• Lukas Erhard
• Susanne Hindennach