LLM Tool: improving transparency and validity in computational text analysis
Presented at EPSS (Belfast, June 18-20, 2026), this short presentation focuses on LLM Tool as a methodological infrastructure for transparent and validated computational text analysis. Rather than treating LLM annotation as a black box, the pipeline records prompts, codebooks, model settings, annotation outputs, training choices, and validation results so that analytical decisions can be inspected and reproduced.
Transparency through recorded analytical choices
LLM Tool logs the parameters that shape each annotation run, from concept definitions and prompts to model configuration and classifier training. Even when generation remains stochastic, the workflow preserves the information needed to reconstruct the operation.
Validation through human-consensus benchmarks
The presentation summarizes validation results across four annotation tasks, comparing LLM annotators and trained classifiers against human consensus labels and showing where task complexity drives performance differences.
Scroll Navigate between slides Click Interact with data Fullscreen for a better experience
Use arrow keys or click to navigate through the slides.