LLM Tool: a transparent hybrid pipeline for computational policy analysis
Presented at IPPA IWPP5, workshop T09W02 "Machine learning, natural language processing and large language models for policy issues" (Ottawa, July 7, 2026), this talk locates LLM Tool within computational policy analysis. Once parliamentary debates, committee hearings, briefs, evaluations and media coverage are brought to a common textual format, the analytical question shifts to the annotation itself: how it is defined, how it is validated against human judgment, and how it is documented so that other teams can reuse it.
Transparency through recorded analytical choices
LLM Tool logs the parameters that shape each annotation run, from concept definitions and prompts to model configuration and classifier training. Even when generation remains stochastic, the workflow preserves the information needed to reconstruct the operation.
Validation through human-consensus benchmarks
The presentation summarizes validation results across four annotation tasks, comparing LLM annotators and trained classifiers against human consensus labels and showing where task complexity drives performance differences.
Scroll Navigate between slides Click Interact with data Fullscreen for a better experience
Use arrow keys or click to navigate through the slides.