Work done
The tracker on the other pages is the visible end of a larger piece of work: a pipeline that collects official documents, extracts every manifesto promise, finds evidence for it as of any past date, has language models judge it under strict checks, validates the judges against hand labels, and fits survival models. This page shows what was built, how each stage was checked, and where to find it. Every count below is computed from the repository and database when the site is built.
In numbers
In numbers
The pipeline
Stage by stage, and how each was checked
Open a stage for what it did and how it was checked.
01Corpus collection
Official Tamil Nadu documents, each with a defensible availability date.
Crawled and parsed Tamil Nadu government documents (policy notes, budget speeches, Governor's addresses, Government Orders, audit reports, scheme pages), recovering older files from web archives, and assigned each a defensible availability date.
Checked: 10 of 10 flagged dates reviewed by hand were fine; all dates traced to a source; 95.6% dated to the day, 220 undated documents excluded from every dated run.
02Promise extraction
Three manifestos cut into atomic promises, frozen before any evidence was read.
Read three manifestos (an English text layer, Tamil in a legacy font mapped to Unicode, and a 96-page scanned Tamil manifesto transcribed page by page and cross-checked against OCR) and cut them into atomic promises with the original text kept. Lists were frozen before any evidence was read.
Checked: verbatim, number and name checks on every promise; hand checks of the most problematic items (6 of 8, then 8 of 10, the two misses being a labelling error on the sheet).
03Classification and indicators
Features from the manifesto text only; vague promises split into checkable indicators.
Labelled each promise from its manifesto text only: sector, type, cost band, vague or specific, Union dependence, continues an existing scheme, headline. Vague promises were split into two to four checkable indicators with search queries.
Checked: a rule that no number or name may appear that the manifesto does not contain; hand checks (9 of 9, then 8 of 9 with the miss fixed across 14 promises).
04Evidence retrieval
For every indicator and landmark date, the five best passages that existed by then.
For every indicator and every landmark date, found the five most relevant passages from documents that existed by that date, combining keyword (BM25) and meaning-based (dense embedding) search with rank fusion.
Checked: a database-wide leakage check (zero violations over 84,330 retrieval rows); hand check of the worst cases (3 ok, 3 partly right, 3 wrong, all query-wording misses, recorded as limitations).
05Judging
Two language models read only the supplied passages; code rejects unsupported verdicts.
Two judges read only the supplied passages: a free-tier Gemini model for every task, and Claude in isolated subagents for a priority subset. Code rejects any verdict whose quotes are not in the passage, whose date is not written in a cited passage, or that claims delivery without evidence. Within each judge, a realisation found at an earlier date is carried forward, so "realised by month m" never flips back to no.
Checked: a memory probe (0 of 40 answers claimed a status with no evidence), the hand check (5 ok, 4 wrong, fixes applied or logged), and a gold-set comparison.
06Evaluation
A hand-labelled sample used only to measure the judges.
A random stratified sample of 40 promises was labelled by hand without seeing any verdict, then used only to measure the judges: agreement on realised-or-not, recall, precision, specificity, date error and error causes, with 95% Wilson intervals.
Checked: the labels never enter any prompt (overlap test), and only the evaluation code reads them.
07Survival modelling
Kaplan–Meier, Cox model, reverse backtest, three-line TVK projection.
Kaplan–Meier curves, a Cox model with five promise characteristics, a backtest (fit on DMK 2021, predict AIADMK 2016 from its early months) and a projection for TVK under three stated assumptions, with a bootstrap band. The whole analysis was run under two single-judge settings, Claude-only and Gemini-only.
Checked: the training-term-only rule is enforced by an assertion with a test; sensitivity runs for coarse dates and for the fulfilled-only event.
08This website
Static, generated from the committed results.
A static site generated from the committed results: nothing on it is typed by hand, no hand-labelled data is published, and each status is shown with its evidence and its judge.
Timeline
Timeline
Method of work
How the work was organised
Each stage began with a written specification: purpose, inputs, outputs, behaviour, a leakage warning, acceptance criteria and tests. Open questions were put to the project owner before code depended on them, and every settled choice is in a numbered decision log. If the code had to differ from the spec, the spec was changed in the same step. The rules that protect the results from leakage (no memory-based judging, hard date cutoffs, manifesto-only covariates, the test set kept out of everything, training-term-only fits) were repeated in every spec and enforced in code and tests.
Who did what. The project owner set the questions, approved every decision, did the hand reviews and labelled the test set. Claude (an AI assistant) drafted specifications, wrote and tested the code, and acted as one of the two judges inside the isolation rules above. Gemini acted as the other judge and as a bulk text processor. The measured accuracy of both judges against the owner's labels is on the whitepaper.
Sources