Whitepaper: method, accuracy and limits

Course project, Data Science & AI, 2026. See everything that was built and checked, stage by stage. This page states what was done, how it was checked and what it cannot tell you. Numbers on this page are read from the published data files.

In short

    1. Question and scope

    Do Tamil Nadu governments realise their manifesto promises, how fast, and what can the past say about the current government? Three terms are studied: DMK 2021–26 ( promises), AIADMK 2016–21 () and TVK 2026– (, in progress). Only the party that formed the government is analysed. The project does not judge whether a policy was good, and it does not rank parties.

    2. Data

    Promises. Promises come from the manifestos (DMK in English; AIADMK and TVK in Tamil, translated to English with the original text kept beside each promise).

    Evidence. Evidence is a corpus of 9,254 English official documents (about 60,000 passages): departmental policy notes, budget speeches, Governor's addresses, Government Orders, audit reports and official scheme pages. Documents are dated, and a document counts as available only from the date it could have been public. About 220 documents have no usable date and are excluded. Newspapers and party trackers are not evidence.

    3. Pipeline

    1. Promise extraction. Each manifesto is cut into items and each item into atomic, independently checkable promises (1,674 in total). The lists were frozen before any evidence was read.
    2. Classification. Each promise gets features read only from the manifesto text: sector, type, cost band, vague or specific, whether it depends on the Union government, whether it continues an existing scheme. A vague promise is split into two to four checkable indicators.
    3. Retrieval. For each indicator and each landmark date, the five most relevant passages are found (keyword and meaning-based search combined), only among documents available by that date.
    4. Judging. A language model reads only those five passages and returns a status, a date, the passages it relies on and a word-for-word quote from each. Code rejects any verdict whose quote is not in the passage, whose date is not written in a cited passage, or that claims delivery without a citation.

    Status and date

    Statuses: fulfilled, partial, diluted, in progress, not started, blocked by an outside authority, unverifiable. A promise is realised (the event analysed) when it is fulfilled, partial or diluted with a date. The date is the first official step that put it into operation: a Government Order, a scheme launch or an enacted law. Budget announcements and statements of intent do not count. "Not started" means documents on the topic exist and show no official step; "unverifiable" means nothing in the corpus was about it.

    4. Keeping the answers leak-free

    Language models already know a lot about Tamil Nadu. The controls:

    1. the judge sees only the supplied passages and must quote them;
    2. at each landmark month only documents available by that date are shown, and a verdict at an earlier date is never revised from later evidence except by carrying an earlier finding forward;
    3. promise features use only manifesto text;
    4. the hand-labelled test set is never shown to any model;
    5. every model fit uses the training term only.

    A test with no evidence at all (the "memory probe") found 0 of 40 answers that claimed a status without evidence.

    5. Who judged what

    Two judges were used, and their verdicts are never mixed: each published setting uses one judge for every promise it covers, so within a setting all terms are treated identically. Claude-only: Claude (the "strong" judge, in fresh isolated sessions that read only the instructions and the passages) judged every promise at the end of the term for both completed terms, and month 5 for DMK 2021–26 and TVK 2026–. Eight promises never passed the judge checks in any attempt and are excluded from this setting. AIADMK's early months were not judged in full by Claude, so they are not part of it. Gemini-only: a lighter, free model (Gemini 3.5 Flash-Lite, the "bulk" judge) judged every task at every landmark for all three terms. Three hand corrections from the spot-check are recorded in the project repository but applied to neither setting, so that each is a pure single-judge result.

    6. How accurate are the verdicts?

    A random, stratified sample of promises was labelled by hand from the manifesto text, the corpus and the web, without seeing any verdict. The judges' verdicts at the end of the term (TVK: month 5) are compared with these labels. Reported: how often the judge agrees on realised or not (the event the survival models use), and on the exact status. Intervals are 95% Wilson intervals; with a sample this small they are wide.

    Accuracy figures appear here when the hand-labelled evaluation is published.

    Hand spot-checks of the worst cases

    Before this, the worst cases from each stage were reviewed by hand: retrieval (3 of 9 wrong, all search-query misses), judging (4 of 9 wrong: a date a few days outside the availability cap, an unresolved case, a general committee taken as evidence, and a cumulative amount that cannot be checked from five passages).

    7. Survival analysis

    For each term, time to realisation is measured in days from the swearing-in. A Kaplan–Meier curve shows the share realised by each month. A Cox model estimates which promise characteristics go with faster delivery (hazard ratio above 1 = faster). Promises with no realisation date are treated as not yet realised at the end of the observation period. Coarse dates (year or month only) are placed at the middle of their period.

    8. Reverse backtest

    The model is fitted on DMK 2021–26 only and asked to predict AIADMK 2016–21 from what was documented at its landmark months. Under the assumption that terms are independent draws, this is a fair test across terms, but it is not a forecast forward in time. The error is the average gap, in percentage points, between predicted and actual share realised.

    9. The TVK projection

    The projection starts from what is documented for TVK after five months and adds the number of further promises the historical hazards would realise, using a model fitted on both completed terms with equal weight. Three lines are shown because the result depends on one assumption, how to read TVK's slow documented start:

      The bands show fitting uncertainty only. Nothing here predicts what TVK will do: it states what historical patterns would imply.

      10. How much depends on the judge

      The same evidence was judged two ways: by Claude alone and by the lighter Gemini model alone. Neither is the truth. On the hand-labelled sample Gemini over-claims realisation (it calls some not-realised promises realised), while Claude almost never over-claims but misses many realised promises. The results of the two settings:

      Three things follow.

        11. Limitations

          12. Community corrections, now and later

          Now. Every promise in the explorer has a Suggest a correction link that opens a pre-filled GitHub issue (promise ID, the verdict shown, a form for the suggested status, date and evidence). Issues are reviewed by hand. They are stored separately from the verdicts, never change a verdict automatically, and are never used by the judging or survival models.

          General comments. The promise explorer also has a comments section. A visitor's comment is first checked in the browser against a short list of abusive words, then submitted as a public GitHub issue. Nothing is shown on the site until a person has read and approved it, and approved comments appear labelled "visitor comments, unverified". Comments are stored separately from the verdicts and never change a verdict or feed any prompt, threshold, fit or gold set. The word list is only a first pass and cannot catch everything, which is why approval by hand is the actual filter.

          Possible extension (designed, not built). A public judgement feature would store each submission as a separate record (promise ID, suggested status and date, evidence link, timestamp, a hashed anonymous contributor key) in its own table, never in the verdict tables. The rules would be:

          1. community records are never inputs to a prompt, a threshold, a fit or the gold set;
          2. they are shown beside, not instead of, the evidence-based verdict, labelled "visitor suggestions, unverified";
          3. anti-abuse limits per contributor and per promise, and moderation before anything is displayed;
          4. only evidence-bearing suggestions can trigger a re-check of a promise, and a re-check uses the same evidence-only judging as everything else;
          5. counts are never presented as a measurement, since self-selected visitors are not a sample.

          Open voting on a political topic invites coordinated campaigns, which is why this was left as a design rather than a feature.

          13. Reproducibility and licences

          Every stage is a command in python -m ept, every prompt is a versioned file and every strong-judge batch is committed. Code: GPL v3. Data exports and this text: CC BY 4.0. Quoted excerpts of government documents are linked to their official sources and remain their publishers' property.