claude -p session, with the model pinned in judge/manifest.json and one instruction file per task in judge/.claude/skills/. It grades caption mentions (present, absent, not an object, unsure), ROPE answers (correct, related, wrong, unsure) and POPE “no” questions (present, absent, unsure). Scoring reads stored verdicts from judgments/ and never calls the judge, except run.py --live-judge, which grades new caption mentions that have no verdict yet (ROPE has no live path).
Committee
The committee is the 9 models in the results table, and their verdicts are frozen. The objects the judge confirmed in their captions form the coverage pool for Mod-CHAIR and the completed labels for Visual Genome THRONE. A new model is scored against the committee plus its own verdicts and never changes the committee’s rows, so the published numbers stay fixed.Use the published verdicts
The verdicts, the committee and the final table are injudgments/ in the repo, checked against the sha256 hashes in judge/manifest.json. Use them to score the 9 models, or to score a new model against the same committee, without running a judge yourself.
Use your own judge
A judge only has to return the same verdict labels. Pass any object with the same interface asjudge.Judge (its call and committee) to run.main(judge=...), or edit the instruction files in judge/.claude/skills/. The skill files and the stored verdicts are checked against the hashes in judge/manifest.json, so any change needs a new manifest.
Agreement with human labels
A human annotator labelled a sample of judge items in the review page below. Recall and precision are for the positive class: a hallucinated caption mention, a wrong ROPE answer, or a POPE “no” question that should be removed.
The samples are small, and the caption and POPE instructions were revised after seeing these labels.
