Skip to main content

​
Add a model

A model is a class in models/ with load_model, unload_model and generate(prompt, image, **gen), which returns the answer text. A chat model that takes one image and one text turn can subclass HFChatVLM, as Aya Vision does. Register it with one row in REGISTRY in predict/factory.py and one line in _MODULES in models/__init__.py, then run it with --model <key>.
models/aya_vision.py
predict/factory.py
The registry row gives the class, the variable that holds the local model path, the default Hugging Face id, the cache name and the dtype. build_predictor(key) wraps the model in a Predictor, and the cache name is part of every cache key, so keep it fixed once you have results. A new model’s judged scores need verdicts for its own caption mentions, so run it once with --live-judge.

​
Thinking ablation

This is separate from the main results. It scores COCO with Mod-POPE, as in the main table, on the same 500 images and 2,641 questions in both modes; Mod-POPE keeps 2,615 of them. With thinking on, the model gets 1,024 extra tokens on top of the 10-token POPE cap. Thinking moves Gemma-4-12B from 92.5 to 93.6. Over all 2,641 questions it misses fewer objects that are there (false negatives fall from 162 to 132) at about the same false positives (22 to 24). Gemma-4-E4B moves only from 93.6 to 93.8 and gets more cautious: false positives fall from 64 to 47, and false negatives rise from 107 to 117. A thinking run takes 8 to 11 GPU hours, against about 40 minutes with thinking off. Captions, CHAIR and THRONE were not run with thinking on, since each caption takes 130 to 145 seconds, about 6 times the cost. In 11 of the 2,641 Gemma-4-12B answers, the thought did not close within 1,034 tokens. Gemma-4-E4B closed every thought. The wrapper logs Thinking block not closed and returns the cut-off thought as the answer. The POPE parser checks for negative words first, so all 11 count as “no”. 8 of the 11 ask about an annotated object, so they count as wrong. If all 8 had been answered right, Gemma-4-12B’s Mod-POPE would be at most 94.0. The keys are gemma4-e4b-think and gemma4-12b-think. They use a separate class, so their cache keys never mix with thinking off. Their Mod-POPE was scored from the cached answers with the same drop list as the main table.