Mirage - Mirage

Mirage is an evaluation framework for studying object hallucination in vision-language models. It provides shared interfaces for datasets, model inference, prediction caching, and metric scoring so evaluation runs can be inspected and repeated. The runnable entry point, demo.py, connects the supported COCO and Visual Genome dataset paths with model wrappers and evaluation metrics. Its prediction cache lets you score saved model outputs again without repeating inference. Each metric keeps its own protocol and assumptions, so results should be interpreted within the metric being used.

Get started

Install Mirage and run the one-image smoke test.

Datasets

Prepare COCO and Visual Genome inputs.

Run evaluations

Choose a dataset, model, metric, and output path.

Metrics and models

Review available model wrappers and scoring protocols.

Run on an HPC cluster

Prepare a GPU job and submit it with SLURM.

Maintain the docs

Preview changes and see what the automated checks cover.

Evaluation data flow

Mirage turns dataset examples into model responses, scores those responses, and writes a result file. The prediction cache avoids repeating model inference when the same inputs are evaluated again.

Load COCO or Visual Genome examples

The dataset loader reads the image files and annotations, then provides each image with its ground-truth objects and any metric-specific data.

Get a model response

The predictor checks the response cache using the model and request inputs. On a cache hit, Mirage reuses the saved response. On a miss, the selected model wrapper runs inference and the predictor saves its response in the cache.

Score responses with a metric

The selected metric compares model responses with the ground truth and applies its own evaluation protocol. You can run a different metric against cached responses without running the model again.

Write the evaluation result

The demo writes the scores and run details to the requested JSON output path.

Reference model results

These reference results compare three model configurations across COCO and Visual Genome. Values are percentages from 1,000 images per model and dataset. Higher is better for POPE accuracy, ROPE strict accuracy, and THRONE F1. Lower is better for CHAIR-I hallucination rate. The tables use Mintlify’s native tabs and tables so the values remain readable on every screen size.

COCO
Visual Genome

Model	POPE accuracy ↑	ROPE strict accuracy ↑	CHAIR-I hallucination rate ↓	THRONE F1 ↑
PaliGemma-3B	94.2%	21.6%	9.1%	72.9%
Qwen2-VL-2B	93.6%	3.6%	12.8%	73.4%
Qwen2-VL-7B	93.5%	13.1%	12.7%	72.9%

Model	POPE accuracy ↑	ROPE strict accuracy ↑	CHAIR-I hallucination rate ↓	THRONE F1 ↑
PaliGemma-3B	88.0%	8.2%	67.8%	21.5%
Qwen2-VL-2B	88.7%	8.6%	75.6%	24.0%
Qwen2-VL-7B	89.5%	10.3%	69.3%	23.6%

ROPE strict accuracy requires an exact label match. THRONE F1 is normalized to 38 distractor categories per ground-truth object. Metrics measure different behaviors and should be compared within each metric, not combined into a single model ranking. The source values are in the model results CSV. A new model belongs in these tables only after its results have been measured and reviewed.