> ## Documentation Index
> Fetch the complete documentation index at: https://mirage-eval.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Mirage

> A modular framework for evaluating object hallucination in vision-language models.

Mirage is an evaluation framework for studying object hallucination in vision-language models. It provides shared interfaces for datasets, model inference, prediction caching, and metric scoring so evaluation runs can be inspected and repeated.

The runnable entry point, `demo.py`, connects the supported COCO and Visual Genome dataset paths with model wrappers and evaluation metrics. Its prediction cache lets you score saved model outputs again without repeating inference. Each metric keeps its own protocol and assumptions, so results should be interpreted within the metric being used.

<CardGroup cols={2}>
  <Card title="Get started" icon="rocket" href="/getting-started">
    Install Mirage and run the one-image smoke test.
  </Card>

  <Card title="Datasets" icon="database" href="/datasets">
    Prepare COCO and Visual Genome inputs.
  </Card>

  <Card title="Run evaluations" icon="terminal" href="/running-evaluations">
    Choose a dataset, model, metric, and output path.
  </Card>

  <Card title="Metrics and models" icon="chart-column" href="/metrics-and-models">
    Review available model wrappers and scoring protocols.
  </Card>

  <Card title="Run on an HPC cluster" icon="server" href="/hpc">
    Prepare a GPU job and submit it with SLURM.
  </Card>

  <Card title="Maintain the docs" icon="book-open" href="/maintaining-docs">
    Preview changes and see what the automated checks cover.
  </Card>
</CardGroup>

## Evaluation data flow

Mirage turns dataset examples into model responses, scores those responses, and writes a result file. The prediction cache avoids repeating model inference when the same inputs are evaluated again.

<Steps>
  <Step title="Load COCO or Visual Genome examples">
    The dataset loader reads the image files and annotations, then provides each image with its ground-truth objects and any metric-specific data.
  </Step>

  <Step title="Get a model response">
    The predictor checks the response cache using the model and request inputs. On a cache hit, Mirage reuses the saved response. On a miss, the selected model wrapper runs inference and the predictor saves its response in the cache.
  </Step>

  <Step title="Score responses with a metric">
    The selected metric compares model responses with the ground truth and applies its own evaluation protocol. You can run a different metric against cached responses without running the model again.
  </Step>

  <Step title="Write the evaluation result">
    The demo writes the scores and run details to the requested JSON output path.
  </Step>
</Steps>

## Reference model results

These reference results compare three model configurations across COCO and Visual Genome. Values are percentages from 1,000 images per model and dataset. Higher is better for POPE accuracy, ROPE strict accuracy, and THRONE F1. Lower is better for CHAIR-I hallucination rate. The tables use Mintlify's native tabs and tables so the values remain readable on every screen size.

<Tabs>
  <Tab title="COCO">
    | Model | POPE accuracy ↑ | ROPE strict accuracy ↑ | CHAIR-I hallucination rate ↓ | THRONE F1 ↑ |
    | - | -: | -: | -: | -: |
    | PaliGemma-3B | 94.2% | 21.6% | 9.1% | 72.9% |
    | Qwen2-VL-2B | 93.6% | 3.6% | 12.8% | 73.4% |
    | Qwen2-VL-7B | 93.5% | 13.1% | 12.7% | 72.9% |
  </Tab>

  <Tab title="Visual Genome">
    | Model | POPE accuracy ↑ | ROPE strict accuracy ↑ | CHAIR-I hallucination rate ↓ | THRONE F1 ↑ |
    | - | -: | -: | -: | -: |
    | PaliGemma-3B | 88.0% | 8.2% | 67.8% | 21.5% |
    | Qwen2-VL-2B | 88.7% | 8.6% | 75.6% | 24.0% |
    | Qwen2-VL-7B | 89.5% | 10.3% | 69.3% | 23.6% |
  </Tab>
</Tabs>

ROPE strict accuracy requires an exact label match. THRONE F1 is normalized to 38 distractor categories per ground-truth object. Metrics measure different behaviors and should be compared within each metric, not combined into a single model ranking. The source values are in the [model results CSV](https://github.com/Advanced-Vision-and-Learning-Lab/mirage/blob/main/docs/model-results.csv). A new model belongs in these tables only after its results have been measured and reviewed.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.