> ## Documentation Index
> Fetch the complete documentation index at: https://mirage-eval.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics and models

> Implemented metric paths and model backends available in the demo.

## Metrics

The demo exposes these metric choices:

| Metric | What the implementation evaluates | Notes |
| - | - | - |
| CHAIR | Object mentions in generated image captions against annotated objects. | Uses COCO synonym handling for COCO; Visual Genome uses its object names. |
| POPE | Yes/no answers to questions about objects present or absent in an image. | The demo uses random negative-object sampling by default. |
| ROPE | Object recognition for numbered, boxed image regions. | The demo uses five boxes and asks the model to identify each region. |
| THRONE | Caption object predictions against the dataset's object categories. | Uses the loaded dataset's category or object-name list. |
| FaithScore | Faithfulness checks on generated captions. | Requires a local LLaMA model path via `--llama-model` or `MIRAGE_LLAMA_MODEL_PATH`, plus a PaliGemma verification model. |
| NOPE | Scores responses to the NOPE question set. | Requires pre-generated Turn 2 responses via `--nope-turn2`; Turn 1 input is optional. |

Metric availability can depend on the dataset path and local files. For example, ROPE needs annotated boxes, while NOPE reads its response JSON input rather than the normal dataset caption path.

## Models

The prediction factory supports two local vision-language model wrappers:

* `paligemma` uses `google/paligemma-3b-mix-224` by default. Override the model identifier or local path with `MIRAGE_PALIGEMMA_MODEL_PATH`.
* `qwen2vl` uses `Qwen/Qwen2-VL-2B-Instruct` by default. Override it with `MIRAGE_QWEN_MODEL_PATH`.

These are the model keys currently supported by `demo.py`. A different checkpoint can be used through the model-path override only when it is compatible with that wrapper. DeepSeek-VL is not currently supported; adding it requires a model wrapper and factory registration before it can be selected as a separate model key. See [Maintain the docs](/maintaining-docs) for the implementation handoff requirements.

Set `MIRAGE_VL_MODEL` to choose the default model when `--model` is omitted. Models are loaded for inference at runtime; model weights are not included in the repository.

## Result caching

Predictions pass through the `predict` package, which wraps model inference with a response cache. Cache identity and storage are implemented in `predict/identity.py` and `predict/cache.py`; the demo's SLURM example sets `MIRAGE_CACHE_DIR` to a job-specific directory.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.