LensBench

LensBench is a segmentation benchmark for explicit and implicit instructions across general, aerial, driving, and panoramic images. It contains 1,200 human-reviewed paired evaluation units and 4,800 instruction-level samples.

Paper · Project page · Code · Model

Dataset design

Each evaluation unit has four instructions crossing single/multiple targets with explicit/implicit wording. For a given target-cardinality condition, both wordings share the same target mask. Multiple targets are represented by their union mask. Targets and masks are fixed before instruction synthesis, followed by automated consistency checks and human review of each complete unit.

Domain	Paired units	Explicit / single	Explicit / multiple	Implicit / single	Implicit / multiple	Total samples
General	300	300	300	300	300	1,200
Aerial	300	300	300	300	300	1,200
Driving	300	300	300	300	300	1,200
Panoramic	300	300	300	300	300	1,200
Total	1,200	1,200	1,200	1,200	1,200	4,800

The release provides a single test split for evaluation.

Files and fields

LensBench/
├── README.md
└── test.parquet

Images are embedded in the Parquet file; no separate image download is needed.

Field	Type	Description
index	int64	Unique instruction-level sample index; not a paired-unit identifier.
image	Hugging Face Image	Embedded image bytes and original filename; decoded to a PIL image by datasets.
instruction	string	English instruction describing the segmentation target.
segmentation	struct	COCO compressed RLE with size and counts.
segmentation.size	list of int32	Mask dimensions in [height, width] order.
segmentation.counts	string	Compressed COCO RLE counts; encode as ASCII bytes before decoding with pycocotools.
domain	string	general, aerial, driving, or panoramic.
explicitness	string	explicit or implicit.
cardinality	string	single or multiple.

The release does not contain a bounding-box field. When a box is needed, derive it from the target mask; for multiple targets, use the enclosing box of the union mask.

Quick start

Install the loading and mask-decoding dependencies:

pip install datasets pillow pycocotools huggingface_hub

Load the dataset and decode the first target mask:

from datasets import load_dataset
from pycocotools import mask as mask_utils


dataset = load_dataset("SeerRay-Lab/LensBench", split="test")
sample = dataset[0]
image = sample["image"]
rle = {
    "size": sample["segmentation"]["size"],
    "counts": sample["segmentation"]["counts"].encode("ascii"),
}
mask = mask_utils.decode(rle)

assert mask.shape == (image.height, image.width)
print(sample["instruction"])
print(sample["domain"], sample["explicitness"], sample["cardinality"])
print(image.size, mask.shape)

To download the release and load it locally:

hf download SeerRay-Lab/LensBench --repo-type dataset --local-dir LensBench

from datasets import load_dataset


dataset = load_dataset(
    "parquet", data_files={"test": "LensBench/test.parquet"}, split="test"
)
implicit_samples = dataset.filter(lambda sample: sample["explicitness"] == "implicit")

Evaluation

Results below are reproduced from Table 7 in Section 4.3 of the Xiaomi-Lens-0 Technical Report (arXiv v1). The report evaluates all listed models with the same instructions, with each model predicting masks directly.

Marginal results pool single and multiple targets. Each cell reports mask cIoU / box Prec@0.5 (%, higher is better). Prec@0.5 uses a box IoU threshold of 0.5. Multiple targets use the union mask and its enclosing box. Bold marks the best score, and underlining marks the second-best score for each metric and column, including ties.

Method	Explicit	Implicit	General	Aerial	Driving	Panoramic
GPT-5.6-Sol	63.8 / 73.2	61.2 / 70.0	68.4 / 67.9	42.2 / 58.5	68.4 / 81.8	57.1 / 78.3
Gemini-3.6-Flash	61.4 / 74.5	61.0 / 70.3	63.6 / 65.8	50.1 / 62.5	66.8 / 81.0	60.8 / 80.3
MiMo-V2.5	51.4 / 52.2	47.4 / 46.8	54.2 / 45.2	32.6 / 38.6	50.5 / 52.8	45.4 / 61.4
Seed-2.1-Pro	62.1 / 70.2	55.9 / 64.5	60.6 / 56.7	50.1 / 58.5	66.8 / 77.4	59.9 / 76.7
Qwen3.8-Max	67.6 / 69.2	64.5 / 63.8	74.2 / 79.7	33.5 / 62.5	58.0 / 57.7	60.8 / 66.2
Kimi-K3	29.4 / 39.1	28.3 / 33.0	38.5 / 26.1	13.8 / 44.9	47.2 / 31.1	13.4 / 42.1
GLM-5.3-Flash	44.3 / 37.5	43.5 / 33.3	48.9 / 26.3	28.0 / 33.4	51.6 / 47.5	30.6 / 34.4
DeepSeek-V4-Flash-Vision-Exp	41.8 / 30.7	41.1 / 27.3	46.1 / 28.1	25.9 / 28.9	43.9 / 27.9	32.8 / 31.0
SAM 3	17.3 / 38.1	15.4 / 19.5	23.8 / 24.8	4.3 / 27.3	27.7 / 35.1	11.1 / 27.9
Xiaomi-Lens-04B	71.4 / 85.0	54.7 / 70.0	63.9 / 79.1	57.1 / 79.6	68.6 / 79.4	62.3 / 72.1

See the paper and project results for the evaluation protocol and comparisons. Preserve the domain, explicitness, and cardinality labels when reporting subgroup results.

License

The images and annotations in this release are licensed under the Apache License 2.0.

Citation

If you use LensBench, please cite:

@misc{xiaomilens0,
  title         = {Xiaomi-Lens-0 Technical Report},
  author        = {Pei Fu and Yike Liu and Yue Sun and Hanmo Wang and
                   Xiuwen Xi and Ruoceng Zhang and Shaojie Zhang and
                   Chao Jiang and Hang Li and Bingrui Liu and Zhenbo Luo and
                   Heng Qu and Changjian Xu and Ming Xu and Jian Luan},
  year          = {2026},
  eprint        = {2610.23279},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2610.23279}
}