Multi-turn means the agent does not answer in one shot; it iteratively crops, zooms, and re-observes the image over several rounds to gather evidence — the "Thinking with Images" paradigm popularized by OpenAI o3.

Turn count is bounded, not fixed. Each task runs under a turn budget. The paper's whole thesis is that the structure of the search matters more than the raw number of turns: two agents show the extremes — Mini-o3 scales turns aggressively (dozens of bbox crops per trajectory), while ZwZ collapses zooming into a single forward pass. PixelEyes keeps trajectories short by making each turn precise.

Yes. Localization (grounding) is a perception task: given a query, find where the target sits in the image, expressed as a bounding box or a pixel mask. It is distinct from reasoning, which interprets the content once found. PixelEyes' core move is to hand this perception step to a dedicated tool (SAMTok) instead of asking the reasoning model to do it.

The paper backs it with three observations rather than asserting it:

• A measured trade-off: one jointly-trained model is consistently weaker at grounding than perception specialists and weaker at reasoning than strong general VLMs. Fine-tuning grounding into a reasoner degrades VQA ability — the two objectives fight.
• Two observed failure modes: weak perception → long "blind crop" trajectories; and correct crop but weak reasoning → inattentional blindness.
• A direct measurement: the LSR−accuracy gap on Pinpoint-Bench quantifies how often a model reaches the target but still fails — the entanglement made visible.

So the assumption is supported empirically, though it is presented as the paper's motivating diagnosis rather than a controlled ablation of "entangled vs not" on the same base model.

Yes — an agentic VLM loop has several moving parts beyond those two:

• Control / policy: decides the next action — call a tool, crop, or answer. This is the orchestrator that sequences turns.
• Tool use / action: the interface that invokes SAMTok, performs the crop, and feeds results back.
• Memory / context management: the trajectory history (past crops, masks) held in context — the thing that gets "polluted" by noisy crops.
• Stop / answer decision: when to terminate the search and commit.

At the raw-LLM level you would also list tokenization/embedding, attention, and decoding (plus expert routing in MoE), but in this paper's framing the relevant extras are the controller, tool interface, and trajectory memory.

It is a clean and mostly accurate division of labor, but a slight simplification.

• Perception tool = purely where: pixel-level localization of a named target. Accurate.
• Reasoner = more than just what to look for. It formulates the referring query, proposes regions, judges whether the tool succeeded, decides the next region, and — crucially — performs the final content recognition and answer. "What to look for" captures the query-formulation half but understates that the reasoner also does the concluding reasoning. (See Round 2, F4 — does this mean the reasoner still has perception ability?)

So: a great mnemonic for the control split, slightly under-describing the reasoner's full job.

Why a mask: a pixel-level mask is far more precise than a coarse bounding box, which matters enormously when targets occupy under 1% of a high-resolution image. Precise localization means the reasoner no longer has to spend turns compensating for sloppy grounding.

What "sloppy grounding" means: a localization that is coarse or slightly off — a box that is too large, misplaced, or clips a tiny target — so the resulting crop doesn't cleanly isolate the object. The reasoner then has to patch it up with extra crops and extra reasoning turns, which is precisely the trajectory bloat PixelEyes removes.

Which model: SAMTok [52], a referring-segmentation tool that unifies mask generation inside the language-model interface.

Important correction: the external expert does perception, not reasoning. SAMTok only answers where. Reasoning stays entirely with the VLM. So PixelEyes introduces an external perception specialist and keeps a single reasoner — the opposite of outsourcing reasoning. (Could outsourcing reasoning help instead? See Round 2, F6.)

The trick is coverage before depth. Concretely:

• Anchor in the original image. Every candidate region's coordinates are referenced to the full original image, not to the previous crop — so errors don't compound as you zoom.
• Propose a low-IoU region on failure. Whenever SAMTok fails to ground the target, the reasoner proposes a new region that barely overlaps the ones already tried (low IoU = genuinely new area), instead of re-cropping the same wrong spot.
• Expand siblings first. It explores neighboring semantic regions at the same level (breadth) before descending into any single one (depth).

This is precisely what kills the redundant DFS-style loops of prior agents that keep cropping deeper into an already-wrong sub-region. See Q15 for "low-IoU."

Inattentional blindness is borrowed from cognitive psychology (the "invisible gorilla" effect): you look right at something but fail to notice it. Here it names a specific gap — the agent visits the correct region (it is in the crop) but fails to recognize / answer it. The paper measures it directly as the gap between LSR (reached the target) and accuracy (answered correctly).

Annotation levels. Pinpoint-Bench uses instance-level masks and boxes, which is what cleanly separates localization failures from reasoning failures. Other granularities exist and could be used: category/semantic-level, panoptic or region-level, part-level, and box vs pixel-mask precision. Instance-level is the sweet spot for attributing a failure to "didn't find it" vs "found it but got it wrong."

Agreed — a good framing. The setting: VLMs shift from passively captioning a whole image to actively cropping/zooming/re-examining to gather evidence (popularized by OpenAI o3, studied as "Thinking with Images"). The difficulty is spelled out well: decisive evidence often sits in objects under 1% of a high-res image, so the agent must find a needle in a haystack and reason about it within a bounded turn budget. This one sentence justifies both halves of PixelEyes: precise localization (find the needle) and short search (bounded budget).

This is the crux of the motivation, and it is a strong argument. Existing methods ask one model to do both fine-grained region-level perception and general reasoning. But that same model is:

• consistently weaker at grounding than perception-oriented specialists [19, 44], and
• consistently weaker at reasoning than strong general-purpose VLMs [4, 37, 28, 27, 15].

In other words, joint training buys you a jack-of-both-trades that is master of neither — which is exactly why decoupling (let each specialist operate at its native granularity) is the proposed fix.

Two failure modes the paper names:

• Blind crops from weak perception: when the agent can't localize the correct region, it produces long trajectories of wrong crops.
• Inattentional blindness from degraded reasoning: even when the correct region is cropped, weak reasoning means the agent sees the target but fails to recognize it — the gap between visiting and answering, quantified in the benchmark.

The compared / contrasted methods, organized:

Named methods (Mini-o3, ZwZ) are the only ones the excerpts name explicitly; the bracketed clusters are cited as groups, so I list them by role rather than inventing names. Two shared properties PixelEyes drops: (1) reliance on the base VLM's grounding, and (2) search over coarse rectangular crops.

The original related-work text is embedded as screenshot D above, as you asked.

IoU = Intersection over Union, the overlap ratio between two regions (intersection area ÷ union area; 1 = identical, 0 = no overlap). A low-IoU region is a newly proposed region that overlaps little with the regions already tried.

It is the novelty constraint that drives the breadth-first behavior: when grounding fails, jump to a genuinely different area rather than re-cropping near the same wrong spot. That's what prevents the redundant loops.

Scaling turns is brute force with real costs: more compute, longer trajectories, and a context window that fills with noisy crops that degrade later decisions — with diminishing returns. Training the structure of search (mask-precise grounding + BFS coverage) instead makes each turn count. Benefits:

• Shorter trajectories for the same or better accuracy.
• Less context pollution — fewer wrong crops carried forward.
• Better sample efficiency and generalization — the agent learns how to search rather than memorizing longer rollouts.

The paper calls this a "substantially stronger lever" than adding turns.

What they're doing: they need expert demonstrations of the decoupled search behavior to train on, but hand-labeling trajectories is expensive. So they bootstrap: take a strong general VLM as a teacher, give it the same mask_based_crop tool, and let it interact in a closed loop on existing image–question pairs. They keep only trajectories that reach the correct answer, yielding PixelEyes-6K (5.8K clean trajectories). Those become supervised fine-tuning data for the student, Qwen-3-VL, and vanilla GRPO then sharpens the policy.

Why Gemini-3-Flash: the "Flash" tier is a fast, cheap, yet strong general VLM — you need to run thousands of multi-turn rollouts, so cost and speed matter, while quality must stay high enough that filtered trajectories are genuinely expert. It is a practical teacher choice, not a claim that it is the single best model.

The VLMs referenced across the paper:

This lists the models the paper cites, which is what matters for reading it. The absolute "current SOTA" leaderboard shifts month to month — treat Gemini-3-Flash / Qwen-3-VL as the paper's generation, and verify against a live leaderboard if you need today's ranking.

GRPO = Group Relative Policy Optimization, a reinforcement-learning method for LLMs (introduced with DeepSeekMath and used in DeepSeek-R1). Instead of training a separate value/critic network like PPO, it samples a group of responses per prompt and normalizes each response's reward against the group's mean and standard deviation to get its advantage. Dropping the critic makes it cheaper and simpler, which is why it's popular for reasoning RL.

"Vanilla" just means the standard, unmodified formulation — no custom reward shaping or algorithmic tweaks. The point they're making is that even off-the-shelf GRPO is enough to sharpen the policy once the search structure is already taught by SFT.

The paper argues existing benchmarks make it hard to evaluate visual search cleanly — some are saturating, some reward reasoning over search, and one lacks spatial labels. Hence Pinpoint-Bench.

Pinpoint-Bench adds a strict zero-hint protocol (no spatial cue, no macro-anchors), multi-alias answer matching, and reports LSR + Turn-to-Answer alongside accuracy.

Short version, then the table:

• Mean ROI / target area — the average share of the image the target occupies (0.07% here). A difficulty descriptor, not a scored metric.
• LSR (Localization Success Rate) — a trial counts as a success if any crop covers the target. Isolates perception/search from the final answer.
• TAE (Turn-to-Answer Efficiency) — how few turns it takes to reach the answer. Isolates trajectory efficiency.
• LSR − Accuracy gap — the model reached the target but still answered wrong. This is the direct measure of inattentional blindness.

The individual metrics are natural and partly have precedent in active-perception work. What is distinctive is the packaging: zero-hint + instance masks let the LSR−accuracy gap read out inattentional blindness cleanly.

How: through a tool-call interface inspired by visual programming. The reasoner emits a tool call (a referring expression); SAMTok returns a mask; the system crops to it and feeds the crop back into the next turn. Perception is a pluggable external model, so grounding is never fine-tuned into the reasoner — which is what avoids the perception–reasoning trade-off. A Switchable Tool Use fallback handles cases where masks are ill-defined (charts, maps, dense text) by reverting to a plain bounding-box crop.

Convenient? Reasonably, with caveats:

• Upside: modular — you can swap the segmentation tool, each module runs at its native granularity, and you avoid destructive joint training.
• Cost: you need a strong segmentation tool available at inference; tool calls add orchestration and latency; and the model still has to be taught to use the tool well, which is exactly why PixelEyes-6K + GRPO exist. Decoupling isn't free — it moves the effort from joint training into building the tool interface and the demonstration data.

It is not a ranking of power — grounding and reasoning are different competencies, exactly as you suspect. Prior methods fine-tune grounding into the reasoner purely for convenience: one model, one forward loop, no external tools to orchestrate. But the two objectives then compete for the same weights. Optimizing precise box/mask prediction pulls parameters toward spatial regression; optimizing open-ended VQA pulls them toward broad semantic reasoning — push on one and the other degrades (the perception–reasoning trade-off). So neither is "stronger"; they simply don't co-habit one set of weights gracefully. PixelEyes' fix is to stop fusing them.

Partly — and this is the subtle bit. "Perception" is not one skill; it splits into recognition (what is in this crop?) and localization (where exactly is it?). The reasoner (a VLM) keeps the recognition half — that's how it reads a crop and produces the final answer. What it lacks is precise localization, especially for tiny targets. So the reasoner does hold some perception, just not the pixel-precise grounding SAMTok provides (other models can do this job too — see F24). PixelEyes outsources only the where and lets the reasoner keep the what — which is why the neat "reasoner = what / tool = where" slogan slightly hides the overlap.

Perception = recognition + localization

In principle yes — you could call a stronger reasoning model as a tool too. But three things make it the wrong lever here: (1) the diagnosed bottleneck was perception, not reasoning — the base VLM already reasons well; (2) outsourcing reasoning adds latency, cost, and a harder coordination problem (you must ship rich visual context to a second model); (3) it reintroduces exactly the complexity the paper is shedding.

Did they test it? The excerpts I have don't report a "reasoning-outsourced" ablation, so I can't confirm it either way — that would need the full experiments section (Sec. 4). Flagging this as not shown in the provided text rather than a definite no.

These are two coordinate-referencing schemes:

• Relative to previous crop: each new region is described inside the last crop's frame. A small error in one crop shifts the frame for the next — errors compound (drift) as you zoom.
• Anchor to original: every region maps back to the full original image, so a bad crop never poisons the next; the agent can jump anywhere.

Do these exhaust the options? No. You could also anchor to an intermediate parent region (a hierarchical zoom stack), use absolute vs normalized coordinates, or a hybrid that anchors to the original but remembers a zoom path. PixelEyes just picks the most robust end of that spectrum — always anchor to the original.

IoU (Intersection over Union) is the overlap ratio between two regions. A low-IoU proposal barely overlaps the regions already tried — so it points the search at fresh territory instead of re-cropping the same wrong spot.

The regions are organized semantically and hierarchically, not as a fixed pixel grid: the whole image splits into meaningful regions (e.g., "left building", "storefront", "street"), each of which can split further. BFS visits siblings at the same level first, then descends — cover breadth before committing to depth.

• Cropping — cutting out a sub-region of the image to focus on it.
• Zooming — enlarging (up-sampling) a region so fine detail becomes legible; usually crop + upscale.
• Re-examining — looking again: re-observing a region (or a new one) at higher resolution to gather more evidence.

These are the actions an active-perception agent takes each turn, versus a passive model that captions the whole image once.

An English idiom for something extremely hard to find because it is tiny within a vast space (literally, a needle lost in a haystack). Here it is near-literal: the decisive object can occupy under 1% — often ~0.07% — of a high-resolution image, so locating it really is a needle-in-a-haystack search.

A heuristic is a rule-of-thumb: a practical strategy that is cheap and usually works but carries no guarantee of being optimal (e.g., "always zoom into the center first"). "Heuristic or tree-based search" therefore means hand-crafted search rules, as opposed to a search policy learned from data — which is what the RL-trained agents and PixelEyes use.

Yes — the paper's constraints (anchor-to-original + low-IoU novelty + breadth-first ordering + switchable fallback) are one point in a larger design space. Natural additions:

• a depth budget or confidence threshold that flips from BFS to DFS once a promising region is found;
• best-first / beam search — rank candidate regions by a learned score and expand the most promising, blending breadth and depth;
• a revisit penalty or per-scale IoU thresholds.

Your instinct is a real research direction: an adaptive BFS↔DFS (best-first) semantic search that spends breadth when uncertain and depth when confident. This paper's fixed BFS is a strong, simple baseline such work could build on.

General-purpose does not mean "perception + reasoning" exactly. A general-purpose VLM is one trained broadly to handle many multimodal tasks — VQA, captioning, OCR, chart/diagram reading, dialogue, reasoning — across domains, via instruction-following and transfer, without task-specific fine-tuning. Contrast a specialist (e.g., a segmentation-only model good at one thing). Fuller definition: broad, transferable multimodal competence; strong at recognition and reasoning; instruction-following — but typically only coarse at precise localization. That last gap is exactly why PixelEyes pairs it with a perception specialist.

PPO (Proximal Policy Optimization) is an actor–critic policy-gradient method: it improves the policy while a clipped objective keeps each update from moving too far, which stabilizes training. In RLHF it needs a separate value/critic network and a learned reward model. GRPO's whole appeal is dropping the critic. The broader landscape:

Trend for reasoning models: drop the critic (PPO → GRPO / RLOO) and often replace the learned reward model with rule-based / verifiable rewards. "Vanilla GRPO" = this table's GRPO row, unmodified.

It means the hard part is already handled by supervised fine-tuning on the expert trajectories — SFT teaches the model the structure of the search (how to call the tool, how to run BFS, when to stop). Once that behavior is in place, you don't need a bespoke RL algorithm to improve it; plain, unmodified ("off-the-shelf") GRPO is enough to sharpen the remaining decisions. In short: good demonstration data does most of the work; RL is a light polish, not the main engine.

1. The reasoner writes a referring expression (e.g., "the red sign on the left building").
2. That goes as a tool call to SAMTok.
3. SAMTok runs referring segmentation and returns a pixel mask of the referred object.
4. The system converts the mask to a crop region (its bounding box, usually padded) and crops/zooms the original image to it.
5. The crop is fed back to the reasoner to recognize/answer, or to decide the next region.

If SAMTok fails to ground the target, BFS proposes a new low-IoU region; if masks are ill-defined (charts/maps/dense text), Switchable Tool Use falls back to a plain bounding-box crop.

Exactly — that's the right reading. The agent's crops are its probes into the image. LSR asks: across the whole trajectory of crops the agent made, did any crop actually cover the target region? If yes, localization succeeded — regardless of whether the final answer was right. So LSR scores the search / probing process; the gap between LSR and accuracy is the target being probed but still mis-answered (inattentional blindness).

Visual programming (in this line of work, e.g. VisProg and ViperGPT) means an LLM solves a visual task by writing a short program — a sequence of calls to modular vision tools (detect, crop, count, OCR, segment…) — and composing their outputs, much like calling functions from a library. The LLM is the "programmer"; the vision modules are the "library". PixelEyes borrows this: the reasoner issues tool calls (to SAMTok) as program steps, letting each module run at its native granularity, instead of forcing one end-to-end model to do everything.

Because a mask is geometry. A mask marks, per pixel, whether that pixel belongs to the target. From the set of "target" pixels you can compute the object's bounding box (the min/max x and y of those pixels) or its centroid — a deterministic step. The system then crops the original image to that box (often padded, then upscaled). So the chain is mask → box → crop, pure geometry with no learning needed. A mask beats a raw predicted box because it outlines even a small or irregular object tightly, giving a cleaner, better-centered crop.

• Binary mask — an H×W array, 1 = object, 0 = background.
• Soft / probability mask — values in [0,1], a per-pixel confidence.
• RLE (run-length encoding) — a compressed binary mask (COCO uses this).
• Polygon — an ordered list of boundary vertices.
• Instance / multi-class mask — integer labels per pixel saying which object or class.

Example: for a 4000×3000 image, the mask is the same size, almost all 0s with a small blob of 1s where a road sign is; its bounding box is read off from that blob.

The alignment problem: the reasoner must phrase referring expressions the segmentation tool can act on, and correctly interpret the masks it gets back. How PixelEyes keeps them aligned:

• Shared interface. SAMTok "unifies mask generation within the language-model interface" — it's designed to be called in the model's token space (special call/return tokens), so there's less representational mismatch than bolting on a foreign API.
• Demonstration training. PixelEyes-6K teaches the reasoner, by example, how to phrase queries the tool handles well and how to react when a mask fails — alignment learned from expert trajectories.
• GRPO then tunes the policy to call the tool effectively.
• Switchable Tool Use is a safety valve: when the mask modality is ill-suited (charts/maps/dense text), fall back to a box crop rather than force a bad mask.

On architecture: the paper deliberately keeps perception external (a clean tool boundary), avoiding fusion of the two representation spaces. A heavier alternative — feeding mask features directly into the reasoner's latent space — would tighten alignment but reintroduce joint-training coupling, which is what they're avoiding. (Some of this is inferred from the tool-call design; the excerpts describe the interface, not every training detail.)

Yes — mapping a text query to a precise mask is an active area, and several families could slot into the same tool interface. Roughly:

• Promptable segmentation (need a point/box, not language): SAM and SAM 2 (adds video). Best-in-class mask quality, but they expect a spatial prompt rather than a referring expression — so they're usually paired with a text-to-box stage.
• Language → mask (referring / reasoning segmentation) — the closest match to SAMTok's role: LISA (LLM + SAM, "reasoning segmentation"), GLaMM, PixelLM, SEEM, GSVA, and classic referring-segmentation nets like LAVT, CRIS, ReLA. These take a phrase directly and return a mask — exactly what an agent wants.
• Open-vocabulary detection / grounding (text → box), then hand the box to SAM for a mask: GroundingDINO, OWL-ViT / OWLv2, GLIP. The popular open-source recipe Grounded-SAM = GroundingDINO → SAM does text → box → mask.
• Unified VLMs with grounding output: Florence-2, Kosmos-2, Qwen-VL grounding, PaliGemma — emit boxes (some masks) from text, though often coarser than a dedicated segmenter.

Trade-off: SAM-family gives the cleanest masks but needs a prompt; language-native tools (LISA / GLaMM / SEEM) accept a referring expression directly; Grounded-SAM is a strong, widely-used open pipeline. What PixelEyes actually needs is any tool that turns a referring expression into a precise mask — SAMTok is their choice, but a LISA-style or Grounded-SAM-style tool could occupy the same pluggable slot. (The field moves quickly, so newer options keep appearing; I'm naming established ones, not asserting SAMTok's internal design, which the excerpts don't detail.)