Figure 1 — Four representative DF-World 0.1 domains with control overlays: post-apocalyptic wasteland, a fantasy scene (WASD overlay), an aerial mountain flight, and a sci-fi FPS (arrow-key overlay).

DreamForge-World 0.1 Preview: A Low-Compute Real-Time Controllable World Model · Daniyel Ayupov, Artur Markov-Tsoy · DreamForge AI Lab, Kazakhstan · arXiv:2606.30292 · cs.LG, cs.CV · June 2026 · trydreamforge.com

Abstract (paraphrased). A preview foundational world model for real-time interactive world simulation. It adapts the LongLive 1 autoregressive video stack (itself derived from Wan2.1-T2V-1.3B) and adds a residual action pathway inspired by the Matrix-Game family. Instead of chasing frontier-scale simulators, it targets a complementary axis: low-compute adaptation, consumer-GPU runtime, and broad interactive-capability coverage. It supports live keyboard/mouse control, multimodal initialization, mid-stream reprompting, dual-view operation, and minute-scale interactive rollouts at native 480p — reaching ~14–15 FPS on a single RTX 4090 with a small memory footprint. By reusing open video backbones with targeted adaptation runs, it is built cost-efficiently. The authors are explicit that it is not yet memory-complete or frontier-quality — it is a practical low-compute route to real-time controllable world-model previews on consumer GPUs.

“Causal AR + KV cache” is one option among several decoding paradigms. Two axes are independent: sequence structure (bidirectional vs. chunk-causal vs. frame-causal vs. token-AR) and per-step denoiser (multi-step diffusion vs. few-step distilled vs. next-token). Representative models per family:

Row colours: red = LongLive (study focus) · amber = Matrix-Game 2.0. Together they are DF-World's lineage: LongLive's frame-causal AR runtime + a Matrix-Game-style action pathway. Model names link to their papers.

Alibaba Tongyi Lab open T2V model (Feb 2025). DiT backbone + Wan-VAE (3D causal video autoencoder), flow-matching training, ~5s 480p, ~8.19 GB VRAM. It is the model LongLive fine-tunes, so DF-World inherits it.

• Frame-level rollout — one frame/step at a time, each conditioned on the past.
• Prompt switching via KV recache — recompute the cache from new prompt + existing prefix; smooth switch without a visual jump or semantic lag.
• Cacheable causal structure — causal attention lets past K/V be stored & reused.
• Short-window attention — attend only to a recent window; caps per-step compute.
• Frame-sink context — a few permanent anchor frames (attention sink) preserve long-range consistency.
• Efficient streaming inference — the above yields interactive frame rates (~20.7 FPS on H100).

Same engine, three additions: controllability (action conditioning — LongLive only takes text), multimodal entry (image/video init — LongLive only starts from a prompt), and deployment behavior (async streaming, KV-cache quantization, drift control).

LoRA learns ΔW = B·A with A (r×k) and B (d×r). r = 64 is the shared inner dimension — the matrix rank of the update, i.e. the capacity knob (8–16 light, 128+ heavy). It's called “rank” because the update is constrained to be low-rank, which is the efficiency trick.

Formulas. For a frozen weight W₀ ∈ ℝ^d×k, LoRA adds a low-rank update ΔW = (α/r)·B·A, with B ∈ ℝ^d×r, A ∈ ℝ^r×k, r ≪ min(d,k). Forward pass: h = W₀x + (α/r)·B(Ax).

Annotations. d = output dim, k = input dim; r = rank (=64 here) = how expressive the update is; α = scaling constant, so the effective update size is α/r (change r without re-tuning LR). Init A ~ small Gaussian, B = 0 → ΔW = 0 at start (training begins exactly at the base model). Trainable params fall from d·k to r·(d+k): a 4096×4096 layer at r=64 goes ~16.8M → ~0.52M (≈3%).

NitroGen (NVIDIA/MineDojo, arXiv 2601.02427): 40,000 h of internet gameplay across 1,000+ games, per-frame gamepad action labels; for BC policies (video→actions) and world models (actions→video).

GameGen-X / OGameData (ICLR 2025): first large open-world game-video dataset — ~1M video-text pairs from 150+ games, generation + control subsets; GameGen-X is a DiT with an InstructNet for control.

Games uniquely supply action-labeled, interactive, diverse, cheap, engine-clean data (real footage has no action stream). But it is not a faithful real-world simulation — it learns game dynamics, rendering, and priors, with a real domain gap. Honest framing: a controllable visual dynamics model of game worlds, not a physical-reality simulator.

Steering generation with a text prompt: the prompt is embedded (T5/CLIP) and injected via cross-attention so each denoising step is pulled toward the described content — i.e. modeling p(video | prompt) instead of unconditional p(video).

Matrix-Game 2.0 (Skywork AI, arXiv 2508.13009): frame-level keyboard/mouse action injection inside a few-step AR diffusion loop; ~1200 h of interaction-annotated data (Unreal Engine + GTA5); mouse → concat + MLP + temporal self-attn, keyboard → cross-attention; ~25 FPS. See comparison table in № 17.

DiT = Diffusion Transformer (transformer denoiser instead of a U-Net). LongLive's DiT is the Wan2.1-1.3B DiT made causal + frame-level AR with short-window attention, frame sink, and a KV cache.

Action-to-observation mapping = the learned dynamics p(o₍ₜ₊₁₎ | o₍≤ₜ₎, a₍ₜ₎): given past frames and the current action, predict the next frame. Controllability = quality of this mapping.

DA3 (ByteDance Seed, arXiv 2511.10647): depth + camera pose from any views, single plain transformer. Alternatives: VGGT, DUSt3R/MASt3R, MonST3R, COLMAP/SfM, MiDaS, Metric3D, UniDepth, Marigold, ZoeDepth, or engine ground-truth / Plücker camera embeddings. Metrics: pose (ATE, RPE), depth (AbsRel, δ<1.25), geometry (Chamfer).

Merge the adapter into the base weight: W_fused = W₀ + (α/r)·B·A, precomputed once per layer at load time. Inference then uses a single merged matrix of the original shape — zero extra runtime cost (vs. an extra low-rank multiply each step if left unfused).

Prompt switching over AR rollout: while generating frame-by-frame, the native way to change the world is to feed a new prompt and continue from the current state (no image-init slot). DF-World reuses the autoregressive history interface: encode an image/video into the same latent space and insert it as clean initial history — reasonable and natural.

Video VAE (not MAE), not an image encoder: rollout lives in the video VAE's latent space, so inserted history must be encoded by that same VAE to be compatible. (VAE = variational autoencoder; MAE = masked autoencoder — different.)

Latent history is a standard interface (cached latent/KV context). Live action stream example: per-frame controller inputs, e.g. t: {W, dx=+3, dy=0}, t+1: {W+A, dx=−2}, one entry consumed per generated frame.

Yes — general to context-conditioned AR generation: fixed initial conditioning decays, and long-term consistency then rests on the memory mechanism (KV cache / retrieval). It's mitigable with explicit memory (DreamX-World scene persistence, Matrix-Game 3.0 long-horizon memory), so it describes vanilla AR, not an inescapable law.

Encode = pixels→latent (input); Decode = latent→pixels (output).

Encode/decode modules: text encoder (T5/umT5/CLIP); VAE encoder; DiT patch-embed/tokenizer; VAE decoder; Wan-VAE (3D causal, both ways); TAE/TAESD (tiny distilled decoder); LightVAE/LightTAE (LightX2V optimized); pixel-shuffle / causal-3D-conv decoders.

KV-cache quantization: INT8/FP8 KV; INT4/2-bit (KIVI, KVQuant); per-token vs per-channel; group-wise; asymmetric/outlier-aware; keep attention-sink/recent tokens in higher precision.

LightX2V / ComfyUI-LightVAE / TAE: LightX2V = inference-acceleration framework (distilled models + optimized VAEs); ComfyUI-LightVAE wraps LightVAE/LightTAE nodes; TAE decoder = Tiny AutoEncoder, a ~0.5 GB distilled replacement for a full VAE for fast decode. (NVIDIA OmniDreams swaps in LightVAE/LightTAE for real-time world-model decode — same pattern.)

Canonical real-time path? Yes — DiT execution → action conditioning → VAE decoding → streaming overhead is the standard per-frame decomposition.

External geometry map necessary? No, but it helps — pose/depth supervision anchors geometry and reduces drift; a quality/consistency aid, not a hard requirement.

Components of a generative world model: (1) observation encoder/tokenizer (VAE); (2) dynamics/transition model (DiT / RSSM); (3) action conditioning; (4) memory/context (KV cache / retrieval); (5) decoder (VAE); (6) control/policy interface; (7) training objective (diffusion/flow/BC, sometimes RL). Classic Ha & Schmidhuber shorthand: Vision · Memory · Controller.

Highlighted row = the paper under study.

Per-frame loop: action+prompt → cache → causal DiT few-step denoise → latent → (append to cache) and (async VAE decode → display). Overlapping decode(t) with DiT(t+1) is what sustains interactive frame rates.

The whole clip is denoised jointly: at every one of the T denoising steps, a transformer with full attention lets each frame attend to all others, including future ones. That global view is why quality and long-range coherence are best-in-class — and why it cannot stream or be made interactive: you must process the entire sequence (future included) to produce any single frame, so latency is high, length is fixed, and there is no causal mask to KV-cache. Examples: Sora, Wan base, HunyuanVideo. In this whole area, the bidirectional model is the high-quality teacher that the real-time AR methods distill down from.

DiT = Diffusion Transformer: a transformer does the denoising instead of a U-Net. A spacetime patch is a small 3D block of the video latent spanning a few frames × a few latent pixels. The video is cut into these cubelets, each is embedded (patch + positional embedding) into a token, and the transformer denoises the resulting token sequence; “unpatchify” reassembles frames. “Spacetime” = the patches span both space and time. This is the trick Sora popularised: represent video as tokens so one scalable transformer handles it.

Dimensions. After the VAE, the video latent is T × h × w × c: T = number of latent frames (time), h,w = latent spatial height/width (each ~8× smaller than the pixel frame), c = latent channels. A spacetime patch spans p_t frames × p_h × p_w cells → one token, so #tokens ≈ (T/p_t)·(h/p_h)·(w/p_w).

Positional embedding. It is not a CLS token. CLS is a single extra "summary" token (BERT/ViT classification); a positional embedding is a vector added to every patch token encoding where the patch sits in space and time (its (t,i,j) index) — attention is otherwise order-blind and wouldn't know frame order or pixel location.

Real-time = the model produces frames at least as fast as they play back (≈ the display rate, e.g. ~24 FPS) with low latency, so you can watch — and interact — while it generates. Non-real-time (batch) = you wait for the whole clip (seconds to minutes) before seeing anything. Three things decide which side you land on: (1) causal vs bidirectional — can a frame be emitted without seeing the future? (2) steps per frame — multi-step diffusion vs few-step distilled; (3) KV-cache reuse. “Interactive” is a stricter bar than real-time: it also needs per-frame control and low enough latency that your input visibly changes the next frame (roughly ≤ ~100 ms/frame and ≥ playback rate).

Concrete bars (no single universal standard). The practical criterion is generation FPS ≥ playback FPS. Common targets: film ≈24 FPS, video ≈30 FPS, games 30–60 FPS. For interaction what matters is input→response latency: <~50 ms feels immediate, <~100 ms responsive, >~150 ms laggy (human motor-perception thresholds sit around 13–100 ms). So "real-time" here means ≥ playback rate and latency low enough that a person can't easily tell generation from playback. By that yardstick DF-World's 14–15 FPS on a 4090 is "interactive-ish", not yet smooth 30+.

Wan-VAE is the compression layer under the DiT. The encoder maps pixels → a much smaller latent (spatial + temporal compression); the decoder maps latent → pixels. It uses 3D causal convolutions — “causal” in time so a frame never sees the future, which is what makes streaming possible — and handles any-length 1080p while preserving temporal information. The model denoises in latent space; decoding is the last step of the streaming loop (see № 18).

Why 3D causal conv? "3D" convolves over time×height×width together, so it compresses spatially and temporally. "Causal" in time = each output sees only current + past frames, never the future — which (a) lets the first frame be encoded alone (unifying image & video) and (b) is exactly what a streaming/AR model needs. Conv is also cheaper than attention on long video.

Have others replaced it? Yes: transformer tokenizers (OmniTokenizer, C-ViViT) — flexible but need positional embeddings and struggle with unseen resolutions; Mamba/SSM tokenizers (Cosmos MambaVideo) for long-sequence efficiency; factorized 2D-spatial + 1D-temporal causal conv and group causal conv (IV-VAE) for speed; and quantizer swaps (LFQ in MAGVIT-v2, FSQ in Cosmos). Still, causal 3D CNN (MAGVIT-v2 → CogVideoX, HunyuanVideo, Step-Video, Wan-VAE) remains dominant for efficiency + clean image/video unification.

Flow matching is a training objective (an alternative to score-based diffusion). Instead of learning to denoise, it learns a velocity field vθ(x,t) that transports a simple noise distribution to the data distribution along a nearly straight ODE path; sampling just integrates that ODE. Straighter paths need fewer steps, so it is faster — which is why Wan2.1 and most recent video models train this way.

Score-based diffusion (detail). A forward SDE gradually adds noise: dx = f(x,t)dt + g(t)dW. The model learns the score sθ(x,t) ≈ ∇_x log p_t(x) — the gradient of the log-density of the noisy data — and generation runs the process backward.

ODE / PF-ODE. ODE = ordinary differential equation. Every diffusion SDE has a matching deterministic probability-flow ODE with the same marginals: dx = [f(x,t) − ½g(t)²∇_xlog p_t(x)]dt. Integrating it from noise gives a sample deterministically (same noise → same output), which is what makes it distillable into few steps. The "PF-ODE trajectories" in № 24 are the paths CausVid samples from the teacher to regress the student onto. Flow matching and score-based both become noise→data ODEs; flow matching just learns the (straighter) velocity directly → fewer steps.

(1) ODE initialization. Sample PF-ODE trajectories from the frozen bidirectional teacher and train the causal AR student by regression to match them. This just gives the student a competent starting point that already behaves causally.

(2) Asymmetric DMD (Distribution Matching Distillation). The teacher and the critic stay bidirectional, while the student is the causal few-step model; the student is trained so its output distribution matches the teacher's (an approximate reverse-KL expressed as the difference of two score functions), under diffusion forcing. “Asymmetric” = teacher bidirectional, student causal.

Net effect: distill a slow 50-step bidirectional model into a fast ~4-step causal one. Later work patches each stage — Self-Forcing fixes stage 2's train/test gap (№ 26), Causal Forcing fixes stage 1's teacher mismatch.

Expanded — two weak points, one fix each:

• Stage 1 → Causal Forcing. The student is initialised by regressing to a bidirectional teacher's ODE trajectories, but the student is causal — the two don't line up (a bidirectional path isn't reproducible frame-by-frame; it "violates frame-level injectivity"). Fix: initialise from a causal teacher, so the target trajectories are ones a causal student can actually follow.
• Stage 2 → Self-Forcing. In asymmetric DMD, CausVid trains the student on ground-truth past frames, but at test time it conditions on its own outputs — training never matched inference (the exposure-bias gap of № 26). Fix: autoregressive self-rollout during training, so the student always conditions on its own generated history.

Same CausVid skeleton: Causal Forcing repairs the start (init), Self-Forcing repairs the middle (the DMD regime).

A KV cache stores past keys/values so each new frame reuses history instead of recomputing it — only possible with causal attention. It is not unique to CausVid; it is the shared enabling trick of essentially every real-time AR method: CausVid (key to its 9.4 FPS), Self-Forcing (rolling KV cache, even during training), LongLive (KV cache + KV-recache for prompt switching), Matrix-Game 2.0, MAGI-1 (block-causal, chunk-level caching), and the Causal-/Rolling-Forcing line. What CausVid was first at is combining causal KV-cached AR with few-step distillation so quality matched bidirectional models at interactive speed. The methods that can't KV-cache are the bidirectional ones (Sora, Wan base, HunyuanVideo) — full attention includes future frames, so there is no causal prefix to cache.

What prompt switching does. Yes — it is semantic steering of the ongoing world: mid-rollout you feed a new text prompt (e.g. "sunny street" → "the same street at night in the rain", or "now a dragon appears") and generation continues from the current state under the new meaning. It varies the observations/content being generated, not the low-level action interface (movement still comes from the action stream). KV-recache makes the switch clean by recomputing the cached keys/values so the old prompt stops leaking in while temporal continuity is kept. In DF-World it is one interactive control alongside the action stream and image/video init.

Yes, exactly that contrast. Teacher forcing (the supervised default): during training the model always conditions on ground-truth past frames. The flaw is exposure bias — at inference it must instead condition on its own imperfect outputs, a regime it never practiced, so errors compound and the video drifts. Self-rollout (Self-Forcing): during training the model generates the sequence autoregressively on its own previous outputs (with KV caching) and is supervised by a holistic, video-level loss — so training now matches inference. “Self” = conditioned on self-generated history rather than ground truth; it is the on-policy fix to the supervised/teacher-forced train-test mismatch.

SOTA is per-paper and time-sensitive (each team reports on its own setup) — treat ticks as “strong as reported”, not a settled leaderboard. Interactive world models are shifting evaluation from pure video quality (VBench) toward action-control + memory + physics.

Recent surveys / living lists:

• Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends (Liu et al., 2026) — the most on-point.
• A pedagogical walk-through: “From Teacher Forcing to Self-Forcing++” (tutorial, 2026).
• Living lists: Awesome-Interactive-World-Model, Awesome-From-Video-Generation-to-World-Model, Awesome-World-Models.

Where the attention is right now:

• Real-time via few-step AR distillation — the CausVid → Self-Forcing → Causal Forcing → Rolling-Forcing / Self-Forcing++ line.
• Long-horizon consistency & memory — the central open problem: frame sink, rolling KV cache, explicit memory/retrieval, scene persistence.
• Action controllability & unified action spaces — keyboard/mouse → shared camera representation (Matrix-Game, GameCraft).
• Multimodal / instruction control — text + image + action together (GameCraft-2, DF-World).
• Geometry grounding — pose/depth supervision (DA3) and SLAM-based metrics (WorldScore) to fight drift.
• Low-compute / consumer-GPU — LongLive, DF-World, LightX2V, KV-cache quantization & compression.
• Physics fidelity & better benchmarks — Physics-IQ, WorldModelBench, MIND, WBench.

The five competing axes are quality ↔ latency ↔ controllability ↔ long-horizon consistency ↔ compute. No family maxes all of them — each picks a corner. DF-World deliberately picks the frame-causal AR + action corner, trading frontier quality for real-time control on a consumer GPU.

Why this matters here: DF-World uses a VAE (Wan-VAE) to move between pixels and latents. The earlier "video-VAE not image-MAE" point (№ 14) was exactly this — a generative codec vs a representation-learning autoencoder.

The loop: an observation (frame) is encoded to a latent (1), the dynamics model predicts the next latent conditioned on action (3) and memory (4), the decoder renders it back to pixels (5), and the control/policy (6) supplies the next action — then it repeats. Ha & Schmidhuber's shorthand maps on cleanly: Vision (encode), Memory (dynamics + context), Controller (act). The training objective (diffusion / flow / behaviour cloning, sometimes RL) sits on the dynamics model.

Reconstructed schematic — the paper doesn't publish a full pipeline figure; this is inferred from its stated components. Three input paths stay separate and fuse inside the DiT: text via cross-attention (semantics), image/video encoded to a latent prefix in the history (starting state), actions via the residual pathway (control). The core is the reused LongLive causal DiT; output decodes async and the latent is appended to the cache for the next step.

How many / how much. StreamingLLM keeps only ~4 initial tokens as sinks (1–2 are insufficient, 4 suffice, more is marginal), and those first 4 tokens capture >40% of all attention.

What info they hold. Almost none semantically — sinks get high attention but tiny value norms ("value-state drain"): an attention "parking lot", not an information store.

Why the first tokens. Softmax forces attention to sum to 1, and under causal attention the first tokens are visible to everyone — so a head with nothing useful to attend to parks its excess attention there (a no-op outlet). Linked to "massive activations" (the first token's unusually large hidden-state norm).

Compression takeaway. Your instinct is right in practice but for a flipped reason: standard KV-cache compression/quantization keeps sinks (and the recent window) in higher precision and compresses the middle harder — not because sinks are information-rich, but because they are structurally load-bearing (evicting them breaks the model). The useful information lives in the recent window + a few important middle tokens. (A model can even be trained with one dedicated learnable sink token to replace the four.)

Between memory / compute budget & latency and long-horizon fidelity (recall accuracy, scene persistence / coherence, robustness to drift). More memory or retrieval → better long-range consistency but more compute, latency, and complexity, plus a retrieval-precision problem (fetch the right thing, not noise). A bounded cache → cheap and fast but forgets. In short: budget / speed ↔ long-range coherence / reliability.

Yes — it is prefix conditioning / prefill, the same idea as prefilling an LLM's KV cache with a prompt. Encode the image/video → latents → seed the history so autoregressive generation continues from that context instead of from nothing. Image-init = a latent prefix that supplies the starting state (see № 14, № 32).

NitroGen provides (frame, action) pairs, and the same action-labeled data trains both directions: video→actions = a policy / inverse-dynamics model (behavior cloning: given frames, predict the action taken), and actions→video = a world model (given action + past frames, predict the next frame). One dataset, two uses.

CLIP's text encoder is contrastively trained to compress a caption into a single pooled vector aligned with a whole image, with a short context (~77 tokens) and caption-style data — so multi-object, relational, counted, long-clause structure gets flattened. T5 emits a per-token sequence, preserving that compositional detail.

Combine? Yes — and the field does. SDXL uses two CLIP encoders; Imagen used T5; eDiff-I used T5 + CLIP together — CLIP for global alignment, T5 for fine detail. Combining is standard, not a conflict.

Discriminative learns p(y|x) — a label/decision given an input (classifiers, regressors; decision boundaries: "is this a cat?"). Generative learns p(x) or p(x,y) — the data distribution itself, so it can sample new data ("draw a cat"). Diffusion / DiT are generative (p(video), or conditioned p(video|prompt)); a CLIP classification head is discriminative. Conditional generation is generative-but-conditioned — trained on paired data like supervised learning, yet it produces samples, not labels.

T = number of latent frames (time); h, w = latent spatial height/width (each ~8× smaller than the pixel frame after VAE downsampling); c = latent channels (feature depth per cell). The full rollout latent is T×h×w×c; each autoregressive step emits one latent frame of shape h×w×c, and T grows by one (or a small chunk) per step — so h×w×c is "per frame", T indexes the whole sequence.

In RL terms, the policy is the student model itself — the distribution it induces over the next frame, given its own prior observations and current weights. On-policy = train on data produced by that current policy. Teacher forcing conditions on ground-truth past → off-policy (data from the true distribution, not the model's own rollouts) → the train/test mismatch (exposure bias). Self-Forcing does autoregressive self-rollout during training — the student conditions on its own generated frames — so the training distribution equals the inference distribution → genuinely on-policy. So yes: the policy is decided by the student, determined by the model conditioned on its own prior (self-generated) observations. (An analogy — this is distribution-matching distillation, not literal RL — but it maps cleanly.)

LUMOS (Language-Model Unified Machine-Readable Operating-System Semantics), Yogeswar Reddy Thota, UT Dallas.

Problem. OS interfaces are built for humans (pixels, icons, windows, mouse, shortcuts), but agents need compact semantic state, grounded actions, reliable feedback. Forcing agents onto screenshots / OCR / crops brings high token cost, visual ambiguity, latency, and coordinate uncertainty.

What it does. Inserts a semantic layer between agent and OS: converts native accessibility metadata + browser UI structures into machine-readable semantic blueprints (stable IDs, roles, names, values, bounds, action affordances); adds live pointer grounding via OS automation APIs; the LLM then acts through an accessibility-grounded observe–act loop using constrained visible-UI primitives, not app-specific scripts. It explicitly does not claim to replace vision agents — it reduces screenshot reliance when the OS already exposes semantics.

Core idea. Change the interface representation handed to the agent from human-friendly (pixels it must reverse-engineer) to agent-friendly (semantic structure the OS already holds). A shift in observation modality, not the model — same LLM, different "eyes".

Claimed effects (honest read). Efficiency + reliability — fewer tokens (blueprint vs multi-thousand-token screenshot), precise element grounding (act on A7, not a fragile (x,y)), lower latency/ambiguity — not a task-success leaderboard win. Hedged by "when semantic APIs are available", so it degrades on canvas / games / custom-drawn / some Electron UIs. Measured token/latency/success gains would live in the experiments (not in the abstract).

Start from a pixel (x, y) — where the cursor is, or where the agent wants to act. The vision path crops a screenshot and runs OCR/vision to guess. LUMOS instead hit-tests the OS's element tree (the OS already stores every element's bounds, role, name, value) to find the element whose bounds contain (x, y), and returns the actual element.

Per platform:

• Windows (UI Automation): IUIAutomation::ElementFromPoint(pt) → an element; read CurrentName, CurrentControlType, CurrentValue (ValuePattern), BoundingRectangle, AutomationId, states. (Lower level: WindowFromPoint, MSAA AccessibleObjectFromPoint.)
• macOS (AX): AXUIElementCopyElementAtPosition(x,y) → element; then AXRole, AXTitle, AXValue, AXPosition, AXSize.
• Browser (DOM): document.elementFromPoint(x,y) + ARIA role/name + getBoundingClientRect().

Output = a grounded element with a stable ID (the A7/W4 in Fig 3) plus role/name/value/bounds — so the agent acts "invoke A7" rather than "click (x, y)". Robust to DPI scaling, window moves, and layout shifts (which break raw coordinates), and it re-grounds live as the cursor moves.

Flip the framing. It is not that a screenshot contains the semantics; the semantics the agent needs (what's a button, its label, value, enabled state, position) already exist in the OS, because the OS rendered the UI from that structured data. A screenshot is a lossy projection of that structure into pixels.

So the vision path is a wasteful round-trip: structure → pixels (render) → hard inference (OCR/detect/guess) → structure (recovered, imperfect). LUMOS skips the pixel detour and reads the original structure the OS still holds. "When the OS exposes semantics" = when the accessibility tree is populated — and that "when" is load-bearing: canvas / games / some Electron apps don't expose it, so vision is still required there. Screenshots = a degraded encoding; the OS = the original.

All three are pre-existing, screen-reader-oriented APIs exposing the same kind of thing: a tree of elements with roles, names, values, bounds, and available actions, built so assistive tech can understand a UI without pixels.

• Windows UI Automation (UIA): Microsoft's accessibility framework (successor to MSAA). Apps expose a tree of AutomationElements with properties (ControlType, Name, AutomationId, BoundingRectangle, Value, states) and control patterns — Invoke, Toggle, Value, Selection, ExpandCollapse, Scroll, Text, Grid — describing what you can do. Powers Narrator / NVDA / JAWS; gives ElementFromPoint, focus queries, tree walking, events.
• macOS Accessibility (AX / NSAccessibility): the protocol behind VoiceOver. Apps expose AXUIElements with attributes (AXRole, AXTitle, AXValue, AXPosition, AXSize, AXEnabled, children) and actions (AXPress, AXIncrement, AXShowMenu). Read via AXUIElementCopyAttributeValue, act via AXUIElementPerformAction; needs accessibility permission.
• Browser DOM / ARIA: the DOM is the element tree; the browser computes an accessibility tree from the DOM + native HTML semantics (<button>, <input>) + ARIA attributes (role, aria-label, aria-checked…). This is what screen readers consume for web pages and what Playwright / Chrome DevTools Protocol expose as an "accessibility snapshot"; document.elementFromPoint is its hit-test.

Yes — a bridge / adapter / normalization layer. It doesn't invent the semantics (the accessibility trees already carry them). What LUMOS adds:

• Reads from three heterogeneous sources (UIA / AX / DOM-ARIA) and normalizes them into one uniform representation.
• Compacts them into a single semantic blueprint (stable IDs, roles, names, values, bounds, affordances) small enough for an LLM to consume cheaply (vs a multi-thousand-token screenshot).
• Assigns stable IDs (A7/W4) so the agent can refer to elements durably across steps.
• Exposes a constrained action vocabulary (visible-UI primitives) that maps back down to accessibility patterns/actions for execution.

So it's bidirectional: OS accessibility → agent (observation), and agent action → OS accessibility action (execution). The neat irony: accessibility APIs were originally a bridge built for screen readers; LUMOS repurposes that same bridge for AI agents. (Mental model: an ORM bridging objects↔database, or an API gateway normalizing many backends into one interface.)

Core move. Instead of judging a dense-supervision signal by training with it (expensive, confounded by training engineering), QVAL judges it directly and training-free: does its scores rank actions the way a near-optimal policy's Q-values do?

Common / requirements? It is the default RL framework. Its assumptions are idealisations: Markov state, full observability (else it's a POMDP), stationary transitions/rewards, a scalar reward, discrete time. Real problems often violate these (partial observability, history-dependence → you need memory).

A world model is essentially the learned observation-transition (the "T") of a POMDP — the environment half only. Because observations are partial (not Markov states), it must condition on history — which is exactly why it needs frame sink / KV cache (№ 04, № 33). QVAL instead works inside a clean MDP and uses Q^π as ground-truth labels.

Ordering, not magnitude. QVAL grades a method by whether its ranking of (s,a) pairs matches the ranking by Q^π. Absolute values don't matter; only order does.

"Label with, not train with". π is the oracle that supplies labels — exactly like ground-truth labels in supervised learning. Q^π(s,a) annotates each decision point with a reference value, and a method is graded on reproducing that ordering. The policy you eventually train with the signal is a different thing. You want π near-optimal, because Q^π = "expected return if you take a, then continue with π" — if π continued badly, even a genuinely good action could get a low Q^π (a bad label). A strong π makes "high Q^π" mean "genuinely good action".

Formula (2): k(s,a) = φ(Q^π(s,a)) for some strictly increasing φ. This defines "Q-aligned": a signal k is aligned if it is a monotonic transform of the reference Q — it may stretch/squash the values however it likes, but must preserve their order. A perfectly aligned signal ranks every decision point exactly as Q^π does.

Computation (you do NOT fit φ): (a) compute reference labels Q^π(s,a) over many pairs via rollouts of the near-optimal π; (b) compute the method's scores k(s,a); (c) measure the rank correlation between the two. Perfect monotonic agreement ⇒ alignment 1. The "for some strictly increasing φ" is precisely what makes rank correlation the right tool (QV5).

A dense supervision signal is useful for training only if it tells the policy, at each step, which action is better. Q^π (with near-optimal π) is the ground truth of "which action is better". So if a method's scores order actions like Q^π, the signal carries the right information → likely useful downstream. It is cheap because you measure this alignment without ever running the training pipeline. The "π close to optimal" clause is load-bearing: it's what makes Q^π a trustworthy target.

One correction on phrasing: QVAL is not scoring an agent's task ability. It scores the supervision signal / method — how good a dense-reward method is — by checking whether it ranks actions like a near-optimal Q. The object under evaluation is the scorer, not the trained agent.

The operationalisation of QV2–QV3: take the method's scores k(s,a) and the reference labels Q^π(s,a) over a set of decision points and compute a rank correlation. High → the signal orders decisions like the near-optimal Q → good; low → it lacks the ordering information and would need other mechanisms to help training.

• Spearman's ρ (1904): Pearson correlation computed on the ranks instead of raw values. Measures monotonic association, ranges [−1, 1] (ρ=1 = perfectly monotonic increasing), and is invariant to any strictly increasing transform of either variable — exactly the φ in formula (2), which is why it's the natural main metric. Standard for meta-evaluating an automatic scorer against reference judgments.
• Kendall's τ (1938): based on concordant vs discordant pairs — τ = (C − D)/(total pairs). Interpretable as ~ P(a random pair ordered correctly) − P(incorrectly); ranges [−1, 1]; the τ-b variant handles ties. More conservative (usually smaller than ρ) and more robust to outliers — reporting both is common.
• Calibration point. A scorer (e.g. an LLM/VLM judge) can have poorly calibrated absolute numbers yet still order candidates correctly. Rank metrics capture exactly that useful part and ignore the miscalibration — matching the "aligned up to a monotonic φ" definition. QVAL deliberately measures ordering fidelity, not value accuracy.

The whole method is internally consistent: alignment is defined only up to a strictly increasing φ (2), so the metric must be invariant to monotonic transforms — and Spearman/Kendall are exactly that. Everything is about rank, not magnitude.

Fig 1 (teaser): a scene with dynamic objects, occluders, a camera trajectory that revisits earlier viewpoints, and the generated key frames — the hard case rule-based retrieval fails on.

Problem. Video world models predict future frames from actions + history, but lack memory, so extended rollouts drift and scenes become inconsistent. Prior fixes retrieve past context by hand-crafted rules, which break under occlusion and moving objects.

Proposal. MemLearner makes the network learn to adaptively query historical frames end-to-end, via query tokens (Q) that bridge context tokens (C) and predicted tokens (P). It reuses the pre-trained video generation model itself for querying (no scratch module), plus efficiency tricks.

Does "video world model" cover DreamForge? Yes — same family. A generative world model (DreamForge) that generates observations as video is a video world model. MemLearner tackles the memory sub-problem of exactly this family; DreamForge’s frame-sink/KV-cache is one (simpler) memory scheme, MemLearner’s learned query is another.

Fig 2: (a) C/Q/P interaction — Q extracts info from C, P conditions on Q; (b) a separate context-query module (fails); (c) the adopted design that queries via the video generation model itself.

Yes, the query token is the heart of it. Q tokens are learnable slots that attend to C to pull out the relevant history, and P attends to Q as its generation condition. So Q is an information bridge: C → Q → P. This replaces "pick frames by a rule" with "learn what to read."

Why the separate module fails. Design (b) trains a fresh context-query module from scratch and bolts it onto a pre-trained video DiT. In Table 2 it collapses (PSNR 9.16 vs 21.23 for the adopted design). The paper’s reading: jointly training a scratch module with a pre-trained DiT is ineffective — a feature-space / prior-alignment problem. The scratch module’s features don’t live in the same representation the frozen-ish DiT expects, and there’s no pre-trained prior to guide querying, so the two never align. Design (c) avoids this by doing the querying inside the DiT, reusing its visual prior. This is a recurring lesson in the field (add-a-module-from-scratch often underperforms reusing the backbone), not a one-off bug.

Fig 3: a DiT with Patchify → 2D-attention → 3D-attention (+ optional camera encoder) → cross-attention → FFN → Unpatchify. C/Q tokens sit in History Context; N (noised P) + camera condition in Current Output.

Is the left→right token order "real"? It’s a logical layout, not spatial memory addresses. C (context) and Q (query) are the history side; N (the noised P) and the optional camera condition are the current-output side. They are concatenated along the frame dimension and fed jointly — so "history first, current after" reflects the concatenation order, and causal/3D attention lets current tokens read history.

Why 2D → 3D → cross? A cost/scope ordering: 2D (spatial) attention first refines each frame internally (cheap, per-frame); 3D (spatiotemporal) attention then mixes across frames/time (where C/Q/P actually interact); cross-attention injects external conditioning (text/camera) last; FFN transforms. Camera features are added between 2D and 3D so trajectory guidance is present before temporal mixing.

"added to z" = residual injection? Yes — the camera encoder’s output is added to the intermediate features (a residual/additive conditioning path), not concatenated as extra tokens. It nudges the trajectory without changing token count.

Camera pose cam = [R,t] ∈ ℝ^f×(3×4). f = number of frames; for each frame a pose is a 3×4 matrix = R (3×3 rotation) concatenated with t (3×1 translation) — the extrinsic camera matrix. So the whole tensor is f frames, each a 3×4 pose.

Eq 1 (loss): L(θ) = E[ ‖εθ(Z_t, cam, p, t)‖ − ε^P ]. It’s a diffusion denoising loss: the network εθ predicts the noise added to the noisy tokens Z_t (given camera cam, prompt p, timestep t), and is trained to match the true sampled noise ε^P ~ N(0,I). Key detail: supervision is applied only to the predicted (P) tokens — C and Q stay unperturbed; only P is noised at the input, so the loss lives on P’s noise.

Eq 2 (standard 3D attention): F_out = F_in + o( sm( q(F_in)·k(F_in)^T )·v(F_in) ), abbreviated F_in + g(F_in,F_in,F_in). This is ordinary self-attention with a residual:

• q,k,v are linear projections producing queries/keys/values; sm = softmax.
• Why v last: sm(qk^T) first computes the attention weights (how much each token attends to each other), then those weights are applied to the values — you weight what you retrieve (v) by how relevant it is (qk^T). Order matters: weights × values.
• o(·): the output projection — a final linear map on the attended result before the residual add. g(·,·,·) is just shorthand for "(query-input, key-input, value-input)".

This full version is expensive because context frames make F_in huge — motivating Strategies 1&2 (M5–M6), which rewrite g’s inputs (Eq 3/4).

Fig 4: (a) shallow Query Layers (≤5) process C/Q/P; deep Generative Layers (dozens) process only Q/P. (b) attention pruning — keep only the three needed directions.

Strategy 1. Split the n+m DiT layers into n shallow Query Layers (C, Q, P interact) and m deep Generative Layers (only Q, P), with n ≪ m. Rationale: querying is like encoding — extracting info needs fewer params/layers than generating (video VAEs ≪ video generators). So context reading is done early and cheaply, then dropped.

Prior evidence? Yes, indirect but real: it’s well established that early transformer layers do more low-level/encoding-style work and later layers do higher-level synthesis; encoders are far smaller than generators; and their own ablations (Sec 5.5) validate that shallow-only querying suffices. Your intuition that "shallow layers are semantically abundant" is close — more precisely, shallow layers carry the local/contextual detail useful for matching to history, which is exactly what querying needs.

Eq 3 vs Eq 4. Both use the residual attention g(query, key-set, value-set) from Eq 2. In Query Layers (Eq 3): C_out=C (context is never updated — C is read-only), Q_out=Q+g(Q,{C,P},{C,P}) (Q reads from both C and P), P_out=P+g(P,{P,Q},{P,Q}) (P reads from P and Q). In Generative Layers (Eq 4): C is removed, so Q_out=Q+g(Q,P,P) (Q now only sees P) and P_out=P+g(P,{P,Q},{P,Q}) (unchanged). The single difference: Eq 4 drops C entirely — once the shallow layers have distilled context into Q, the deep layers no longer touch the (huge, expensive) context tokens.

Standard 3D attention (Eq 2) lets every token type query every other — but many directions are useless. Strategy 2 keeps only three: (1) Q→P (Q learns what to extract given the target), (2) Q→C (Q extracts from context), (3) P→{P,Q} (P reads from itself + Q). Everything else — especially C as a query — is dropped, since we never need context to attend outward. That prunes the most expensive part (C is the long history).

Soundness: the pruning is directional, not temporal — it’s not the same as a causal mask, but shares the spirit of "don’t compute attention you’ll never use." Because C dominates the token count, cutting C-as-query is where the savings come from.

Context retrieval is the attractive one: no reconstruction or compression, so no added cost/error — but you must pick which past frames to use. Prior work does that with rules:

• FOV overlap — Field-Of-View overlap: retrieve past frames whose camera frustum overlaps the current view (geometric heuristic).
• Point-cloud estimation — estimate a 3D point cloud from frames and match current vs past points.
• Surfel matching — represent surfaces as surfels (oriented disks) and match them across time to find revisited geometry.

Occlusion = when a nearer object blocks a farther one from the camera’s view (e.g. a pillar hiding a hut). It breaks geometric heuristics because two frames can share the same FOV/geometry yet show different content (something stepped in front), and dynamic objects move between visits.

Rule-based vs learning-based — deeper reasons. Rule-based retrieval assumes a static, purely geometric world: "same viewpoint ⇒ same content." That assumption fails exactly under (a) occlusion (geometry matches but appearance doesn’t) and (b) dynamic objects (content changed since last visit). The rules also can’t weigh semantic relevance or adapt per scene — they’re fixed. A learned query optimizes retrieval for the generation objective itself, can use appearance + semantics (not just geometry), and adapts across scenes — so it generalizes where rules are brittle.

Table 1: no existing long-video dataset satisfies all four requirements at once; MemLearner collects a 16.7h rendered dataset that does.

Is camera especially important? In this sub-area, yes — camera pose is treated as first-class because memory is fundamentally about viewpoint: "have I seen this place/direction before?" Revisit + camera trajectory is the entire memory stress-test. That said, the paper stresses camera is used only to guide the trajectory; the query method itself does not depend on poses (Sec 5.5). So camera matters for data/control, but the learned memory is pose-free — a deliberate decoupling (see M3, and the interactive-control note below).

Table 2: MemLearner ("Ours") leads all quality metrics; note the scratch-module Fig 2(b) row collapses (PSNR 9.16). fps here is generation throughput, not the quality axis.

Two different axes:

• Memory / reconstruction fidelity — PSNR↑ (pixel accuracy) and LPIPS↓ (perceptual distance) compare the generated frame to a ground-truth frame. On revisit splits these measure whether the model remembered the scene correctly.
• Visual quality / realism — FID↓ (image realism vs the real distribution) and FVD↓ (video realism + temporal coherence) measure how plausible the output looks, not whether it matches a specific target.

Relation to latency metrics (FPS / motion-to-photon). Orthogonal but coupled by a trade-off. PSNR/LPIPS/FID/FVD are quality axes (is it right / does it look real); FPS + latency are speed axes (how fast per frame). They interact: adding memory (more context tokens, querying) improves PSNR/FID but costs compute → lowers fps (note VMem/CaM/Ours sit below the memory-less DFoT on fps). The efficiency strategies (M5–M6) exist precisely to buy back speed without giving up the quality the memory provides. So: quality metrics say whether the memory works; latency metrics say whether you can afford it in real time — and the whole design is navigating that Pareto front.

Denoising stages = diffusion timesteps within generating one frame. Diffusion denoises from pure noise to a clean latent over many steps: early steps (high noise) fix global layout/structure; late steps (low noise) fill in fine texture/detail. The paper observes query tokens attend differently across these steps: early timesteps attend broadly to context (get the scene/structure right), later timesteps focus on fine-grained local correspondences (align exact textures/edges to remembered detail). That’s why a static/rule-based retrieval is suboptimal — the useful context changes within a single frame’s generation.

Careful with the mapping. This is about diffusion timesteps (a temporal axis of the sampling process), not network depth. So it does not directly say "shallow layers = context, deep layers = abstract." The depth story is Strategy 1 (shallow layers query, deep layers generate). Both are true but they’re different axes: timestep (coarse→fine over denoising) vs layer depth (encode→generate). Conflating them is a common slip — keep them separate.

Shared idea: learnable query tokens that aggregate features. Both take a big set of visual features and a small, fixed set of learned query vectors, and use cross-attention so the queries "pull out" a compact summary for a downstream model. MemLearner’s Q tokens are the same trick, applied to querying history.

• Perceiver Resampler (Flamingo): a fixed number of latent queries cross-attend to variable-length visual features → a fixed-size set of tokens for the LM. "Resampler" = variable→fixed length.
• Q-Former (BLIP-2): a Querying Transformer — learnable query tokens cross-attend to a frozen image encoder’s features, bridging vision→LLM. Trained with contrastive + generative objectives.
• How to tell them apart: both use learnable queries + cross-attention; Q-Former additionally does vision-language pre-training objectives and self-attention among queries, while the Perceiver Resampler is a lighter latent-bottleneck resampler. If it’s "a few latents cross-attending to features to fix the length," it’s Perceiver-style; if it’s "a BERT-like query transformer pre-trained to align image&text," it’s Q-Former.

RAG = Retrieval-Augmented Generation — fetch relevant external documents and feed them to a generator. Same spirit as context retrieval here (bring back relevant memory to condition generation), but RAG retrieves from an external text corpus via embedding search, whereas MemLearner retrieves from its own visual history via learned attention — retrieval is inside the model, not a separate database lookup.

Judging "corresponding parts" in a KV-cache alignment. The alignment question is the same cross-attention logic: correspondence is decided by attention weights = softmax(q·k^T) — a query slot corresponds to the cached key it scores highest. To inspect it, look at which cached K (past frame/token) each query attends to most; high attention = the "matched" part. That’s exactly what MemLearner’s Q tokens learn to do over the KV of history.

It resolves a mismatch: the vision side emits a variable, often huge number of feature tokens, but the downstream model wants a small, fixed, cheap input. K learned queries cross-attending over N features turn "N features" into "K tokens" for any N. Reasons, in order:

• Variable → fixed length. Different resolutions / frame counts give different N. Cross-attention's softmax is over the N keys, so any N works, and the output is always exactly K vectors — a stable interface decoupled from input size.
• Huge → small (cost). An image is hundreds–thousands of tokens, a video far more; attention is O(N²) and context is expensive. Compressing to a small fixed K (e.g. 32–256) makes downstream cost constant and cheap, independent of input size.
• Learnable queries ≠ pooling. Average/max pooling also gives fixed size but is content-agnostic (discards info uniformly). Learned queries are trainable "slots" that specialize (via cross-attention) in gathering specific aspects — objects, layout, motion — so the model learns what to keep.
• A learned information bottleneck. Forcing everything through K latents forces distillation of the most task-relevant information, trained end-to-end against the downstream objective.

Why it matters here: MemLearner's Q tokens use exactly this trick (M2, M11) — the history/context is variable-length and enormous, so a fixed, small set of learned queries cross-attends over it to produce a compact, adaptive condition for generation, instead of feeding the whole context into the deep generative layers. Same trick, same payoff: bound the cost, learn the summary, stabilise the interface.

VLA models drive instruction-following robot manipulation, but they inherit oversized language backbones from pretrained VLMs — far more capacity than short robot instructions require. The basic question: how much of a VLA is actually necessary for closed-loop control?

The paper studies architectural redundancy via transformer block removal as a controlled intervention. It introduces Drop-Then-Recovery (DTR) — remove selected blocks, fine-tune the result, and measure whether the removed capacity was needed — and GateProbe, a one-shot virtual-gate sensitivity metric ranking blocks by their contribution to the downstream action loss.

Across multiple VLA architectures, manipulation benchmarks, and real-robot industrial scenarios, they find a strong asymmetry in post-removal recoverability: language backbones are highly redundant for standard manipulation, while vision and action pathways are much less tolerant to removal. On LIBERO, removing half the LLM blocks even improves OpenVLA-OFT from 95.0% to 98.3% under the same fine-tuning budget, and keeping only two language blocks still recovers baseline. Implication: current benchmarks may exert limited pressure on deep language grounding / compositional understanding, so future VLAs should allocate capacity more deliberately across language, vision, and action. Code: github.com/s1ghhh/VLADrop.

Drop-Then-Recovery (DTR) is an analysis protocol: remove selected transformer blocks from a pretrained VLA, then fine-tune the reduced model on the downstream task, and check whether it recovers baseline performance.

The logic: if performance recovers after removal, the deleted capacity was redundant (you could retrain around it); if it fails to recover, that capacity was genuinely necessary. This "ablation-by-surgery + recovery test" separates capacity you can retrain around from capacity you truly need — a cleaner measure than removal alone (which conflates "important" with "hard to retrain").

To decide which blocks to drop without brute-forcing every subset, GateProbe inserts a virtual (scalar) gate on each block and measures, in one shot, how sensitive the downstream action loss is to closing that gate (roughly ∂action-loss/∂gate). Blocks are then ranked by contribution: low sensitivity → safe to remove; high sensitivity → keep.

It’s a cheap, one-shot proxy for importance — the same spirit as QVAL’s cheap-proxy-before-training idea (QV3): score components first, commit to the expensive fine-tuning later. This is what makes DTR "reliable" — you remove the right blocks rather than random ones.

Strong asymmetry in recoverability: language backbones are highly redundant for standard manipulation — you can remove many LLM blocks and fine-tuning recovers (or even improves) — whereas vision and action pathways are substantially less tolerant: removing them hurts and doesn’t recover.

Headline numbers (LIBERO): removing half the LLM blocks improves OpenVLA-OFT from 95.0% → 98.3% under the same fine-tuning budget; retaining only two language blocks still recovers baseline-level performance.

Interpretation: if most of the language stack is disposable, current VLA benchmarks probably don’t stress deep language grounding or compositional instruction understanding (instructions are short/simple, so the huge LLM is overkill). Takeaway: future VLAs should allocate capacity deliberately across language / vision / action rather than inheriting a giant LM wholesale.