Data Composition vs Architecture in Robot Imitation Learning

Inference 1. Marked positions are a trap.

Training only on the 64 known-position demos (DX-K) lets a policy nail the marked positions but collapses on random ones. Both ACT and SmolVLA land at exactly 10/20 on ST1 from D1-K — the score a policy gets by perfectly handling the 8 known trials and missing all 8 random ones. The pattern holds across architectures, so the failure is in the training distribution, not the model class.

Inference 2. Single-task training does not compose to multi-object (ST2).

Why we expected it to: humans who can pick one sponge can trivially pick several — each additional sponge is the same skill, repeated. Large language models show analogous behavior: a model that learns to add two numbers usually generalizes to adding three. So a reasonable prior is that a policy fluent in single-sponge picking should compose into multi-sponge picking with little or no additional data.

What actually happened: a policy trained on all 128 single-sponge demos (D1-Combined) does not stitch together a multi-sponge rollout on its own. ST2 scores from D1-Combined are 3.5 (ACT), 2.5 (SmolVLA), 7.0 (Pi0.5). Adding just 64 multi-sponge episodes (D1+D2-Half) raises ACT to 18.5 and Pi0.5 to 17.0 — close to the full D_Universal result. SmolVLA recovers more partially (13.0) and only reaches ceiling with the full universal mixture. Small amounts of the right data beat larger amounts of the wrong data — though the T1 → ST3 and T1 → ST4 analogues weren't trained, so the claim is anchored to ST2.

Inference 3. Cross-task data helps.*

* Terms & conditions apply — the useful mixture differs by architecture, and we only tested this on ST1.

Looking only at the ST1 cross-task cells:

• ACT prefers D1+D3-Half (20.0) over D1+D2-Half (15.5) — a 4.5-point gap.
• SmolVLA flips it: D1+D2-Half (20.0) over D1+D3-Half (17.0).
• Pi0.5 follows the same direction as SmolVLA, milder: D1+D2-Half (17.5) over D1+D3-Half (14.5).

So the two VLAs lean toward extra multi-object demonstrations, while the from-scratch ACT leans toward extra distractor demonstrations. One plausible hypothesis: the VLAs already inherit strong visual priors from pretraining, so the marginal value of more data is in additional grasping/sequencing instances; ACT trains its visual encoder from scratch on this dataset alone, so distractor exposure helps it build a more discriminative visual representation. This is a hypothesis, not a measurement — the per-cell n = 16 trials and the result is only checked on ST1.

Inference 4. On single-target tasks (ST1, ST3), broader data makes policies more decisive.

Time-to-success is only meaningful on single-target tasks, where a faster successful rollout reasonably implies a more confident policy. On multi-object tasks (ST2, ST4) the total time depends on how many sponges get picked and in what order, so a longer run can mean more thoroughness rather than less decisiveness — we omit those.

We also drop Pi0.5 from this analysis: it runs at ~6 Hz on the laptop GPU vs ~30 Hz for ACT and SmolVLA, so its absolute times are dominated by control-loop rate rather than policy decisiveness. The two 30 Hz architectures are directly comparable.

ST1 median time-to-success — broader training reduces completion time.

ST3 median time-to-success — trend is much weaker than ST1.

ST1 is clean for SmolVLA: 21 s → 6 s as data broadens. ACT is noisier (the 3.5 s D1-Combined cell is suspicious — small successful-trial count), but D_Universal (5.5 s) is still faster than D1-K (18 s). On ST3 the trend is much weaker — ACT goes 31.5 s → 29 s and SmolVLA 43 s → 38.5 s, single-digit improvements that could easily be noise. So the "decisiveness" story is real on ST1 and at best directional on ST3.

Inference 5. Emergence of architectural differences — distractor tasks (ST3/ST4) finally pull the three policies apart, and the split looks like a vision-backbone story.

On the 144 scored ST3 rollouts: ACT 28/48, Pi0.5 29/48, SmolVLA 10/48. On the held-out 60 ST4 rollouts: ACT 17/20, Pi0.5 12/20, SmolVLA 6/20. The architectures that looked roughly tied on ST1/ST2 diverge sharply once visual distractors are introduced.

Our hypothesis is that the ordering tracks how each architecture's vision-language backbone is treated during training:

• ACT — no VLM at all; the visual encoder is trained from scratch on this dataset, so it learns features that are specialized to discriminating blue sponges from these specific distractors.
• Pi0.5 — a large pretrained VLM backbone (PaliGemma 3B) is frozen during fine-tuning. The rich color/object priors from web-scale pretraining are preserved intact; only the action expert is trained.
• SmolVLA — a smaller VLM (SmolVLM-2 ~256 M) is fully fine-tuned. With only a few hundred episodes, full fine-tuning likely overwrites parts of the pretrained visual representation without enough data to rebuild it cleanly — the worst-of-both-worlds middle ground.

So the two architectures that don't disturb a competent visual representation (ACT specializes one; Pi0.5 preserves one) do well; the one that disturbs without replacing lags. This is a hypothesis consistent with the ranking, not a controlled ablation — we did not train a frozen-backbone SmolVLA or a from-scratch Pi0.5 to isolate the effect.

Inference 6. Emergence of architectural differences — universal training isn't uniformly best on ST3/ST4.

Pulling the ST3/ST4 cells from the score matrix:

Pi0.5 actively prefers in-task data on ST3 (D3-Combined 15.5 > D_Universal 13.0), while ACT does the opposite (D3-Combined 5.0 → D_Universal 15.5). This fits the Inference 5 hypothesis: Pi0.5's frozen visual backbone already knows "blue sponge", so adding T1/T2 single-color demos just dilutes its action expert; ACT's scratch-trained vision needs every bit of cross-task exposure.

The failure vocabulary also shifts from grasp-bound (ST3) to color-selection-bound (ST4), most sharply for Pi0.5:

So distractor robustness is genuinely a distinct capability from clean pick-and-place — it has to be trained and evaluated explicitly, and the bottleneck (grasping vs selection) differs by architecture.

Inference 7. Miscellaneous observations — shape generalized, but sponge material state didn't.

A positive surprise: although training used regular cuboidal sponges, the learned policy could still pick up an irregularly shaped blue sponge at inference. So it isn't memorizing the exact outline.

The harder story was material state. A 30-second diagnostic clip showed that after a sponge was left out during robot work, it stiffened enough that the same pinch strategy used successfully in ST1/ST2 could make it pop out of the gripper. That gives a concrete physical explanation for the elevated grasp-miss counts on ST3/ST4 — alongside the color-selection errors.