Hackathon · Claude Opus 5.5 + MCP · SO-101 arm

Zero Demos vs Imitation Learning

Same robot arm, same two tasks, same scoring as our 384-demo imitation-learning study. Can an LLM agent with generic robot tools and no training keep up?
Aswin & Jeeva  ·  Hackathon  ·  October 2026
Opus 5.5 finds the one blue sponge among ten and drops it in the bowl 4× speedOpus 5.5 + MCP
Task 2, run 7: the one blue sponge among 9 others · 82 s · $0.27 · zero demos

Two Tasks, One Instruction

Both systems get the same text: “Pick the blue sponge or sponges and place them in the bowl”
TASK 1demos: D2
Several blue sponges, clean table
Every blue sponge must end up in the bowl.
IL eval: 2 sponges · 16 trialsOpus: 1–6 sponges · 10 runs · 32 sponges
TASK 2demos: D3
One blue sponge among distractors
Only the blue sponge may go in the bowl. White, pink, green and aqua sponges stay out.
IL eval: 16 trialsOpus: 9–11 distractors · 10 runs

IL baseline · D_Universal

384 teleoperated demos: D1 + D2 + D3 Combined, 128 per task (one sponge, several sponges, distractors), 64 at marked and 64 at random positions. ACT, SmolVLA and Pi0.5 were each trained on all of it. Scored /20: 8 marked trials ×1.0 + 8 random ×1.5.

Opus 5.5 + MCP

Zero demos, no training. Every scene is a new random layout, so every success counts 1.5: 10 runs = 15, scaled to /20. A human checked every run.

Teleoperated demo of Task 1
Task 1 teleop demo (D2)
Teleoperated demo of Task 2
Task 2 teleop demo (D3)
02 / 12Aswin & Jeeva  |  Claude Opus 5.5 + MCP  |  Hackathon · October 2026

From Teleoperation to 16 Dataset Recipes

384
episodes collected
5.7 hours
total collection time
2.51 h video + 30 s/episode reset
128 episodes × 3 tasks (T1, T2, T3)
🤗 HF Open dataset:  huggingface.co/datasets/aswinkumar99/LeRobot-SO101-Pick-Place — Page 1, Trending LeRobot tag
KRMixedCombinedCross-task
T1D1-KD1-RD1-MixedD1-Combined D_Universal  ·  D_D2D3
D_D1D2-Half  ·  D_D1D3-Half
T2D2-KD2-RD2-MixedD2-Combined
T3D3-KD3-RD3-MixedD3-Combined
Total: 16 datasets
ST1 sample
ST1 sample
ST1 sample
ST2 sample
ST2 sample
ST2 sample
ST3 sample
ST3 sample
03 / 12Aswin & Jeeva  |  Claude Opus 5.5 + MCP  |  Hackathon · October 2026

Four Architectures, Four Design Philosophies

ACT
~84 M params
Transformer encoder–decoder + CVAE

Properties

  • Action Chunking Transformer (Zhao et al., ALOHA)
  • Trained from scratch, no visual or language priors
  • Predicts a chunk of future actions per observation

Recipe

  • --policy.type=act · 60 000 steps · batch 32
  • Cameras: overhead + wrist
Prediction: should plateau without data scale.
SmolVLA
~450 M params
SmolVLM-2 (256 M) + action expert (~200 M)

Properties

  • Hugging Face open VLA (2025)
  • Pretrained on community LeRobot datasets
  • VLM backbone gives language + visual grounding

Recipe

  • --policy.path=lerobot/smolvla_base · 20 000 steps
  • Full fine-tune on each dataset
Prediction: should benefit from limited data via VLM grounding.
Pi0.5
~3.4 B params
PaliGemma (3 B) + flow-matching action head

Properties

  • Physical Intelligence π₀.₅ (2025)
  • Trained across many robot embodiments
  • High-level VLA reasoning + low-level action expert

Recipe

  • --policy.path=lerobot/pi05_base · BF16 · H200
  • Action expert only, vision encoder frozen · 20 000 steps
Prediction: robust priors but higher inference cost.
Opus 5.5
0 robot demos
Claude Code agent + so101 MCP server (28 tools)

Properties

  • Anthropic’s general-purpose model, never trained on this robot
  • Sees the cameras through tool calls (look, detect)
  • Plans, acts, checks and retries; writes its own skills

Setup

  • Headless claude -p · effort high · robot tools only
  • Same one-line instruction as the IL policies
Prediction: should handle new layouts and setups, but slower and with a cost per run.
DiT excluded after pre-flight (failed in-distribution evaluation). Parameter counts are approximate; Opus’s size is not published.
04 / 12Aswin & Jeeva  |  Claude Opus 5.5 + MCP  |  Hackathon · October 2026

Training Structure — 202 GPU-hours on Cloud Infrastructure

202
total GPU-hours
ACT  48 h
SmolVLA  72 h
Pi0.5  46 h
DiT (excluded) 36 h
Per-model averages: ACT ≈ 3 h × 16 = 48 h · SmolVLA ≈ 4.5 h × 16 = 72 h · Pi0.5 ≈ 6.6 h × 7 = 46 h · DiT ≈ 9 h × 4 = 36 h.
ACT SmolVLA
6 × RTX PRO 6000 Blackwell
parallel training pool
Wall-clock
26 h
×
GPUs
6
=
GPU-hours
156 h
  • • 32 models — ACT × 16 (~3 h each) + SmolVLA × 16 (~4.5 h each)
  • • Docker image: huggingface/lerobot-gpu
  • • Launched via train_matrix.sh all & train_combo_matrix.sh
  • • ACT: 60 K steps / SmolVLA: 20 K steps
Pi0.5
NVIDIA H200
single-GPU run for the 3.4 B-param model
Wall-clock
46 h
=
GPU-hours
46 h
  • • 7 models (3 base × -all + 4 combo)
  • • --policy.dtype=bfloat16 · batch size 32
  • • train_expert_only=true, vision encoder frozen
  • • 20 K steps · 1 K-step warmup · LR 5e-5
Inference
RTX 4090 Mobile
on-robot laptop
ACT
30 Hz
SmolVLA
~30 Hz
Pi0.5
~6 Hz
Only Pi0.5 is much slower (~6 Hz), which matters for its time-to-success results.
DiT × 4 attempted but excluded after pre-flight (zero in-distribution success). All other 39 models converged.
05 / 12Aswin & Jeeva  |  Claude Opus 5.5 + MCP  |  Hackathon · October 2026

How Claude Drives the Arm: MCP Tools + Project Memory

Claude Code · Opus 5.5

headless on a laptop · effort high · robot tools only

Gets the one-line task, loads its project memory, then plans, acts and checks one tool call at a time.

MCP over HTTP · 28 tools · camera images come back as tool results

so101d MCP server

NVIDIA Jetson next to the arm
See
look · detect · survey · overhead_to_table
Move
move_to · move_by · set_gripper · go_to_pose · stop
Batch with checks
run_sequence · run_skill, stopping at the first failed check
Remember
set_project · project_values · project_notes · project_book · save_skill
Calibrate
calibrate_overhead · calibrate_table · camera_setup
50 Hz joint control · stops when a move is blocked

SO-101 arm

Overhead camera

Wrist camera

Project memory

Stored on the Jetson, per project. Every run starts by loading it and ends by adding to it. Run 1 started empty; after 21 runs it held:

Skills7 saved
Multi-step recipes with built-in checks, e.g. pick_sponge_to_bowl: hover, align on the wrist image, descend, close, check the grip load, lift, carry, release.
Values9 keys
Facts it measured, e.g. bowl_drop, the release point above the bowl, and max_direct_grasp_x = 33 cm.
Project book4 questions
Questions answered by experiment, with evidence and a re-check step, e.g. “Which ‘blue’ detections are really aqua decoys?”
Notes22 entries
One entry per run: what worked, what failed, what to try next.
load memory→look & detect→run skills with checks→save lessons
06 / 12Aswin & Jeeva  |  Claude Opus 5.5 + MCP  |  Hackathon · October 2026

Results: 20/20 on Both Tasks, With Zero Demos

SystemTrainingScore /20Median timeAPI cost per runDecision speed
Task 1Task 2Task 1Task 2
ACT384 demostrained from scratch19.015.5112.0 s29.0 s≈ $0*~30 Hzaction chunks
SmolVLA384 demosfine-tuned20.05.064.5 s38.5 s≈ $0*~30 Hz
Pi0.5384 demosfine-tuned20.013.0113.5 s48.5 s≈ $0*~6 Hz
Opus 5.5 + MCP0 demosno training20.020.0123 s95 s$0.56 · $0.46Task 1 · Task 2, list price3.5 s per LLM call~12 calls per run; the arm moves at 50 Hz
21 / 21
runs succeeded, including the robustness run
$10.62
API cost for all 21 runs, $0.25 per sponge placed
2–3× slower
than the IL policies on Task 2; on Task 1, about as fast as ACT and Pi0.5
* IL inference costs ≈ $0 per run; the real cost was one-time: 384 teleop demos (5.7 h) and 202 GPU-hours. IL rows: April 2026 eval, 16 trials per task, another bowl and camera position, policies on an RTX 4090 laptop GPU. Opus Task 1 scenes had 1–6 sponges (IL: 2). Times are medians over successful runs.
07 / 12Aswin & Jeeva  |  Claude Opus 5.5 + MCP  |  Hackathon · October 2026

It Gets Faster by Writing Down What It Learns

Task 1 · runs 1–10 Task 2 · runs 11–20 Robust. Time per sponge placed (s) 0 200 400 600 Run 1 (Task 1 #1): 134 s per sponge, 5 sponges, 670 s in total Run 2 (Task 1 #2): 42 s per sponge, 2 sponges, 83 s in total Run 3 (Task 1 #3): 42 s per sponge, 3 sponges, 125 s in total Run 4 (Task 1 #4): 81 s per sponge, 4 sponges, 323 s in total Run 5 (Task 1 #5): 40 s per sponge, 3 sponges, 121 s in total Run 6 (Task 1 #6): 103 s per sponge, 6 sponges, 619 s in total Run 7 (Task 1 #7): 50 s per sponge, 1 sponge, 50 s in total Run 8 (Task 1 #8): 54 s per sponge, 2 sponges, 108 s in total Run 9 (Task 1 #9): 50 s per sponge, 5 sponges, 252 s in total Run 10 (Task 1 #10): 53 s per sponge, 1 sponge, 53 s in total Run 11 (Task 2 #1): 123 s per sponge, 1 sponge, 123 s in total Run 12 (Task 2 #2): 75 s per sponge, 1 sponge, 75 s in total Run 13 (Task 2 #3): 66 s per sponge, 1 sponge, 66 s in total Run 14 (Task 2 #4): 281 s per sponge, 1 sponge, 281 s in total Run 15 (Task 2 #5): 85 s per sponge, 1 sponge, 85 s in total Run 16 (Task 2 #6): 574 s per sponge, 1 sponge, 574 s in total Run 17 (Task 2 #7): 82 s per sponge, 1 sponge, 82 s in total Run 18 (Task 2 #8): 87 s per sponge, 1 sponge, 87 s in total Run 19 (Task 2 #9): 254 s per sponge, 1 sponge, 254 s in total Run 20 (Task 2 #10): 102 s per sponge, 1 sponge, 102 s in total Run 21 (robustness run): 124 s per sponge, 1 sponge, 124 s in total 134 s: cold start 574 s: two dropped grasps, then 2 new skills API cost per sponge placed ($, list price) 0.0 0.4 0.8 1.2 Run 1 (Task 1 #1): $0.35 per sponge, $1.75 for the run Run 2 (Task 1 #2): $0.07 per sponge, $0.14 for the run Run 3 (Task 1 #3): $0.05 per sponge, $0.14 for the run Run 4 (Task 1 #4): $0.12 per sponge, $0.48 for the run Run 5 (Task 1 #5): $0.05 per sponge, $0.16 for the run Run 6 (Task 1 #6): $0.30 per sponge, $1.78 for the run Run 7 (Task 1 #7): $0.15 per sponge, $0.15 for the run Run 8 (Task 1 #8): $0.15 per sponge, $0.30 for the run Run 9 (Task 1 #9): $0.09 per sponge, $0.46 for the run Run 10 (Task 1 #10): $0.20 per sponge, $0.20 for the run Run 11 (Task 2 #1): $0.36 per sponge, $0.36 for the run Run 12 (Task 2 #2): $0.27 per sponge, $0.27 for the run Run 13 (Task 2 #3): $0.24 per sponge, $0.24 for the run Run 14 (Task 2 #4): $0.96 per sponge, $0.96 for the run Run 15 (Task 2 #5): $0.23 per sponge, $0.23 for the run Run 16 (Task 2 #6): $1.12 per sponge, $1.12 for the run Run 17 (Task 2 #7): $0.27 per sponge, $0.27 for the run Run 18 (Task 2 #8): $0.26 per sponge, $0.26 for the run Run 19 (Task 2 #9): $0.55 per sponge, $0.55 for the run Run 20 (Task 2 #10): $0.31 per sponge, $0.31 for the run Run 21 (robustness run): $0.47 per sponge, $0.47 for the run What it saved to project memory Skills saved (7) Run 1 (Task 1 #1): pick_sponge_to_bowl, drag_sponge_closer2 Run 6 (Task 1 #6): hover_low_look, grasp_lift_to_bowl2 Run 14 (Task 2 #4): grasp_slow_carry_to_bowl Run 16 (Task 2 #6): grasp_lift_look, carry_to_bowl2 Values written (20) Run 1 (Task 1 #1): 9 values written9 Run 4 (Task 1 #4): 1 values written Run 6 (Task 1 #6): 4 values written4 Run 7 (Task 1 #7): 1 values written Run 8 (Task 1 #8): 1 values written Run 9 (Task 1 #9): 1 values written Run 21 (robustness run): 3 values written3 Book updates (7) Run 1 (Task 1 #1): 3 book updates3 Run 11 (Task 2 #1): 2 book updates2 Run 12 (Task 2 #2): 2 book updates2 Notes written (22) Run 1 (Task 1 #1): 1 notes written Run 2 (Task 1 #2): 1 notes written Run 3 (Task 1 #3): 1 notes written Run 4 (Task 1 #4): 1 notes written Run 5 (Task 1 #5): 1 notes written Run 6 (Task 1 #6): 1 notes written Run 7 (Task 1 #7): 1 notes written Run 8 (Task 1 #8): 1 notes written Run 9 (Task 1 #9): 1 notes written Run 10 (Task 1 #10): 1 notes written Run 11 (Task 2 #1): 1 notes written Run 12 (Task 2 #2): 1 notes written Run 13 (Task 2 #3): 1 notes written Run 14 (Task 2 #4): 2 notes written2 Run 15 (Task 2 #5): 1 notes written Run 16 (Task 2 #6): 1 notes written Run 17 (Task 2 #7): 1 notes written Run 18 (Task 2 #8): 1 notes written Run 19 (Task 2 #9): 1 notes written Run 20 (Task 2 #10): 1 notes written Run 21 (robustness run): 1 notes written 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 run
134 s → 52 s per sponge
Task 1, cold start vs the median of runs 7–10: 2.6× faster.
$0.35 → $0.15 per sponge
API cost over the same runs.
Slow run → new skill
Run 6 took 103 s per sponge and saved 2 skills; run 7 took 50 s. Run 14: 281 s → 85 s. Run 16: 574 s → 82 s.
670 s → 123 s
First Task 1 run vs first Task 2 run: the new task reused Task 1’s calibration and skills instead of starting cold.
a skill was saved in that run
08 / 12Aswin & Jeeva  |  Claude Opus 5.5 + MCP  |  Hackathon · October 2026

Seven Skills Claude Wrote for Itself

SkillSavedWhat it does, and why it exists
pick_sponge_to_bowlrun 1First full recipe: hover, align on the wrist image, descend, close, check the grip load, lift, carry, release.
drag_sponge_closerrun 1Sponge out of reach (beyond 33 cm): put the closed gripper behind it and pull it 9 cm back.
hover_low_lookrun 6After missed grasps: hover 4 cm above the sponge and return the wrist image to fine-tune the aim.
grasp_lift_to_bowlrun 6Go down to ~1 cm (2 cm slipped), close, check load ≥ 8 %, lift, carry, release.
grasp_slow_carry_to_bowlrun 14A soft grip slipped at normal speed: the same grasp, then a slow lift and carry.
grasp_lift_lookrun 16After two drops: grasp, lift 3 cm and check the wrist image before carrying.
carry_to_bowlrun 16Slow carry to the saved drop point, then release.
The arm pulls a far sponge closer with the closed gripper, then picks it up4× speed
Task 1, run 9: the last sponge sits at 35 cm, out of reach. Claude runs drag_sponge_closer, then picks it with the grasp skills.
# saved in run 1, used whenever x > 33 cm
drag_sponge_closer(x, y)
  set_gripper 0             # close the jaws
  travel      x, y, z=10
  move_by     dx=+1.5 dz=-6.5  # just beyond it
  move_by     dz=-2.7       # down to the table
  move_by     dx=-9 speed=0.2  # pull 9 cm closer
  move_by     dz=+6         # lift, then re-detect
09 / 12Aswin & Jeeva  |  Claude Opus 5.5 + MCP  |  Hackathon · October 2026

How Flexible Is It? Light, Bowl and Bowl Position Changed at Once

Baseline scene with the terracotta bowl
Runs 1–20: terracotta bowl, same spot, room light
Robustness scene with an unseen white bowl on the other side
Run 21: unseen white bowl, moved, new lighting
Lightingchanged
Bowlnever seen before
Bowl positionother side of the arm
“The saved drop point was for the old orange bowl at (24, 16), so I didn’t use it. I checked the new bowl with the wrist camera before letting go.”Claude’s report after the run. Nobody told it what had changed.
Opus 5.5 picks the blue sponge and drops it into the new white bowl4× speed
✓ success, first grasp124 s14 LLM calls$0.470 new demos
Afterwards it saved the new bowl position and drop point, so later runs start from the new setup. The IL policies only ever saw one bowl, in one spot, under one lighting setup.
10 / 12Aswin & Jeeva  |  Claude Opus 5.5 + MCP  |  Hackathon · October 2026

Everything Is Open

Project page

aswinkumar.me/so-arm101

GitHub

aswinkumar1999/so-arm101-imitation-learning

Hugging Face dataset

aswinkumar99/LeRobot-SO101-Pick-Place

Eval gallery clips

aswinkumar99/so101-eval-gallery

The study, the 384 demos and every IL rollout, open on GitHub and Hugging Face.

11 / 12Aswin & Jeeva  |  Claude Opus 5.5 + MCP  |  Hackathon · October 2026