Tactile-Augmented Mixture-of-Horizons Flow Policies for Bimanual Dexterous Origami

1Abstract

We propose Origami-MoH, a vision-language-action (VLA) policy for autonomous bimanual dexterous paper folding, targeting the IROS 2026 Robotic Origami Challenge (traditional paper-airplane folding, a fixed six-fold sequence on a 65-DoF system: two 7-DoF arms and two 22-DoF hands). We fine-tune the π0.5 flow-matching VLA and augment it along two axes we argue are decisive for contact-rich deformable manipulation: (i) a Mixture-of-Horizons action head that fuses predictions across chunk lengths and replans adaptively on cross-horizon disagreement, and (ii) progressive tactile fusion of a 60-D fingertip force/torque signal, escalating from proprioceptive concatenation to a force-aware mixture-of-experts action decoder. Because the dataset ships no language and the scene and sequence are fixed, we train one vision-conditioned policy over the whole sequence rather than a language-switched hierarchy, relying on the visual and tactile state to disambiguate the active fold. We give a staged plan (Phases 0–4) with go/no-go criteria, an evaluation over held-out sessions, and the systems work to train on a per-session-sharded, 65-DoF, tactile-rich dataset. This document is a living proposal; empirical results will be reported as phases complete.

2Introduction

Autonomous origami is a demanding testbed: the manipuland is thin and deformable, creases must be placed and pressed with sub-centimeter precision, folds self-occlude the very seams the policy must reason about, and the task is long-horizon and sequential. The challenge instantiates this on a bimanual dexterous platform, judging success by whether the folded result is a recognizable paper airplane.

VLAs such as π0 and π0.5 [1,2] show that a pretrained vision-language backbone plus a flow-matching action expert can be fine-tuned into competent policies from modest demonstration sets. But three properties of the origami setting are not addressed by a stock VLA: the task is contact-rich, and creasing is sensed as much through force as vision; execution must be reactive, since paper does not deform deterministically; and the demonstration set is small and un-annotated. We address these through progressive tactile fusion, a Mixture-of-Horizons head with disagreement-based replanning, and single-policy multi-session co-training rather than per-fold sub-policies.

2.1Contributions

1. A concrete architecture, Origami-MoH, combining π0.5 flow matching, a Mixture-of-Horizons action head, and tactile fusion, specialized to a 65-DoF bimanual dexterous embodiment.
2. A staged, falsifiable plan (Phases 0–4) that isolates the value of tactile sensing, then equips the policy with long-horizon procedural memory.
3. The systems work to make this trainable in openpi: per-session multi-dataset training via MultiLeRobotDataset, a fix enabling 65-DoF fine-tuning from a 32-DoF base checkpoint, and a video-free normalization pass.

3Problem Setup

Embodiment and observations. The platform is bimanual: two 7-DoF arms and two 22-DoF Sharpa hands plus a 7-DoF torso, giving a 65-D joint action/state vector,

Sensing comprises six synchronized camera streams (head L/R, wrist L/R, two tactile-video streams) at 30 Hz, a 60-D fingertip signal τ_t ∈ ℝ⁶⁰ (ten fingertips × six-axis force/torque), and 65-D joint torque.

Dataset. The dataset contains 143 collection sessions ("seasons"), each a self-contained LeRobot dataset, totaling ~4.8M frames. Two properties shape the systems design: it is sharded per session (no flat root), and it carries only a placeholder task string — i.e., effectively no language. All sessions are training sessions; we build an eval split by holding out whole sessions.

Objective. Learn π_θ(a_t:t+H | o_t) mapping an observation o_t = (images, s_t, τ_t, ℓ) — with ℓ a fixed prompt — to an H-step action chunk, such that closed-loop execution produces a recognizable airplane. Because the scene and fold order are fixed, the current observation identifies the active fold, motivating a single vision-conditioned policy over explicit sub-task conditioning (§5.4).

4Related Work

4.1VLA backbones & flow-matching policies

π0/π0.5 [1,2] pair a PaliGemma-style backbone with a flow-matching expert; π0.5 adds a hierarchical structure and discrete state tokenization. Diffusion Policy [14] and ACT/ALOHA [15] are strong single-task, low-data baselines — motivating our choice to keep the pretrained visual prior rather than train an encoder from scratch on this visually hard deformable task.

4.2Precision & multi-horizon prediction

Chunk length trades reactivity against smoothness. Mixture-of-Horizons [3] fuses truncated horizons through a shared expert. Coarse-to-fine / spectral methods — CARP [10], CF-VLA [11], Causal Spectral Policy [12], related to the FAST frequency view [13] — improve fine-grained precision; we treat a spectral head as a deferred, MoH-complementary option.

4.3Tactile fusion

ForceVLA [6] inserts a force-aware mixture-of-experts (FVLMoE) at action decoding; ViTacFormer [7] cross-attends vision and touch with an autoregressive contact head on multi-fingered hands; 3D-ViTac [8] builds a unified 3D visuo-tactile representation; Tactile-VLA [9] fuses four modalities with a hybrid position-force controller. Reported gains on contact-rich dexterous tasks are consistently 20–50%. Our FVLMoE (Phase 2) follows ForceVLA most closely — its low-D force interface matches our 60-D signal.

4.4Reactive execution, coherence & long-horizon

Real-Time Chunking [4] enables asynchronous execution under latency; Action Coherence Guidance [5] is training-free guidance that reduces the drift flow policies inherit from noisy demos. We adopt MoH's disagreement-based replanning for reactivity and ACG for coherence. For long-horizon memory, RoboTTT [16] inserts gated test-time-training layers alongside attention — a fixed-size fast-weight recurrent state that compresses thousands of timesteps into weight space and conditions on the full rollout at constant inference latency, demonstrated on multi-minute bimanual dexterous assembly. We view this not as a generic drift patch but as the principled form of observation history (§5.4): a memory architecture whose near-zero gate initialization preserves the pretrained backbone. Because the fold is long and strictly ordered, we adopt it as a dedicated phase (Phase 4, §5.6) rather than a contingency.

4.5Deformable manipulation

DeMaVLA [17] is a single-checkpoint VLA for multi-category garment folding, with a data-scaling trend supporting broad pretraining. We note the gap: garment folding is not rigid-crease paper folding, and no origami-specific baseline is established — we treat cloth results as suggestive, not transferable.

5Method: Origami-MoH

5.1Base policy: π0.5 flow matching

The backbone is π0.5: ~3B parameters — a PaliGemma vision-language module (SigLIP + Gemma LM) and a Gemma-300M action expert, a conditional flow-matching model. With a clean chunk a, noise ε ∼ N(0, I), and interpolation x_t = tε + (1−t)a, the expert v_θ regresses the target velocity:

Actions are produced by integrating the field from noise via Euler steps x_t+Δt = x_t + Δt v_θ(x_t,t,o). For π0.5, proprioceptive state is discretized and injected as language tokens — which we exploit below.

5.2Mixture-of-Horizons action head

Rather than commit to one chunk length, MoH runs horizons ℋ = {5,10,15,20,25,30} through the shared expert and fuses their per-step velocities with a linear gate. With v^(h)_t the velocity under horizon h and g_h,t the gate weight (masked softmax over active horizons):

Training combines a per-horizon flow-matching loss, an auxiliary loss on the fused prediction, and a gate load-balancing term. At inference we compute per-step cross-horizon disagreement (the gate-weighted ℓ₁ deviation between each horizon's Euler update and the fused update) and use it to set an adaptive replanning length: extend the executable prefix while horizons agree, truncate when they diverge — e.g., near a crease. This gives reactivity without an external real-time-chunking mechanism.

5.3Progressive tactile fusion

We introduce tactile in increasing architectural depth, so its value is established before the most invasive change.

(P0) Proprioceptive concatenation. Append the 60-D force/torque to the state, s̃_t = [s_t; τ_t] ∈ ℝ¹²⁵. Under π0.5, state is discretized into the token stream, so tactile is treated exactly like joint proprioception with no model-code change — a go/no-go probe on whether the signal is informative at all.

(P1) Tactile tokens. A learned encoder maps the force/torque stream (and optionally the tactile-video streams via the vision tower) into prefix tokens that co-attend with vision and language, à la ViTacFormer [7]. Tokens enter the KV-cached prefix, encoded once and shared across all MoH horizons.

(P2) Force-aware mixture-of-experts (FVLMoE). Replace the action expert's feed-forward block with a force-aware MoE (ForceVLA [6]): a router conditioned on the post-attention token (which, after joint attention over tactile/vision/language, carries fused force context) mixes modality-specialized expert FFNs. Crucially this MoE acts on a different axis and stage than the MoH gate — modality experts at the trunk FFN vs. horizon fusion at the output head — so the two mixtures compose rather than compete.

5.4A single vision-conditioned policy

With no language annotations and a fixed fold order, we condition on a single constant prompt ("fold a paper airplane"); a constant token carries no discriminative signal, so the policy is effectively vision-conditioned. We train one policy over the whole sequence rather than per-fold sub-policies: chunk-based training already samples short (~1 s) windows from anywhere in an episode, so the model learns fold-local behavior implicitly, and the current image plus proprio/tactile state disambiguate the active fold. Splitting 143 sessions into per-fold policies would starve each of data. If phase confusion appears, we escalate a ladder of increasing cost: (i) short proprio/tactile history (cheap, no labels), (ii) image history, (iii) constant-latency full-rollout memory via gated TTT layers [16] — the principled limit of observation history, which we commit to as a dedicated phase (Phase 4, §5.6) — and (iv) an explicit planner supplying a sub-task ID, the only rung needing segmentation. We develop Phase 4 as part of the plan and evaluate it as an A/B against frame-stacked history on the per-fold diagnostic (§8); the planner rung (iv) remains contingent on that result.

5.5Inference-time coherence

Flow policies trained by imitation inherit demonstration jitter, which can destabilize fine motions. We adopt Action Coherence Guidance [5], a training-free guidance term applied to the fused velocity before each Euler step (single guidance-weight knob); it composes with MoH and needs no retraining. As an optional final smoothing step — applied only once the policy is otherwise competent and we want the cleanest motion — we may fit a smoothing B-spline [18] to the sampled action chunk, yielding a C²-continuous, jerk-free trajectory that preserves crease keyframes. Like ACG, it is post-hoc and needs no retraining.

5.6Long-horizon procedural memory (Phase 4)

Paper folding is a minutes-long, strictly ordered sequence; a Markov policy must re-infer its place in the sequence from each observation. Phase 4 gives the policy an explicit procedural memory — an intuition we summarize as "muscle memory" — by inserting gated test-time-training (TTT) layers into the backbone, following RoboTTT [16]. Each TTT layer maintains a fixed-size fast-weight state updated by an inner-loop gradient step (a key-value binding objective) as the rollout proceeds, compressing the entire fold history into weight space and conditioning on it at constant inference latency. The layers enter through a learned near-zero gate, o = tanh(α)·o_TTT + o_attn, so at init the pretrained backbone is preserved and the memory is acquired during fine-tuning. This is the principled limit of the observation-history ladder (§5.4): instead of a few stacked frames, the policy carries the whole sequence, which we expect to help long-horizon phase tracking and recovery. We evaluate Phase 4 as an A/B against frame-stacked history on the per-fold diagnostic (§8).

Greenlight criterion. Because P4 is the most expensive increment (the π0.5 port plus long-context training), we gate its full build on a cheap early signal rather than committing to it blind. In P1 we run a frame-stacking observation-history probe; we commit to the P4 build only if residual phase confusion remains in the per-fold diagnostic after that probe. If cheap stacked history already closes the long-horizon gap, P4 is treated as optional. This keeps P4 in the plan — so the capability is developed before, not after, we discover we need it — while spending its budget only on evidence that the memoryless policy cannot track the fold sequence on its own.

Two engineering considerations are internal to this phase rather than reasons to defer it. First, the mechanism was demonstrated on the GR00T-N1.7 backbone for multi-minute bimanual dexterous assembly and must be ported to the π0.5 backbone; because the gate is inserted alongside attention with a near-zero init, this is an additive, adapter-style change rather than a backbone rebuild. Second, long-context training (sequence action forcing with truncated backpropagation through time) is heavier than chunk-based fine-tuning and interacts with MoH's multi-horizon batching, so Phase 4 is sequenced after a competent base policy (Phases 0–2) and its inference-time refinements (Phase 3).

6Staged Research Plan

We proceed in phases with explicit success criteria, so each increment is justified by the previous result rather than assumed.

7Implementation

We build on openpi. Three systems issues are notable and resolved on the origami branch.

65-DoF fine-tuning from a 32-DoF base. The released π0.5 checkpoint uses a 32-D padded action space, whereas the challenge action space is 65-D. We make the checkpoint merge shape-aware: the input/output action projections reinitialize (and are whitelisted for fresh init) while the rest of the pretrained backbone and expert load unchanged.

Per-session multi-dataset training. The dataset is sharded into 143 per-session LeRobot datasets, and the pinned LeRobot is v2.1 with no aggregation utility. Rather than physically merge ~2.3 TB of video, we train jointly over all sessions via MultiLeRobotDataset: sessions are staged under a common root and discovered at load time, halving disk vs. a merge. A matching video-free wrapper skips decoding during normalization-statistic computation.

Compute. Full fine-tuning of the ~3B model with MoH's multi-horizon batching is sharded across two 80 GB GPUs (FSDP). Held-out sessions provide the eval split.

8Evaluation Protocol

• Split. Eight whole sessions, evenly spaced across the collection, held out for evaluation; the remaining 135 for training.
• Primary metric. Task success — a recognizable airplane — consistent with the challenge judging.
• Diagnostic metric. Per-fold drop-off: where in the six-fold sequence success degrades. A gradual decline indicates compounding drift (→ ACG, interventional data); a sharp failure at one transition indicates fold-phase confusion (→ observation history, escalating to gated TTT-layer memory [16] if frame-stacking is insufficient). Per-fold scoring requires segmenting held-out episodes — a separate labeling step.
• Ablations. Tactile on/off (P0); tokens vs. concatenation (P1); FVLMoE on/off (P2); ACG on/off (P3).

9Risks and Mitigations

10Conclusion

Origami-MoH is a pragmatic, staged approach to autonomous bimanual origami: a π0.5 flow policy made reactive by Mixture-of-Horizons and contact-aware by progressive tactile fusion, trained as a single vision-conditioned policy over a fixed fold sequence. The plan front-loads a cheap test of tactile value and escalates architectural investment only as evidence warrants.

 References

1. Physical Intelligence. π0: A Vision-Language-Action Flow Model for General Robot Control. 2024.
2. K. Black et al. π0.5: A VLA with Open-World Generalization. arXiv:2504.16054, 2025.
3. Mixture of Horizons in Action Chunking. 2025.
4. K. Black et al. Real-Time Execution of Action Chunking Flow Policies. arXiv:2506.07339, NeurIPS 2025.
5. Action Coherence Guidance for Flow-based VLA Policies. arXiv:2510.22201, ICRA 2026.
6. ForceVLA: Force-Aware Mixture-of-Experts for Contact-Rich Manipulation. arXiv:2505.22159, 2025.
7. ViTacFormer: Cross-Modal Representations for Visuo-Tactile Dexterous Manipulation. arXiv:2506.15953, 2025.
8. 3D-ViTac: Unified Visuo-Tactile Representations for Dexterous Manipulation. arXiv:2410.24091, CoRL 2024.
9. Tactile-VLA: Four-Modality Fusion with a Hybrid Position-Force Controller. arXiv:2507.09160, 2025.
10. J. Gong et al. CARP: Coarse-to-Fine Autoregressive Prediction. ICCV 2025.
11. CF-VLA: Coarse-to-Fine Flow for VLA Models. arXiv:2604.24622, 2026.
12. Causal Spectral Policy: Frequency-Decomposed Action Prediction. arXiv:2606.29570, 2026.
13. Physical Intelligence. FAST: Efficient Action Tokenization for VLA Models. arXiv:2501.09747, 2025.
14. C. Chi et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.
15. T. Zhao et al. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT/ALOHA). RSS 2023.
16. NVIDIA GEAR. RoboTTT: Context Scaling for Robot Policies. arXiv:2607.15275, 2026.
17. DeMaVLA: A VLA Foundation Model for Generalizable Deformable Manipulation. arXiv:2605.31286, 2026.
18. C. de Boor. A Practical Guide to Splines. Springer, 1978.