Neural Cup Pong: A Playable Action-Conditioned World Model for 3D Cup Pong

A hobby project. All art, physics, and assets are original; nothing here reproduces a commercial title.

1Abstract

World models learn to predict how an environment evolves under an agent's actions, and a growing line of work asks whether such models can be run as the environment: a neural network the player controls directly, with no hand-written simulator or renderer [1,3,4,5]. This report documents a small, self-contained instance of that idea, Neural Cup Pong: a playable action-conditioned world model for an original 3D cup-pong game (aim, charge power, throw a ballistic ball at a triangular rack of six cups).

The system is built in six stages: (1) a deterministic, seeded game engine with a fixed-camera 2.5D projection; (2) a scripted data-generation pipeline; (3) a structured dynamics model, a gated recurrent network that predicts the next 21-dimensional game state from the current state and controller action, with a hard "snap" projection that keeps every prediction a legal game state; (4) engine-off inference, where the network alone advances the world; (5) a state-grounded latent renderer that paints each 128×96 frame from state using rasterized geometry hints and FiLM conditioning; and (6) fully-generated play, where both the engine and the graphics renderer are switched off and the two networks produce the game end-to-end.

The learned model is near-perfect at the parts that matter for feel: it tracks aim and power with one-step errors of 0.0025 and 0.0013 (normalized), reproduces phase transitions at 99.4% accuracy, and follows the player's controls with a 100% directional-response rate. It is not good at one specific thing: predicting exactly where a thrown ball lands. The GRU's predicted flight passes no closer than ≈8.4 table-units to the target cup, versus 0.31 for the engine (larger than a cup), so the fully-learned mode almost never sinks. We show this is a genuine capacity limit of a small structured model regressing a fast ballistic arc, not a training or class-imbalance artifact, and resolve it with a hybrid world model: the network keeps driving aim, power, and phase, but the ball's arc is integrated with exact ballistics seeded from the network's own (accurate) aim/power at release. The hybrid sinks 6/6 cups, matching the engine, while control stays neural and the decoder still renders every pixel.

2Introduction

2.1Playable world models

A world model is a learned simulator: given the current state (or observation) and an action, it predicts the next one. Classic formulations use them for imagination-based planning and reinforcement learning [1,2]. A more recent and more visceral use is to let a human drive the world model directly, turning it into a neural video game: GameNGen reproduces DOOM with a diffusion model [4], Genie learns controllable environments from video [5], and DIAMOND trains agents inside a diffusion world model of Atari [3]. These systems are impressive but heavy. This project asks the opposite question: how small can a genuinely playable world model be, and what breaks first as you shrink it?

Cup pong is a deliberate choice. It is simple enough that a deterministic engine fits in a few hundred lines, yet it contains the three ingredients that make world modeling interesting: (i) continuous control (aim and power the player trims in real time), (ii) a ballistic event (a thrown ball whose landing is a sensitive function of the launch), and (iii) discrete, irreversible state changes (a cup is present or sunk; the rack only empties). The last two pull in opposite directions, and the tension between them is the story of this report.

2.2Structured vs. pixel dynamics

Rather than learn dynamics directly in pixel space, we separate concerns. A structured dynamics model predicts a compact, interpretable game state (ball position, cup bitmask, phase, and so on); a separate neural renderer turns state into pixels. This mirrors the "structured latent first" discipline used to make world models controllable and debuggable before committing to pixels, and it lets us measure exactly which faculty (dynamics or rendering) is responsible for any failure. It also means the two networks can be tiny: 409K and 262K parameters respectively.

2.3Contributions

1. A complete, six-stage recipe for a playable action-conditioned world model of an original arcade game, small enough to train on a single consumer GPU and run at interactive rates on CPU.
2. A snap projection operator that guarantees every learned state transition is legal (valid phase automaton, monotone cups, derived score, parked ball outside flight), used identically in training feedback and at inference.
3. A state-grounded latent renderer (SGLR) that reaches L₁ = 0.0077 reconstruction on held-out frames by feeding the decoder rasterized geometry hints plus a static court backdrop, rather than asking a conv net to invent the scene from a state vector.
4. A clean diagnosis of the flight-precision wall: a ~409K structured model tracks control to ≈0.002 but cannot localize a ballistic landing to within a cup radius (8.4u error), and no amount of loss re-weighting, event oversampling, or rule-based sink resolution fixes it because the predicted ball never reaches the cup.
5. A hybrid world model that restores engine-parity sinking (6/6) by integrating the ball's arc with exact ballistics seeded from the network's own aim/power, while leaving control and rendering fully neural: an honest, playable compromise and a concrete data point on what a small model should and should not be asked to learn.

3The Game and the Deterministic Engine

3.1Rules and camera

The player faces a table. Table-space uses x across ([0,60]), y in depth toward the cups ([0,100]), and z up. Six cups sit in a triangular rack (rows of 3, 2, 1) at y ∈ {74, 82, 90}. The player trims a lateral aim aim_x ∈ [−1,1] and a launch power power ∈ [0,1], then throws; the ball follows a fixed-elevation ballistic arc and either drops into a cup, rims out, or misses. Clearing all six cups wins. The engine renders with a fixed-camera 2.5D projection: a static perspective map from table-space (x,y,z) to a 128×96 frame. Because the camera never moves, the projection is a pure function π : ℝ³ → ℝ², later ported to PyTorch to raster geometry hints for the neural renderer (verified bit-for-bit against the engine at 0.0px error).

3.2State, action, and event spaces

The world state is a 21-dimensional vector (Table 1); the controller action is a 5-dimensional multi-hot, [aim_left, aim_right, power_up, power_down, throw]; and the engine emits a 7-dimensional event vector per step, [throw_released, cup_sunk, miss, table_bounce, rim_bounce, rack_cleared, game_over], used as an auxiliary training signal. Observations are decoupled from simulation: physics integrates at 60 Hz but the agent observes every third step (20 Hz), so a single learned step must summarize three physics ticks.

3.3Physics

At release, the launch speed and direction are set from the trimmed controls:

giving initial velocity v = v₀(cosφ sinθ, cosφ cosθ, sinφ) from a fixed origin, with φ held constant so the throw is a one-parameter-per-axis family the player learns to feel. The ball then integrates under gravity:

Cup interaction is resolved only at the descending crossing of the rim plane z = CUP_RIM_Z, the honest geometric definition of "over the cup on the way down." Let d be the distance from the ball to the nearest present cup center at that crossing:

• make if d ≤ SINK_RADIUS = 3.3
• rim-bounce if SINK_RADIUS < d ≤ CUP_R + BALL_R (upward pop, outward kick)
• pass otherwise (then test the table for a miss)

The forgiving SINK_RADIUS (< CUP_R = 4.0) makes "visibly over the cup" drop, which matters greatly once a learned ball has to hit it.

4Data Generation

Trajectories are generated by a deterministic scripted thrower. It caches an (aim_x, power) → landing table by simulating the engine's ballistics on a 41×41 grid, selects the grid cell whose true landing is nearest a chosen present cup, and drives the aim/power controls toward it with a skill-scaled jitter so the dataset contains a realistic mix of makes, rims, and misses. Each episode records the full (s_t, a_t, events_t) stream and, optionally, rendered frames for the visual stage. Because the thrower is seeded, the entire dataset is reproducible from an integer.

5Structured Dynamics Model

5.1Architecture

The dynamics model is a gated recurrent network [6] with a per-field decoder head:

From h_t, four heads predict the next state: a continuous head (11 dims: position, velocity, aim, power, score, throws, timer), a cups head (6 logits), a phase head (4 logits), and an auxiliary event head (7 logits). The full model is 408,860 parameters. A key modeling choice is what each continuous field predicts. Slowly-drifting quantities (position, aim, power, the counters) are predicted as deltas added to the current value, keeping the network near identity and stabilizing long rollouts; velocity is predicted absolutely, because during flight it changes by a large, near-constant gravitational increment each step that is easier to name outright than to correct incrementally.

5.2The snap projection

A raw regression head will happily predict a cup that un-sinks, a fractional throw counter, or an illegal AIM → GAME_OVER jump. We forbid this with a hard projection operator Π (snap) applied to every predicted next-state, identically in training feedback and at inference:

Π enforces: (i) a legal phase transition, masking phase logits by an automaton adjacency matrix before the argmax; (ii) monotone cups (a present cup may sink but a sunk cup never returns); (iii) score = 6 − ∑ cups, derived not predicted; (iv) a rounded, monotone throw counter; (v) bounded aim/power/z/timer; and (vi) phase-conditioned parking: outside FLIGHT the ball is snapped to the throw origin with zero velocity. Snap turns a noisy real-valued prediction into a guaranteed-legal game state, which is what keeps engine-off rollouts coherent for hundreds of steps.

6Training Curriculum and Losses

Training uses a two-phase curriculum. Teacher forcing first fits local dynamics: the model sees ground-truth states and predicts one step ahead, which converges fast and nails aim/power/phase. Scheduled sampling [7] then fights exposure bias: the model is fed its own snap-projected predictions with probability 1 − p, annealed over training, so it learns to recover from its own small errors. We warm the hidden state on a short ground-truth burn-in, then free-run a 24-step horizon; a gentle schedule (final teacher-forcing probability 0.4) was necessary, as more aggressive free-running destabilized the ballistic parts of the rollout. The per-field loss is a weighted Huber in normalized space for the continuous head, BCE for cups and events, and cross-entropy for phase:

A motion mask upweights moving/flight frames (where position error compounds along the arc) and a transition mask upweights the sparse throw/sink/land ticks. Because a cup-sink is a rare one-tick 1 → 0 flip, the cups BCE additionally upweights flip frames and we oversample sink windows in the batch sampler. As Section 10 shows, these help the head fire but cannot by themselves make the game sink, because the problem lies upstream in the predicted ball.

7Neural Rendering: The State-Grounded Latent Renderer

The visual stage learns s → frame so the game can be drawn without the engine's renderer. A conv decoder asked to invent a 128×96 scene from a 21-vector produces blotchy felt and garbled HUD text. The fix is to ground the decoder in geometry it should not have to hallucinate. We rasterize the state into a stack of 16 hint channels using the PyTorch port of the camera projection: Gaussian blobs for the ball and its shadow, filled disks for each present cup (gated by the bitmask), the aim reticle and power bar, a phase one-hot broadcast, and, crucially, three channels of a static empty-court backdrop. The decoder is a small upsampling conv net (261,891 parameters) with FiLM conditioning [8] from the state vector, trained with a foreground-weighted L₁ loss. Adding the backdrop hint channels alone cut held-out reconstruction from L₁ = 0.048 to 0.0077.

8Inference Modes

In F2 the network alone advances the structured state under live controller input; in F3 both the engine and its renderer are off and the game you see and play is produced entirely by the two networks. The controls, HUD, and win condition are identical across modes.

9Results

9.1Control and short-horizon dynamics

On held-out seeds the learned dynamics is excellent at everything except ballistic landing (Table 3). One-step aim and power errors are 0.0025 and 0.0013 on their [−1,1] / [0,1] scales; phase transitions are 99.4% correct; the throw counter is exact; and controllability (does the world move the way the button says?) is a perfect 100% directional-response rate for aim, power, and throw-launch. Subjectively this is what makes F2/F3 feel like the game rather than like a video of it.

9.2Long-horizon drift

Free-run rollouts stay legal indefinitely (snap guarantees it: invariant rate 1.00) but drift in continuous value: positional RMSE grows to ≈22u by 30 steps, and the median trajectory measurably diverges from the engine's after ~14 steps. The whole-game score MAE against the engine is ≈2.8 cups. The dominant contributor to that drift is exactly the flight phase.

10The Flight-Precision Wall

Symptom. Played in F2/F3, the game felt right until you tried to score: the ball would arc plausibly and the felt, cups, and HUD all rendered cleanly, but cups almost never sank. Instrumenting the make-rate confirmed it: the fully-learned mode sank roughly 3% of well-aimed throws, against the engine's 100%.

Diagnosis. We first blamed the sink resolution and attacked it three ways: upweighting the rare cup-flip in the BCE by 25×; oversampling sink windows; and adding a rule-based sink to the snap operator that drops a cup whenever the model's predicted descending ball passes within SINK_RADIUS of a present cup. None moved the make-rate. Tracing an individual throw explained why:

This is a capacity limit, not a bug. Localizing a 60 Hz ballistic landing over an ~85-unit throw to within a 3.3-unit radius demands that a 409K recurrent model integrate a fast arc almost exactly, and small errors in the predicted launch velocity, compounded over the flight, land well outside the cup. The same model that tracks aim to 0.0025 cannot pin the landing, because landing precision is a far harder function of the launch than direction-following is.

Fix: the hybrid world model. The resolution keeps the network where it is strong and defers to physics where it is weak. At release, the ball's velocity is seeded from the network's own aim and power (which it tracks to ≈0.002, so the seed is accurate) and the arc is integrated with exact, deterministic ballistics and the sink test, while the GRU keeps driving aim, power, and the phase machine, and the decoder keeps rendering:

This "hybrid flight" mode (F3) sinks 6/6 cups, matching the engine exactly, and a regression test locks in engine-parity sinking for every cup in the rack. The pure-learned F2 mode is retained as the honest artifact: it shows what the structured dynamics does and does not learn, flight wall included. The concession is stated plainly: in F3 the ball's arc is real physics, not learned; everything else is neural, including the interactive control loop, the phase/score logic, and 100% of the pixels.

11Discussion, Limitations, and Future Work

• The wall is the finding. The interesting result is not that a tiny world model works; it is precisely where it stops working. Control-following and discrete state logic are cheap to learn; sub-cup ballistic localization is not. A world model's weakest link is whichever variable is a sensitive, compounding function of an early continuous choice.
• Could a pure model clear the wall? Plausibly, via a launch-velocity representation that predicts the whole arc from release conditions in closed form rather than integrating step-by-step, or simply more capacity. Both are worth trying, but neither is "small."
• Rendering is the easy, satisfying part. The state-grounded decoder felt most like magic and was least troublesome, because geometry hints do the heavy lifting; the network only has to shade and antialias.
• Drift. Even with legal-state guarantees, continuous drift over long rollouts remains; a learned correction term or periodic re-grounding could extend coherent horizons.

12Conclusion

Neural Cup Pong is a fully playable arcade game in which, at the strongest setting, no game engine and no graphics renderer run: a 409K-parameter recurrent dynamics model driven by a legality-preserving projection handles control and state, and a 262K-parameter state-grounded decoder paints every frame. The build is honest about its one hard limit, that a small structured model cannot localize a ballistic landing to within a cup, and resolves it with a hybrid that seeds exact flight from the network's own accurate control, restoring engine-parity sinking while keeping control and rendering neural. The lesson generalizes past cup pong: when you shrink a world model, the first thing to break is the variable that is a sensitive, compounding function of a continuous action, and the pragmatic fix is to learn everything else and hand that one variable back to physics.

 References

1. D. Ha, J. Schmidhuber. World Models. 2018. arXiv:1803.10122.
2. D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap. Mastering Diverse Domains through World Models (DreamerV3). 2023. arXiv:2301.04104.
3. E. Alonso, A. Jelley, V. Micheli et al. Diffusion for World Modeling: Visual Details Matter in Atari (DIAMOND). NeurIPS 2024. arXiv:2405.12399.
4. D. Valevski, Y. Leviathan, M. Arar, S. Fruchter. Diffusion Models Are Real-Time Game Engines (GameNGen). 2024. arXiv:2408.14837.
5. J. Bruce, M. Dennis, A. Edwards, J. Parker-Holder et al. Genie: Generative Interactive Environments. ICML 2024. arXiv:2402.15391.
6. K. Cho, B. van Merriënboer, C. Gulcehre et al. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (GRU). EMNLP 2014. arXiv:1406.1078.
7. S. Bengio, O. Vinyals, N. Jaitly, N. Shazeer. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. NeurIPS 2015. arXiv:1506.03099.
8. E. Perez, F. Strub, H. de Vries, V. Dumoulin, A. Courville. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI 2018. arXiv:1709.07871.