STEAM Learns Which Moments Matter in Robot Experience

Imagine a robot folding a towel. It begins well: grasping an edge and pulling the cloth flat. Then the grasp drifts, wrinkles form, and the policy stalls. A few seconds later, a human intervenes, restores the towel to a workable state, and guides the task back toward success.

Should the robot learn from this trajectory—or discard it?

The hard question is how to use it. Discarding the entire trajectory means losing both the useful early progress and the later human recovery. Keeping it, however, risks the robot to imitate the failed actions in between. This is what makes real-world robot data usage challenging: progress, hesitation, failure, and recovery can all occur within the same episode.

As embodied AI moves closer to real-world deployment, policy improvement increasingly depends on learning from heterogeneous data. Offline reinforcement learning is one natural fit, but it needs a dense signal that separates useful frames from stalled or regressive ones. Human frame-level annotation does not scale.

Researchers from Chao Yu's team at Tsinghua University and Striding AI propose Self-Supervised Temporal Ensemble Advantage Modeling (STEAM), a label-free method for learning frame-level advantages from expert demonstrations.

STEAM learns a progress signal from the temporal order of expert frames, then uses that signal to score expert data, robot rollouts, and human corrections frame by frame.

The method learns this signal without manually designed rewards or frame-level human labels. It uses the temporal structure inherently present in successful demonstrations, then applies what it learns to mixed-quality data. This signal learning corresponds to a critical question: did this local transition move the task forward?

A frame-level learning signal should reflect how a roboticist would assess task progress at each moment in the video.

In expert demonstrations, the learning signal should stay high for most of the episode and dip only around retries. In successful rollouts, it may be lower and noisier. In failed rollouts, it should drop once the robot gets stuck. In human-correction data, it should recover after the human takes over.

STEAM learns exactly this kind of signal. The visualization below synchronizes four trajectory types. Each panel shows robot video on top and the live frame-level advantage signal A_STEAM below, with a red point marking the current frame.

This visualization shows how STEAM works. The model is not assigning a single label to the whole trajectory. It interprets what happens inside the trajectory: when the robot makes progress, when it begins to make mistakes, and when human intervention restores the state.

That matters for real-robot learning. Valuable training signals often appear in messy places: the useful approach before a failure, the recovery after a failed attempt, and the corrective motion after human takeover.

Many robot-learning pipelines divide experience into successful and failed episodes. This can be adequate for short tasks, but it becomes too coarse for long-horizon manipulation.

Towel folding, for example, requires a sequence of coordinated stages: grasping the cloth, flattening it, repositioning the grippers, and completing multiple folds. A small error in one stage does not make every preceding frame useless. If training keeps only fully successful episodes, it loses useful examples of partial progress, mistakes, and corrective actions from unsuccessful rollouts. If it keeps all rollouts equally, the policy may be pulled toward failure modes.

STEAM changes the unit of analysis from the episode to the frame. The relevant question becomes: how much did this local action segment contribute to task progress?

STEAM can avoid manual rewards because successful demonstrations already carry a free supervisory signal: temporal order.

Within an expert trajectory, later frames are usually closer to task completion than earlier frames. The relative position of two frames can therefore act as a proxy for local progress. Presenting the pair in reverse provides a corresponding signal for regression.

STEAM trains an ensemble of temporal-offset predictors on frame pairs sampled from expert trajectories. Given two observations and a language instruction, each predictor estimates a probability distribution over discrete temporal-offset bins—how far forward or backward the second frame lies relative to the first.

After training, STEAM estimates advantage by comparing predicted local progress against the expected progress for the same normalized time gap. The score is high when the transition advances the task more efficiently than the baseline. It falls when the robot stalls or regresses.

Real rollouts often contain states that may be far from the expert demonstration distribution. A single predictor can be confidently wrong and assign high value to a harmful transition.

STEAM reduces this risk with a small ensemble of independently initialized predictors. The models tend to agree on familiar states but may diverge on unusual ones. Instead of averaging their estimates, STEAM uses a conservative aggregation rule by taking the minimum advantage across the ensemble, which suppresses optimistic false positives.

The goal is not to make the policy conservative. It is to avoid amplifying bad experiences during training. In robot learning, a false positive can be more damaging than missing some useful segments, because a harmful action scored as useful can push the policy into the wrong direction.

STEAM uses π₀ as the VLA backbone [1] and connects its frame-level advantage signal to policy training through CFGRL [2]. The clearest result comes from real-world towel folding.

Behavior cloning from expert data alone reaches 33.3% success on towel folding. STEAM, trained on a mixture of expert demonstrations, human corrections, and rollouts, reaches 92.3%, an improvement of 59.0 percentage points. It also improves stage-completion score and throughput, which shows the gain comes from both performance and efficiency.

On the shorter pick-and-place task, behavior cloning from expert data reaches 63.8%. After rollouts are added, STEAM turns frame-level advantages into optimality labels and guides inference toward high-optimality conditions, reaching 80.0% success.

Against RECAP [4] under the same mixed-data setting, STEAM is consistently stronger on the real-robot tasks reported by the project: 92.3% versus 55.6% on towel folding, 93.8% versus 53.3% on chip checkout, 75.0% versus 52.9% on cola restocking, and 80.0% versus 53.8% on pick-and-place. This is where STEAM is most useful: long-horizon, mixed-quality tasks with local failures and recoveries.

STEAM is most valuable for datasets that resemble real deployment: clean demonstrations mixed with imperfect rollouts, slow successes, failed attempts, and human interventions.

There is also an important boundary case. On a simple pick-and-place task with only a small amount of clean expert data, adding advantage labels and conditional guidance can underperform standard behavior cloning. When the task is short, the demonstrations are clean, and the dataset is small, frame-level preference signals may provide little additional benefit.

The advantage of STEAM becomes clearer once real policy rollouts are added to the training set. It is designed for this messier setting, where the policy must learn from the useful portions of imperfect experience without reproducing the actions that lead to failure.

While other methods first learn a reward or value function, then convert it into an advantage. STEAM takes a more direct route by learning local progress.

This approach offers three practical properties:

• It does not require reward engineering or frame-level human annotation.
• It does not rely on a general-purpose VLM to judge physical task progress zero-shot.
• It does not assume task progress is monotonic, which lets it represent regressions, retries, and recoveries inside real trajectories.

Two technical choices matter. First, finer temporal bins help the model distinguish different degrees of progress and regression. On towel folding with full data, success rises from 27.3% with “N=2” bins to 54.6% with “N=8” and 92.3% with the default “N=32”. Second, a small ensemble is enough to reduce overestimation on out-of-distribution rollout states. Moving from “M=1” to the default “M=3” improves success from 72.7% to 92.3%, while “M=5” adds no further benefit.

STEAM still has a few limitations. It relies primarily on visual observations, so it can miss contact states or proprioceptive errors that are difficult to detect. It also uses temporal offset as a proxy for task progress, while real tasks may assign different importance to different stages.

Natural next steps include stage-aware progress modeling, integrating robot proprioception, and validating self-supervised frame-level advantage signals across more real tasks and robot platforms.

The motivation behind STEAM is simple: robots need more experience, and they also need to distinguish which parts of that experience are worth learning from. By recognizing meaningful progress in failed attempts, retries, and human corrections, STEAM helps policies learn more effectively from the imperfect data generated during real-world robot training.