Taking Reinforcement-Learned Motions from Simulation to Real Robots

1. Anything Is Possible in Simulation. The Problem Is What Comes After

Creating robot motion with reinforcement learning: today, with a single GPU, you can simulate thousands of robots in parallel and teach one to do a backflip in a few hours.

But when you put a policy that worked perfectly in simulation onto a real robot, it usually falls over on its very first step. The mass of the joints, the torque the motors can actually deliver, the few milliseconds of delay before a command arrives. Every real-world detail that the simulation quietly ignores comes back to bite you.

Our job is exactly this: turning "motion inside a simulation" into "motion on a real robot." Over roughly five months starting in June 2026, we achieved the following motions on a real, commercially available Unitree Go2 quadruped:

• Galloping (high-speed running with a specified footfall sequence)
• Hind-leg standing (rising up on the two hind legs)
• Backflips
• Jumping
• Wall climbing from a handstand (5 consecutive runs without a fall; 2/2 on a 30 cm wall)
• Self-righting (recovering unaided from lying on its back or side)

On the Unitree G1 humanoid as well, 5 of the 6 single motions trained in simulation run on the real robot just as they did in simulation (including walking, running, squat walking, kung fu, and a spinning kick).

Training and verifying Unitree Go2 policies in simulation. From top left: parallel training in Isaac Sim (jump), hind-leg stand in MuJoCo, parallel training on stair terrain in Isaac Sim, and crouched walking in MuJoCo (red dots are height-map sample points)

Unitree G1 whole-body motion imitation policy (MuJoCo)

2. The Pipeline: Training → Sim-to-Sim → Real Robot

① Training ── Isaac Sim / Isaac Lab
   PPO with thousands of parallel environments; reward design, curricula, domain randomization
      ↓
② Sim-to-sim ── Re-verify in MuJoCo with the same C++ controller as the real robot
   "Does it behave the same in a different physics engine?"
      ↓
③ Real robot ── Convert to ONNX → 50 Hz inference on the robot's onboard Jetson
   Staged testing (suspended → flat ground → target motion)

Our training stack is Isaac Sim 5.1 / Isaac Lab 2.3 / rsl-rl, running on our in-house on-premises GPU servers (5 in total). To give a sense of scale, training with 4,096 parallel environments × 3,000 iterations takes 2.5 to 3.5 hours, and training 40 full-body humanoid motion-imitation clips takes about 2 hours on a single GPU.

On the robot side, the trained policy is converted to ONNX and run by a C++ controller on the robot's onboard Jetson. Policy inference takes about 0.4 ms per step, and the control loop runs at 50 Hz. Making sure the input/output definitions match exactly between the training environment and the real-robot controller is unglamorous but the single most important point (as we'll see later, a single mismatch is enough to make the robot fall over every time).

3. Reward Design: Robots Cheat

A reinforcement learning robot will do whatever it takes to maximize its reward. Here are a few of the "loopholes" we ran into during development.

Climbing stairs backward. Because the stair-climbing reward didn't distinguish which way the body was facing, the robot learned that it was better off turning around and climbing hind legs first. It was technically climbing. But it was useless in practice.

"Not moving" becomes the optimal solution. When we added a reward for maintaining torso height, we got the policy with the best scores on paper. Then we put it on the real robot, and it barely moved. If you stand still, you can maintain your height perfectly.

The foot-lift reward was always zero. When we used the standard "foot clearance" reward for stairs, we discovered that, due to how coordinate frames were handled, its value was always zero. We wrote our own reward term instead.

These experiences have made us strict about never judging by metrics alone. We review video after every training run, and we have explicitly instructed the AI agent that auto-tunes rewards that it is "prohibited from altering metrics to produce superficially good results."

On the other hand, some ideas worked well:

• Galloping: Explicitly command the footfall timing (phase) of each leg and reward how closely it is followed
• Jumping: Torque logs showed that the knee motors were using less than 70% of their capacity. By adjusting the reward to encourage the knees to work harder, jump height went from 40 cm → 65 cm (in simulation)
• Long jump: Without providing any reference trajectory at all, we used the "jump distance" computed from physics equations directly as the objective function

4. Curricula: Training Wheels On, Then Off

For motions like backflips and cartwheels, if you try to train them from the start, the robot has no experience of success and tends to settle on "doing nothing." So we borrowed the same idea used in a human gymnastics class.

• Assistance via external force: When training the gallop, we initially "pull" the torso forward with up to 150 N. Each time the success rate exceeds 60%, we reduce the assistance a little, until it reaches zero. For double cartwheels and backflips too, we first let the robot "experience" the motion with external force, then had it learn to do it unaided.
• Graduated difficulty: Flat ground → rough terrain → stairs → walls, with terrain automatically getting harder as the robot improves. For wall climbing, training in the order "flat ground → rough terrain → wall" produced better accuracy.
• Starting partway through: For wall climbing, we started 30% of episodes from a "midway up the wall" pose, so the robot could concentrate on gaining experience in the hardest part.

5. Closing the Sim-to-Real Gap Through Measurement

There is only so much you can absorb by "vaguely randomizing parameters (domain randomization)" when something doesn't work on the real robot. We were determined to identify the true source of each discrepancy, one at a time, through actual measurement.

We also ran measurements with the torso suspended, feeding signals of various frequencies into the joints and identifying joint inertia and friction from their responses, and incorporated the results into training.

For example, the front-leg handstand initially failed on the real robot. When we investigated, we found that a command delay of just 8–10 ms was the deciding factor. Because delays of a few milliseconds don't exist in simulation, the more delicate the balancing motion, the more it is affected.

"Isaac Performance Does Not Predict MuJoCo Success"

Another lesson was the importance of sim-to-sim. Time and again, a policy that could clear a 45 cm wall in Isaac Lab could barely manage 40 cm in MuJoCo. Different physics engines compute contact and friction differently. If it doesn't pass in a second physics engine, it almost certainly won't pass on the real robot, and we use this as a gate.

Sim-to-sim check: the same walking policy running in Isaac Sim (left) and MuJoCo (right) (Unitree G1)

There was also a case where the denominator used to normalize the jump command differed between training and deployment (1.5 seconds vs. 2.2 seconds), and the robot fell on every single trial. To prevent this kind of "configuration drift," we have built checks of ONNX input dimensions and hash values into our real-robot testing procedure.

Go2 get-up policy before (left) and after (right) retraining with physics parameters matched to the real robot (MuJoCo)

6. Systems for Running Real Robots Safely

Testing acrobatic motions on a real robot can destroy the hardware with a single misstep. That is why we build safety measures into both our software and our operations.

• Staged testing: Suspended → walking → standing up → target motion → gradually increasing height. Each stage has pass/abort criteria
• Two independent emergency stops: A remote controller (which does not go through the network) and a keyboard. The operator and the person responsible for stopping are separate
• Automatic power-off: Automatic stop when tilt exceeds a threshold, when the robot fails to land within a set time, or when output values are abnormal
• Startup checks: The robot won't start without a confirmation input, and unexpected height settings are rejected

In addition, we build our own supporting tools to keep development moving, such as an Android app for controlling the robot with a gamepad while watching the camera feed, a monitoring tool for our GPU server cluster, and a bot that notifies us when training finishes.

7. Conclusion

Reinforcement learning-based robot control looks glamorous in papers and demo videos, but reproducing it on a real robot involves a gritty process of eliminating "unglamorous discrepancies" (latency, torque, range of motion, control-loop timing) one by one. Open Reach Tech runs that process with the quality-management mindset of a software development company: centralized configuration, tests as gates, and standardized procedures.

• Building training and validation infrastructure for motion policies with Isaac Lab / MuJoCo
• Measuring the sim-to-real gap and deploying to real robots (Jetson / ONNX / C++)
• Developing new motions through reward and curriculum design

If you want to bring "motion" to your own robots, or you have policies that work in simulation but not on the real hardware, we would love to hear from you.

Also take a look at our case study on validating robot traversability in a digital twin: "Validating Robot Walking in a Shipyard Digital Twin."