LeRobot documentation
NVIDIA IsaacLab 竞技场 & LeRobot
NVIDIA IsaacLab 竞技场 & LeRobot
LeRobot EnvHub 现已支持通过 IsaacLab Arena 进行GPU 加速仿真,以实现大规模策略评估。 在高保真仿真中训练和评估模仿学习策略——全部集成于 LeRobot 生态系统中。
IsaacLab Arena 与 NVIDIA IsaacLab 集成,提供以下功能:
- 🤖 人形机器人本体:GR1、G1、Galileo 及其多种配置
- 🎯 操作与移动操作任务:开门、抓取与放置、按键等
- ⚡ GPU 加速的 rollout:在 NVIDIA GPU 上并行执行环境
- 🖼️ RTX 渲染:使用逼真的渲染、反射和折射来评估基于视觉的策略
- 📦 兼容 LeRobot 的数据集:可直接用于 GR00T N1x、PI0、SmolVLA、ACT 和 Diffusion 策略的训练
- 🔄 EnvHub 集成:通过一行代码从 HuggingFace EnvHub 加载环境
安装
先决条件
硬件要求与 Isaac Sim 相同,详细信息请参阅 Isaac Sim 要求。
- 支持 CUDA 的 NVIDIA GPU
- 与 IsaacSim 5.1.0 兼容的 NVIDIA 驱动程序
- Linux(Ubuntu 22.04 / 24.04)
设置
# 1. Create conda environment
conda create -y -n lerobot-arena python=3.11
conda activate lerobot-arena
conda install -y -c conda-forge ffmpeg=7.1.1
# 2. Install Isaac Sim 5.1.0
pip install "isaacsim[all,extscache]==5.1.0" --extra-index-url https://pypi.nvidia.com
# Accept NVIDIA EULA (required)
export ACCEPT_EULA=Y
export PRIVACY_CONSENT=Y
# 3. Install IsaacLab 2.3.0
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab
git checkout v2.3.0
./isaaclab.sh -i
cd ..
# 4. Install IsaacLab Arena
git clone https://github.com/isaac-sim/IsaacLab-Arena.git
cd IsaacLab-Arena
git checkout release/0.1.1
pip install -e .
cd ..
# 5. Install LeRobot (evaluation extra for env/policy evaluation)
git clone https://github.com/huggingface/lerobot.git
cd lerobot
pip install -e ".[evaluation]"
# 6. Install additional dependencies
pip install onnxruntime==1.23.2 lightwheel-sdk==1.0.1 vuer[all]==0.0.70 qpsolvers==4.8.1
pip install numpy==1.26.0 # Isaac Sim 5.1 depends on numpy==1.26.0, this will be fixed in next release评估策略
预训练策略
以下已训练的策略可供使用:
| 策略 | 架构 | 任务 | 链接 |
|---|---|---|---|
| pi05-arena-gr1-microwave | PI0.5 | GR1 微波 | HuggingFace |
| smolvla-arena-gr1-microwave | SmolVLA | GR1 微波 | HuggingFace |
评估 SmolVLA
pip install -e ".[smolvla]"
pip install numpy==1.26.0 # revert numpy to version 1.26lerobot-eval \
--policy.path=nvidia/smolvla-arena-gr1-microwave \
--env.type=isaaclab_arena \
--env.hub_path=nvidia/isaaclab-arena-envs \
--rename_map='{"observation.images.robot_pov_cam_rgb": "observation.images.robot_pov_cam"}' \
--policy.device=cuda \
--env.environment=gr1_microwave \
--env.embodiment=gr1_pink \
--env.object=mustard_bottle \
--env.headless=false \
--env.enable_cameras=true \
--env.video=true \
--env.video_length=10 \
--env.video_interval=15 \
--env.state_keys=robot_joint_pos \
--env.camera_keys=robot_pov_cam_rgb \
--trust_remote_code=True \
--eval.batch_size=1评估 PI0.5
pip install -e ".[pi]"
pip install numpy==1.26.0 # revert numpy to version 1.26PI0.5 requires disabling torch compile for evaluation:
TORCH_COMPILE_DISABLE=1 TORCHINDUCTOR_DISABLE=1 lerobot-eval \
--policy.path=nvidia/pi05-arena-gr1-microwave \
--env.type=isaaclab_arena \
--env.hub_path=nvidia/isaaclab-arena-envs \
--rename_map='{"observation.images.robot_pov_cam_rgb": "observation.images.robot_pov_cam"}' \
--policy.device=cuda \
--env.environment=gr1_microwave \
--env.embodiment=gr1_pink \
--env.object=mustard_bottle \
--env.headless=false \
--env.enable_cameras=true \
--env.video=true \
--env.video_length=15 \
--env.video_interval=15 \
--env.state_keys=robot_joint_pos \
--env.camera_keys=robot_pov_cam_rgb \
--trust_remote_code=True \
--eval.batch_size=1To change the number of parallel environments, use the ```--eval.batch_size``` flag.
预期内容
在评估过程中,您会看到一个显示当前成功率的进度条:
Stepping through eval batches: 8%|██████▍ | 4/50 [00:45<08:06, 10.58s/it, running_success_rate=25.0%]视频录制
要在评估期间启用视频录制,请将以下标志添加到您的命令中:
--env.video=true \
--env.video_length=15 \
--env.video_interval=15有关视频录制的更多详细信息,请参阅 IsaacLab 录制文档。
When running headless with `--env.headless=true`, you must also enable cameras explicitly for camera enabled environments:--env.headless=true --env.enable_cameras=true
输出目录
评估视频将按以下结构保存到输出目录:
outputs/eval/<date>/<timestamp>_<env>_<policy>/videos/<task>_<env_id>/eval_episode_<n>.mp4例如:
outputs/eval/2026-01-02/14-38-01_isaaclab_arena_smolvla/videos/gr1_microwave_0/eval_episode_0.mp4训练策略
要了解如何使用 LeRobot 训练策略,请参阅训练文档:
HuggingFace Hub 上提供了 IsaacLab Arena 的示例数据集,供实验使用:
| 数据集 | 描述 | 帧 |
|---|---|---|
| Arena-GR1-Manipulation-Task | GR1 微波操控 | ~4K |
| Arena-G1-Loco-Manipulation-Task | G1 移动操作 | ~4K |
环境配置
完整配置选项
from lerobot.envs.configs import IsaaclabArenaEnvConfig
config = IsaaclabArenaEnvConfig(
# Environment selection
environment="gr1_microwave", # Task environment
embodiment="gr1_pink", # Robot embodiment
object="power_drill", # Object to manipulate
# Simulation settings
episode_length=300, # Max steps per episode
headless=True, # Run without GUI
device="cuda:0", # GPU device
seed=42, # Random seed
# Observation configuration
state_keys="robot_joint_pos", # State observation keys (comma-separated)
camera_keys="robot_pov_cam_rgb", # Camera observation keys (comma-separated)
state_dim=54, # Expected state dimension
action_dim=36, # Expected action dimension
camera_height=512, # Camera image height
camera_width=512, # Camera image width
enable_cameras=True, # Enable camera observations
# Video recording
video=False, # Enable video recording
video_length=100, # Frames per video
video_interval=200, # Steps between recordings
# Advanced
mimic=False, # Enable mimic mode
teleop_device=None, # Teleoperation device
disable_fabric=False, # Disable fabric optimization
enable_pinocchio=True, # Enable Pinocchio for IK
)直接用于高级用途的 Environment Hub
创建一个名为 test_env_load_arena.py 的文件,或 从 EnvHub 下载:
import logging
from dataclasses import asdict
from pprint import pformat
import torch
import tqdm
from lerobot.configs import parser
from lerobot.configs.eval import EvalPipelineConfig
@parser.wrap()
def main(cfg: EvalPipelineConfig):
"""Run random action rollout for IsaacLab Arena environment."""
logging.info(pformat(asdict(cfg)))
from lerobot.envs import make_env
env_dict = make_env(
cfg.env,
n_envs=cfg.env.num_envs,
trust_remote_code=True,
)
env = next(iter(env_dict.values()))[0]
env.reset()
for _ in tqdm.tqdm(range(cfg.env.episode_length)):
with torch.inference_mode():
actions = env.action_space.sample()
obs, rewards, terminated, truncated, info = env.step(actions)
if terminated.any() or truncated.any():
obs, info = env.reset()
env.close()
if __name__ == "__main__":
main()运行命令:
python test_env_load_arena.py \
--env.environment=g1_locomanip_pnp \
--env.embodiment=gr1_pink \
--env.object=cracker_box \
--env.num_envs=4 \
--env.enable_cameras=true \
--env.seed=1000 \
--env.video=true \
--env.video_length=10 \
--env.video_interval=15 \
--env.headless=false \
--env.hub_path=nvidia/isaaclab-arena-envs \
--env.type=isaaclab_arena创建新环境
首先,按照 IsaacLab Arena 文档 创建一个新的 IsaacLab Arena 环境。
克隆我们的 EnvHub 仓库:
git clone https://huggingface.co/nvidia/isaaclab-arena-envs根据新环境修改 example_envs.yaml 文件。上传 修改后的仓库到 HuggingFace EnvHub。
Your IsaacLab Arena environment code must be locally available during evaluation. Users can clone your environment repository separately, or you can bundle the environment code and assets directly in your EnvHub repo.
然后,在评估时,使用你的新环境:
lerobot-eval \
--env.hub_path=<your-env-hub-path>/isaaclab-arena-envs \
--env.environment=<your new environment> \
...other flags...我们期待您的贡献!
故障排除
CUDA 内存不足
减小 batch_size 或使用显存更大的 GPU:
--eval.batch_size=1
未接受 EULA
在运行之前设置环境变量:
export ACCEPT_EULA=Y
export PRIVACY_CONSENT=Y视频录制无法正常工作
在无头模式下运行时启用摄像头:
--env.video=true --env.enable_cameras=true --env.headless=true策略输出维度不匹配
确保 action_dim 与您的策略相匹配:
--env.action_dim=36
Isaac Sim 初始化期间出现的 libGLU.so.1 错误
请确保已安装以下依赖项,这种情况通常发生在无头机器上。
sudo apt update && sudo apt install -y libglu1-mesa libxt6另请参阅
- EnvHub 文档 - EnvHub 通用用法
- IsaacLab Arena GitHub
- IsaacLab 文档
Lightwheel LW-BenchHub
Lightwheel 将包含 268 个任务的 Lightwheel-Libero-Tasks 和 Lightwheel-RoboCasa-Tasks 引入 LeRobot 生态系统。
LW-BenchHub 通过遥操作收集和生成符合 LeRobot 规范的大规模数据集,支持开箱即用的训练和评估工作流。
借助 EnvHub 提供的统一接口,开发者可以快速构建端到端的实验流水线。
安装
假设您已按照安装步骤操作,您可以通过以下方式安装 LW-BenchHub:
conda install pinocchio -c conda-forge -y
pip install numpy==1.26.0 # revert numpy to version 1.26
sudo apt-get install git-lfs && git lfs install
git clone https://github.com/LightwheelAI/lw_benchhub
cd lw_benchhub
git lfs pull # Ensure LFS files (e.g., .usd assets) are downloaded
pip install -e .有关更详细的说明,请参阅 LW-BenchHub 文档。
Lightwheel 任务数据集
LW-BenchHub 数据集可在 HuggingFace Hub 上获取:
| 数据集 | 描述 | 任务 | 帧 |
|---|---|---|---|
| Lightwheel-Tasks-X7S | X7S LIBERO 和 RoboCasa | 117 | ~10.3M |
| Lightwheel-Tasks-Double-Piper | Double-Piper LIBERO | 130 | ~6.0M |
| Lightwheel-Tasks-G1-Controller | G1-Controller LIBERO | 62 | ~2.7M |
| Lightwheel-Tasks-G1-WBC | G1-WBC RoboCasa | 32 | ~1.5M |
有关训练策略,请参阅 训练策略 部分。
评估策略
预训练策略
以下已训练的策略可供使用:
| 策略 | 架构 | 任务 | 布局 | 机器人 | 链接 |
|---|---|---|---|---|---|
| smolvla-double-piper-pnp | SmolVLA | L90K1PutTheBlackBowlOnThePlate | libero-1-1 | DoublePiper-Abs | HuggingFace |
评估 SmolVLA
lerobot-eval \
--policy.path=LightwheelAI/smolvla-double-piper-pnp \
--env.type=isaaclab_arena \
--rename_map='{"observation.images.left_hand_camera_rgb": "observation.images.left_hand", "observation.images.right_hand_camera_rgb": "observation.images.right_hand", "observation.images.first_person_camera_rgb": "observation.images.first_person"}' \
--env.hub_path=LightwheelAI/lw_benchhub_env \
--env.kwargs='{"config_path": "configs/envhub/example.yml"}' \
--trust_remote_code=true \
--env.state_keys=joint_pos \
--env.action_dim=12 \
--env.camera_keys=left_hand_camera_rgb,right_hand_camera_rgb,first_person_camera_rgb \
--policy.device=cuda \
--eval.batch_size=10 \
--eval.n_episodes=100环境配置
通过修改配置文件中的 robot、task 和 layout 设置,即可快速启动评估。
完整配置选项
# =========================
# Basic Settings
# =========================
disable_fabric: false
device: cuda:0
sensitivity: 1.0
step_hz: 50
enable_cameras: true
execute_mode: eval
episode_length_s: 20.0 # Episode length in seconds, increase if episodes timeout during eval
# =========================
# Robot Settings
# =========================
robot: DoublePiper-Abs # Robot type, DoublePiper-Abs, X7S-Abs, G1-Controller or G1-Controller-DecoupledWBC
robot_scale: 1.0
# =========================
# Task & Scene Settings
# =========================
task: L90K1PutTheBlackBowlOnThePlate # Task name
scene_backend: robocasa
task_backend: robocasa
debug_assets: null
layout: libero-1-1 # Layout and style ID
sources:
- objaverse
- lightwheel
- aigen_objs
object_projects: []
usd_simplify: false
seed: 42
# =========================
# Object Placement Retry Settings
# =========================
max_scene_retry: 4
max_object_placement_retry: 3
resample_objects_placement_on_reset: true
resample_robot_placement_on_reset: true
# =========================
# Replay Configuration Settings
# =========================
replay_cfgs:
add_camera_to_observation: true
render_resolution: [640, 480]