LeRobot documentation

NVIDIA IsaacLab 竞技场 & LeRobot

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

NVIDIA IsaacLab 竞技场 & LeRobot

LeRobot EnvHub 现已支持通过 IsaacLab Arena 进行GPU 加速仿真,以实现大规模策略评估。 在高保真仿真中训练和评估模仿学习策略——全部集成于 LeRobot 生态系统中。

IsaacLab Arena - GR1 Microwave Environment

IsaacLab Arena 与 NVIDIA IsaacLab 集成,提供以下功能:

  • 🤖 人形机器人本体:GR1、G1、Galileo 及其多种配置
  • 🎯 操作与移动操作任务:开门、抓取与放置、按键等
  • ⚡ GPU 加速的 rollout:在 NVIDIA GPU 上并行执行环境
  • 🖼️ RTX 渲染:使用逼真的渲染、反射和折射来评估基于视觉的策略
  • 📦 兼容 LeRobot 的数据集:可直接用于 GR00T N1x、PI0、SmolVLA、ACT 和 Diffusion 策略的训练
  • 🔄 EnvHub 集成:通过一行代码从 HuggingFace EnvHub 加载环境

安装

先决条件

硬件要求与 Isaac Sim 相同,详细信息请参阅 Isaac Sim 要求。

  • 支持 CUDA 的 NVIDIA GPU
  • 与 IsaacSim 5.1.0 兼容的 NVIDIA 驱动程序
  • Linux(Ubuntu 22.04 / 24.04)

设置

# 1. Create conda environment
conda create -y -n lerobot-arena python=3.11
conda activate lerobot-arena
conda install -y -c conda-forge ffmpeg=7.1.1

# 2. Install Isaac Sim 5.1.0
pip install "isaacsim[all,extscache]==5.1.0" --extra-index-url https://pypi.nvidia.com

# Accept NVIDIA EULA (required)
export ACCEPT_EULA=Y
export PRIVACY_CONSENT=Y

# 3. Install IsaacLab 2.3.0
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab
git checkout v2.3.0
./isaaclab.sh -i
cd ..

# 4. Install IsaacLab Arena
git clone https://github.com/isaac-sim/IsaacLab-Arena.git
cd IsaacLab-Arena
git checkout release/0.1.1
pip install -e .
cd ..


# 5. Install LeRobot (evaluation extra for env/policy evaluation)
git clone https://github.com/huggingface/lerobot.git
cd lerobot
pip install -e ".[evaluation]"


# 6. Install additional dependencies
pip install onnxruntime==1.23.2 lightwheel-sdk==1.0.1 vuer[all]==0.0.70 qpsolvers==4.8.1
pip install numpy==1.26.0 # Isaac Sim 5.1 depends on numpy==1.26.0, this will be fixed in next release

评估策略

预训练策略

以下已训练的策略可供使用:

策略架构任务链接
pi05-arena-gr1-microwavePI0.5GR1 微波HuggingFace
smolvla-arena-gr1-microwaveSmolVLAGR1 微波HuggingFace

评估 SmolVLA

pip install -e ".[smolvla]"
pip install numpy==1.26.0 # revert numpy to version 1.26
lerobot-eval \
    --policy.path=nvidia/smolvla-arena-gr1-microwave \
    --env.type=isaaclab_arena \
    --env.hub_path=nvidia/isaaclab-arena-envs \
    --rename_map='{"observation.images.robot_pov_cam_rgb": "observation.images.robot_pov_cam"}' \
    --policy.device=cuda \
    --env.environment=gr1_microwave \
    --env.embodiment=gr1_pink \
    --env.object=mustard_bottle \
    --env.headless=false \
    --env.enable_cameras=true \
    --env.video=true \
    --env.video_length=10 \
    --env.video_interval=15 \
    --env.state_keys=robot_joint_pos \
    --env.camera_keys=robot_pov_cam_rgb \
    --trust_remote_code=True \
    --eval.batch_size=1

评估 PI0.5

pip install -e ".[pi]"
pip install numpy==1.26.0 # revert numpy to version 1.26
PI0.5 requires disabling torch compile for evaluation:
TORCH_COMPILE_DISABLE=1 TORCHINDUCTOR_DISABLE=1 lerobot-eval \
    --policy.path=nvidia/pi05-arena-gr1-microwave \
    --env.type=isaaclab_arena \
    --env.hub_path=nvidia/isaaclab-arena-envs \
    --rename_map='{"observation.images.robot_pov_cam_rgb": "observation.images.robot_pov_cam"}' \
    --policy.device=cuda \
    --env.environment=gr1_microwave \
    --env.embodiment=gr1_pink \
    --env.object=mustard_bottle \
    --env.headless=false \
    --env.enable_cameras=true \
    --env.video=true \
    --env.video_length=15 \
    --env.video_interval=15 \
    --env.state_keys=robot_joint_pos \
    --env.camera_keys=robot_pov_cam_rgb \
    --trust_remote_code=True \
    --eval.batch_size=1
To change the number of parallel environments, use the ```--eval.batch_size``` flag.

预期内容

在评估过程中,您会看到一个显示当前成功率的进度条:

Stepping through eval batches:   8%|██████▍    | 4/50 [00:45<08:06, 10.58s/it, running_success_rate=25.0%]

视频录制

要在评估期间启用视频录制,请将以下标志添加到您的命令中:

--env.video=true \
--env.video_length=15 \
--env.video_interval=15

有关视频录制的更多详细信息,请参阅 IsaacLab 录制文档。

When running headless with `--env.headless=true`, you must also enable cameras explicitly for camera enabled environments:
--env.headless=true --env.enable_cameras=true

输出目录

评估视频将按以下结构保存到输出目录:

outputs/eval/<date>/<timestamp>_<env>_<policy>/videos/<task>_<env_id>/eval_episode_<n>.mp4

例如:

outputs/eval/2026-01-02/14-38-01_isaaclab_arena_smolvla/videos/gr1_microwave_0/eval_episode_0.mp4

训练策略

要了解如何使用 LeRobot 训练策略,请参阅训练文档:

HuggingFace Hub 上提供了 IsaacLab Arena 的示例数据集,供实验使用:

数据集描述帧
Arena-GR1-Manipulation-TaskGR1 微波操控~4K
Arena-G1-Loco-Manipulation-TaskG1 移动操作~4K

环境配置

完整配置选项

from lerobot.envs.configs import IsaaclabArenaEnvConfig

config = IsaaclabArenaEnvConfig(
    # Environment selection
    environment="gr1_microwave",      # Task environment
    embodiment="gr1_pink",            # Robot embodiment
    object="power_drill",             # Object to manipulate

    # Simulation settings
    episode_length=300,               # Max steps per episode
    headless=True,                    # Run without GUI
    device="cuda:0",                  # GPU device
    seed=42,                          # Random seed

    # Observation configuration
    state_keys="robot_joint_pos",     # State observation keys (comma-separated)
    camera_keys="robot_pov_cam_rgb",  # Camera observation keys (comma-separated)
    state_dim=54,                     # Expected state dimension
    action_dim=36,                    # Expected action dimension
    camera_height=512,                # Camera image height
    camera_width=512,                 # Camera image width
    enable_cameras=True,              # Enable camera observations

    # Video recording
    video=False,                      # Enable video recording
    video_length=100,                 # Frames per video
    video_interval=200,               # Steps between recordings

    # Advanced
    mimic=False,                      # Enable mimic mode
    teleop_device=None,               # Teleoperation device
    disable_fabric=False,             # Disable fabric optimization
    enable_pinocchio=True,            # Enable Pinocchio for IK
)

直接用于高级用途的 Environment Hub

创建一个名为 test_env_load_arena.py 的文件,或 从 EnvHub 下载:

import logging
from dataclasses import asdict
from pprint import pformat
import torch
import tqdm
from lerobot.configs import parser
from lerobot.configs.eval import EvalPipelineConfig


@parser.wrap()
def main(cfg: EvalPipelineConfig):
    """Run random action rollout for IsaacLab Arena environment."""
    logging.info(pformat(asdict(cfg)))

    from lerobot.envs import make_env

    env_dict = make_env(
        cfg.env,
        n_envs=cfg.env.num_envs,
        trust_remote_code=True,
    )
    env = next(iter(env_dict.values()))[0]
    env.reset()
    for _ in tqdm.tqdm(range(cfg.env.episode_length)):
        with torch.inference_mode():
            actions = env.action_space.sample()
            obs, rewards, terminated, truncated, info = env.step(actions)
            if terminated.any() or truncated.any():
                obs, info = env.reset()
    env.close()


if __name__ == "__main__":
    main()

运行命令:

python test_env_load_arena.py \
    --env.environment=g1_locomanip_pnp \
    --env.embodiment=gr1_pink \
    --env.object=cracker_box \
    --env.num_envs=4 \
    --env.enable_cameras=true \
    --env.seed=1000 \
    --env.video=true \
    --env.video_length=10 \
    --env.video_interval=15 \
    --env.headless=false \
    --env.hub_path=nvidia/isaaclab-arena-envs \
    --env.type=isaaclab_arena

创建新环境

首先,按照 IsaacLab Arena 文档 创建一个新的 IsaacLab Arena 环境。

克隆我们的 EnvHub 仓库:

git clone https://huggingface.co/nvidia/isaaclab-arena-envs

根据新环境修改 example_envs.yaml 文件。上传 修改后的仓库到 HuggingFace EnvHub。

Your IsaacLab Arena environment code must be locally available during evaluation. Users can clone your environment repository separately, or you can bundle the environment code and assets directly in your EnvHub repo.

然后,在评估时,使用你的新环境:

lerobot-eval \
    --env.hub_path=<your-env-hub-path>/isaaclab-arena-envs \
    --env.environment=<your new environment> \
    ...other flags...

我们期待您的贡献!

故障排除

CUDA 内存不足

减小 batch_size 或使用显存更大的 GPU:

--eval.batch_size=1

未接受 EULA

在运行之前设置环境变量:

export ACCEPT_EULA=Y
export PRIVACY_CONSENT=Y

视频录制无法正常工作

在无头模式下运行时启用摄像头:

--env.video=true --env.enable_cameras=true --env.headless=true

策略输出维度不匹配

确保 action_dim 与您的策略相匹配:

--env.action_dim=36

Isaac Sim 初始化期间出现的 libGLU.so.1 错误

请确保已安装以下依赖项,这种情况通常发生在无头机器上。

sudo apt update && sudo apt install -y libglu1-mesa libxt6

另请参阅

Lightwheel LW-BenchHub

Lightwheel 将包含 268 个任务的 Lightwheel-Libero-Tasks 和 Lightwheel-RoboCasa-Tasks 引入 LeRobot 生态系统。 LW-BenchHub 通过遥操作收集和生成符合 LeRobot 规范的大规模数据集,支持开箱即用的训练和评估工作流。 借助 EnvHub 提供的统一接口,开发者可以快速构建端到端的实验流水线。

安装

假设您已按照安装步骤操作,您可以通过以下方式安装 LW-BenchHub:

conda install pinocchio -c conda-forge -y
pip install numpy==1.26.0 # revert numpy to version 1.26

sudo apt-get install git-lfs && git lfs install

git clone https://github.com/LightwheelAI/lw_benchhub
cd lw_benchhub
git lfs pull # Ensure LFS files (e.g., .usd assets) are downloaded

pip install -e .

有关更详细的说明,请参阅 LW-BenchHub 文档。

Lightwheel 任务数据集

LW-BenchHub 数据集可在 HuggingFace Hub 上获取:

数据集描述任务帧
Lightwheel-Tasks-X7SX7S LIBERO 和 RoboCasa117~10.3M
Lightwheel-Tasks-Double-PiperDouble-Piper LIBERO130~6.0M
Lightwheel-Tasks-G1-ControllerG1-Controller LIBERO62~2.7M
Lightwheel-Tasks-G1-WBCG1-WBC RoboCasa32~1.5M

有关训练策略,请参阅 训练策略 部分。

评估策略

预训练策略

以下已训练的策略可供使用:

策略架构任务布局机器人链接
smolvla-double-piper-pnpSmolVLAL90K1PutTheBlackBowlOnThePlatelibero-1-1DoublePiper-AbsHuggingFace

评估 SmolVLA

lerobot-eval \
  --policy.path=LightwheelAI/smolvla-double-piper-pnp \
  --env.type=isaaclab_arena \
  --rename_map='{"observation.images.left_hand_camera_rgb": "observation.images.left_hand", "observation.images.right_hand_camera_rgb": "observation.images.right_hand", "observation.images.first_person_camera_rgb": "observation.images.first_person"}' \
  --env.hub_path=LightwheelAI/lw_benchhub_env \
  --env.kwargs='{"config_path": "configs/envhub/example.yml"}' \
  --trust_remote_code=true \
  --env.state_keys=joint_pos \
  --env.action_dim=12 \
  --env.camera_keys=left_hand_camera_rgb,right_hand_camera_rgb,first_person_camera_rgb \
  --policy.device=cuda \
  --eval.batch_size=10 \
  --eval.n_episodes=100

环境配置

通过修改配置文件中的 robot、task 和 layout 设置,即可快速启动评估。

完整配置选项

# =========================
# Basic Settings
# =========================
disable_fabric: false
device: cuda:0
sensitivity: 1.0
step_hz: 50
enable_cameras: true
execute_mode: eval
episode_length_s: 20.0 # Episode length in seconds, increase if episodes timeout during eval

# =========================
# Robot Settings
# =========================
robot: DoublePiper-Abs # Robot type, DoublePiper-Abs, X7S-Abs, G1-Controller or G1-Controller-DecoupledWBC
robot_scale: 1.0

# =========================
# Task & Scene Settings
# =========================
task: L90K1PutTheBlackBowlOnThePlate # Task name
scene_backend: robocasa
task_backend: robocasa
debug_assets: null
layout: libero-1-1 # Layout and style ID
sources:
  - objaverse
  - lightwheel
  - aigen_objs
object_projects: []
usd_simplify: false
seed: 42

# =========================
# Object Placement Retry Settings
# =========================
max_scene_retry: 4
max_object_placement_retry: 3

resample_objects_placement_on_reset: true
resample_robot_placement_on_reset: true

# =========================
# Replay Configuration Settings
# =========================
replay_cfgs:
  add_camera_to_observation: true
  render_resolution: [640, 480]

另请参阅

Update on GitHub