Data Format

The platform uses the Robot Operating System (ROS) as the baseline for managing robot data. Collected data that is not in a ROS-standard format is converted to ROS-standard format on import, and converted again on export to a format that training frameworks can read directly.

Definitions

Term	Description
ROS	Robot Operating System, a software library and toolset for building robot applications; the data baseline of the platform
MCAP	An open-source multimodal log container format led by Foxglove, pronounced em-cap, storing synchronized messages from each sensor by Topic
Topic	A named channel for messages in ROS; each one corresponds to a camera, a set of joints, or a sensor
Human data	Collected data that records an operator's actions and interactions, including motion capture and tactile sensors
Teleoperation data	Data that records the process of an operator controlling a robot through a VR device
Episode	One complete task sequence; the basic sample unit in HDF5 and LeRobot

Rules and Workflow

Data Flow and Format Relationships

Human data and teleoperated robot data are unified as ROS/MCAP after entering the platform, and converted again on export according to the target framework.

Common Rules

Rule	Content
Unified baseline	All ingested data is described with the ROS message model and converted to the target framework on export
Directory naming	The human data root directory is named {date}_{project}_{scene}_{task}_{staff_id}_{timestamp}; the teleoperation data root directory is named {robot_name}_{date}_{timestamp}_{sequence_id}
Time reference	Annotations record both a nanosecond timestamp and a frame number in the fields start_timestamp, end_timestamp, start_frame_id, and end_frame_id
Adaptive fields	Exports generate fields from the joint and camera counts of the source data, with no manual configuration
Robot model support	Built-in visualization models for 23 robot models, covering dual-arm, humanoid, collaborative arm, and mobile platforms (source at the end of this page)

Human Data Format

Directory Structure

Each collection task generates one folder named by timestamp, with the following entries.

Path	Type	Description
{date}_{project}_{scene}_{task}_{staff_id}_{timestamp}/	Directory	Collection task root directory; the name contains the date, project, scene, task, collector, and timestamp
data.mcap	File	Multimodal data package containing synchronized data from all sensors
annotation.json	File	Annotation data
align_result.csv	File	Timestamp alignment table
config/	Directory	Camera and sensor configuration
config/calib_data.yml	File	Calibration data
config/depth_to_rgb.yml	File	Depth-to-color alignment parameters
config/mocap_main.yml	File	Main motion capture configuration
config/orbbec_depth.yml	File	Depth camera configuration
config/orbbec_rgb.yml	File	Color camera configuration
config/pose_calib.yml	File	Pose calibration parameters

Topics and Message Types

The name, message type, and data meaning of each Topic inside data.mcap are as follows.

Topic name	Message type	Description
/mocap/sensor_data	io_msgs/squashed_mocap_data	Joint velocity, acceleration, angular velocity, rotation angle, and sensor data from motion capture
/mocap/ros_tf	tf2_msgs/TFMessage	All joint TF transforms based on motion capture
/joint_states	sensor_msgs/JointState	All joint JointState values based on motion capture
/rgbd/color/image_raw/compressed	sensor_msgs/CompressedImage	Main head camera RGB image
/rgbd/depth/image_raw	sensor_msgs/Image	Main head camera depth image
/colorized_depth	sensor_msgs/CompressedImage	Main head camera colorized depth image
/left_ee_pose	geometry_msgs/PoseStamped	Left gripper pose in the main head camera coordinate frame
/right_ee_pose	geometry_msgs/PoseStamped	Right gripper pose in the main head camera coordinate frame
/claws_l_hand	io_msgs/claws_angle	Left gripper closure level
/claws_r_hand	io_msgs/claws_angle	Right gripper closure level
/claws_touch_data	io_msgs/squashed_touch	Gripper tactile data; contains two messages, frame_id identifies the left or right gripper, and the first four values in data are valid
/realsense_left_hand/color/image_raw/compressed	sensor_msgs/CompressedImage	Left gripper camera RGB image
/realsense_left_hand/depth/image_rect_raw	sensor_msgs/Image	Left gripper camera depth image
/realsense_right_hand/color/image_raw/compressed	sensor_msgs/CompressedImage	Right gripper camera RGB image
/realsense_right_hand/depth/image_rect_raw	sensor_msgs/Image	Right gripper camera depth image
/usb_cam_fisheye/mjpeg_raw/compressed	sensor_msgs/CompressedImage	Main head fisheye camera RGB image
/usb_cam_left/mjpeg_raw/compressed	sensor_msgs/CompressedImage	Main head left monocular camera RGB image
/usb_cam_right/mjpeg_raw/compressed	sensor_msgs/CompressedImage	Main head right monocular camera RGB image
/ee_visualization	sensor_msgs/CompressedImage	End effector pose visualization over the main head camera RGB image
/touch_visualization	sensor_msgs/CompressedImage	Gripper tactile data visualization
/robot_description	std_msgs/String	Motion capture URDF
/global_localization	geometry_msgs/PoseStamped	Pose of the main head camera in the world coordinate frame
/world_left_ee_pose	geometry_msgs/PoseStamped	Pose of the left gripper in the world coordinate frame
/world_right_ee_pose	geometry_msgs/PoseStamped	Pose of the right gripper in the world coordinate frame

tip

When collecting with tactile gloves, an additional /mocap/touch_data Topic is added, with the message type io_msgs/squashed_touch.

Camera Configuration

Human data falls into four groups by camera position.

Camera	Data type
Main head RGBD camera	Color image and depth image
Left gripper and right gripper cameras	RealSense RGBD
Fisheye camera	Panoramic image
Left monocular and right monocular cameras	Stereo vision image

Raw MCAP data listing example

library:   mcap go v1.7.0
profile:   ros1
messages:  45200
duration:  1m5.625866496s
start:     2025-01-15T18:09:29.628202496+08:00 (1736935769.628202496)
end:       2025-01-15T18:10:35.254068992+08:00 (1736935835.254068992)
compression:
    zstd: [764/764 chunks] [6.13 GiB/3.84 GiB (37.39%)] [59.87 MiB/sec]
channels:
    (1)  /rgbd/color/image_raw/compressed                  1970 msgs (30.02 Hz)   : sensor_msgs/CompressedImage [ros1msg]
    (2)  /joint_states                                     1970 msgs (30.02 Hz)   : sensor_msgs/JointState [ros1msg]
    (3)  /claws_r_hand                                     1970 msgs (30.02 Hz)   : io_msgs/claws_angle [ros1msg]
    (4)  /global_localization                              1970 msgs (30.02 Hz)   : geometry_msgs/PoseStamped [ros1msg]
    (5)  /robot_description                                   1 msgs              : std_msgs/String [ros1msg]
    (6)  /ee_visualization                                 1970 msgs (30.02 Hz)   : sensor_msgs/CompressedImage [ros1msg]
    (7)  /rgbd/depth/image_raw                             1970 msgs (30.02 Hz)   : sensor_msgs/Image [ros1msg]
    (8)  /colorized_depth                                  1970 msgs (30.02 Hz)   : sensor_msgs/CompressedImage [ros1msg]
    (9)  /claws_l_hand                                     1970 msgs (30.02 Hz)   : io_msgs/claws_angle [ros1msg]
    (10) /claws_touch_data                                 1970 msgs (30.02 Hz)   : io_msgs/squashed_touch [ros1msg]
    (11) /touch_visualization                              1970 msgs (30.02 Hz)   : sensor_msgs/CompressedImage [ros1msg]
    (12) /mocap/sensor_data                                1970 msgs (30.02 Hz)   : io_msgs/squashed_mocap_data [ros1msg]
    (13) /mocap/ros_tf                                     1970 msgs (30.02 Hz)   : tf2_msgs/TFMessage [ros1msg]
    (14) /left_ee_pose                                     1970 msgs (30.02 Hz)   : geometry_msgs/PoseStamped [ros1msg]
    (15) /right_ee_pose                                    1970 msgs (30.02 Hz)   : geometry_msgs/PoseStamped [ros1msg]
    (16) /usb_cam_left/mjpeg_raw/compressed                1960 msgs (29.87 Hz)   : sensor_msgs/CompressedImage [ros1msg]
    (17) /usb_cam_right/mjpeg_raw/compressed               1946 msgs (29.65 Hz)   : sensor_msgs/CompressedImage [ros1msg]
    (18) /usb_cam_fisheye/mjpeg_raw/compressed             1957 msgs (29.82 Hz)   : sensor_msgs/CompressedImage [ros1msg]
    (19) /realsense_left_hand/depth/image_rect_raw         1961 msgs (29.88 Hz)   : sensor_msgs/Image [ros1msg]
    (20) /realsense_left_hand/color/image_raw/compressed   1961 msgs (29.88 Hz)   : sensor_msgs/CompressedImage [ros1msg]
    (21) /realsense_right_hand/depth/image_rect_raw        1947 msgs (29.67 Hz)   : sensor_msgs/Image [ros1msg]
    (22) /realsense_right_hand/color/image_raw/compressed  1947 msgs (29.67 Hz)   : sensor_msgs/CompressedImage [ros1msg]
    (23) /world_left_ee_pose                               1970 msgs (30.02 Hz)   : geometry_msgs/PoseStamped [ros1msg]
    (24) /world_right_ee_pose                              1970 msgs (30.02 Hz)   : geometry_msgs/PoseStamped [ros1msg]
channels: 24
attachments: 0
metadata: 0

Annotation Data

annotation.json records the semantic annotation of one piece of human data. The top-level fields and subtask fields are as follows.

Field	Type	Required	Description
belong_to	string	Yes	Name of the dataset it belongs to
mocap_offset	array	No	Motion capture time offset
object_set	string[]	Yes	Set of manipulated objects involved in this piece of data
scene	string	Yes	Scene name
skill_set	string[]	Yes	Set of skill templates involved in this piece of data, such as pick {A} from {B}
subtasks	object[]	Yes	Subtask list; each entry corresponds to one annotation time segment
tag_set	string[]	No	Tag set
task_description	string	Yes	Overall task description

The fields of a single subtask in subtasks are as follows.

Field	Type	Required	Description
skill	string	Yes	Skill template
description	string	Yes	English natural language description
description_zh	string	No	Chinese natural language description
sequence_id	int	Yes	Subtask index, increasing over time
start_frame_id	int	Yes	Start frame number
start_timestamp	string	Yes	Start nanosecond timestamp
end_frame_id	int	Yes	End frame number
end_timestamp	string	Yes	End nanosecond timestamp
comment	string	No	Remarks
attempts	string	Yes	Execution result, such as success

Chinese annotation data example

{
  "belong_to": "20250115_InnerTest_PublicArea_TableClearing_szk_180926",
  "mocap_offset": [],
  "object_set": [
    "paper cup",
    "placemat",
    "trash can",
    "napkin",
    "plate",
    "dinner knife",
    "tableware storage box",
    "wine glass",
    "dinner fork"
  ],
  "scene": "PublicArea",
  "skill_set": [
    "pick {A} from {B}",
    "toss {A} into {B}",
    "place {A} on {B}"
  ],
  "subtasks": [
    {
      "skill": "pick {A} from {B}",
      "description": "pick the paper cup from the placemat with the left gripper",
      "description_zh": "左夹爪 从 餐垫 捡起 纸杯",
      "end_frame_id": 227,
      "end_timestamp": "1736935777206000000",
      "sequence_id": 1,
      "start_frame_id": 159,
      "start_timestamp": "1736935774906000000",
      "comment": "",
      "attempts": "success"
    },
    {
      "skill": "toss {A} into {B}",
      "description": "toss the paper cup into the trash can with the left gripper",
      "description_zh": "左夹爪 扔纸杯进垃圾桶",
      "end_frame_id": 318,
      "end_timestamp": "1736935780244000000",
      "sequence_id": 2,
      "start_frame_id": 231,
      "start_timestamp": "1736935777306000000",
      "comment": "",
      "attempts": "success"
    }
  ],
  "tag_set": [],
  "task_description": "20250115_InnerTest_PublicArea_TableClearing_szk_180926"
}

Teleoperated Robot Data Format

Directory Structure

Teleoperation data records the process of an operator controlling a robot through a VR device. The entries are as follows.

Path	Type	Description
{robot_name}_{date}_{timestamp}_{sequence_id}/	Directory	Collection task root directory
*_{sequence_id}.mcap	File	Multimodal data
*_{sequence_id}.json	File	Annotation data
*_{sequence_id}.metadata.yaml	File	Metadata

Topics and Message Types

Topic name	Message type	Description
/camera_01/color/image_raw/compressed	sensor_msgs/msg/CompressedImage	Main camera RGB image
/camera_02/color/image_raw/compressed	sensor_msgs/msg/CompressedImage	Left camera RGB image
/camera_03/color/image_raw/compressed	sensor_msgs/msg/CompressedImage	Right camera RGB image
io_teleop/joint_states	sensor_msgs/msg/JointState	Joint state
io_teleop/joint_cmd	sensor_msgs/msg/JointState	Joint command
io_teleop/target_ee_poses	geometry_msgs/msg/PoseArray	End effector target pose
io_teleop/target_base_move	std_msgs/msg/Float64MultiArray	Base movement target
io_teleop/target_gripper_status	sensor_msgs/msg/JointState	Gripper state target
io_teleop/target_joint_from_vr	sensor_msgs/msg/JointState	Joint target from the VR device
/robot_description	std_msgs/msg/String	Robot URDF description
/tf	tf2_msgs/msg/TFMessage	TF spatial pose transform information

Raw MCAP data listing example

Files:             RM_AIDAL_250126_091041_0.mcap
Bag size:          443.3 MiB
Storage id:        mcap
Duration:          100.052164792s
Start:             Jan 24 2025 21:37:32.526605552 (1737725852.526605552)
End:               Jan 24 2025 21:39:12.578770344 (1737725952.578770344)
Messages:          62116
Topic information: Topic: /camera_01/color/image_raw/compressed | Type: sensor_msgs/msg/CompressedImage | Count: 3000 | Serialization Format: cdr
                   Topic: /camera_02/color/image_raw/compressed | Type: sensor_msgs/msg/CompressedImage | Count: 3000 | Serialization Format: cdr
                   Topic: /camera_03/color/image_raw/compressed | Type: sensor_msgs/msg/CompressedImage | Count: 3000 | Serialization Format: cdr
                   Topic: io_teleop/joint_states | Type: sensor_msgs/msg/JointState | Count: 1529 | Serialization Format: cdr
                   Topic: io_teleop/joint_cmd | Type: sensor_msgs/msg/JointState | Count: 10009 | Serialization Format: cdr
                   Topic: io_teleop/target_ee_poses | Type: geometry_msgs/msg/PoseArray | Count: 10014 | Serialization Format: cdr
                   Topic: io_teleop/target_base_move | Type: std_msgs/msg/Float64MultiArray | Count: 10010 | Serialization Format: cdr
                   Topic: io_teleop/target_gripper_status | Type: sensor_msgs/msg/JointState | Count: 10012 | Serialization Format: cdr
                   Topic: io_teleop/target_joint_from_vr | Type: sensor_msgs/msg/JointState | Count: 10012 | Serialization Format: cdr
                   Topic: /robot_description | Type: std_msgs/msg/String | Count: 1 | Serialization Format: cdr
                   Topic: /tf | Type: tf2_msgs/msg/TFMessage | Count: 1529 | Serialization Format: cdr

Annotation Data

The annotation structure of teleoperation data matches human data, and subtasks additionally records the manipulated object and target. See the example below.

Teleoperation annotation data example

{
    "belong_to": "RM_AIDAL_250126_091041_0",
    "mocap_offset": [],
    "object_set": [
        "lemon candy",
        "plate",
        "pistachios"
    ],
    "scene": "250126",
    "skill_set": [
        "place {A} on {B}"
    ],
    "subtasks": [
        {
            "skill": "place {A} on {B}",
            "objecta": "lemon candy",
            "objectb": "plate",
            "options": [
                "leftHand"
            ],
            "description": "place the lemon candy on the plate with the left hand",
            "end_timestamp": "1737725886915000000",
            "sequence_id": 1,
            "start_timestamp": "1737725880757000000",
            "comment": "",
            "attempts": "success"
        },
        {
            "skill": "place {A} on {B}",
            "objecta": "pistachios",
            "objectb": "plate",
            "options": [
                "rightHand"
            ],
            "description": "place the pistachios on the plate with the right hand",
            "end_timestamp": "1737725950745000000",
            "sequence_id": 2,
            "start_timestamp": "1737725941657000000",
            "comment": "",
            "attempts": "success"
        }
    ],
    "tag_set": [],
    "task_description": "20250205_RM_ItemPacking_zhouxw"
}

Exporting Model Training Data

The platform converts raw MCAP and JSON data into formats that training frameworks can read directly. HDF5 and LeRobot are the two formats commonly used for training, and exports generate fields adaptively from the joint and camera counts of the source data. For the remaining export formats and operations, see Data export.

HDF5 Format

HDF5 organizes data in a hierarchical structure. The file structure is as follows.

chunk_001.hdf5
├── /data/                        # Data group
│   ├── episode_001/             # Task sequence
│   │   ├── action               # Joint command
│   │   ├── observation.state    # Sensor observation values
│   │   ├── observation.gripper  # Gripper state
│   │   └── observation.images.* # Images from each view
│   └── episode_002/
└── /meta/                        # Metadata group

For the values and types of each dataset field, see the HDF5 field table under "Parameters and Fields".

LeRobot Format

LeRobot is the standard data format in the field of robot learning and is compatible with mainstream robot learning frameworks. See piper_uncap_pen for sample data.

The length and shape of an exported LeRobot dataset adapt to the source data and support any number of cameras and joints. The table below shows the export result for an AgileX desktop 7-DOF robotic arm.

Feature name	Data type	Shape	Description
action	float32	[14]	Joint command, 7 joints for each arm
observation.state	float32	[14]	Joint state, 7 joints for each arm
observation.images.cam_high	image	[3,480,640]	High camera image
observation.images.cam_low	image	[3,480,640]	Low camera image
observation.images.cam_left_wrist	image	[3,480,640]	Left wrist camera image
observation.images.cam_right_wrist	image	[3,480,640]	Right wrist camera image
timestamp	float32	[1]	Timestamp
frame_index	int64	[1]	Frame index
episode_index	int64	[1]	Task sequence index

Complete LeRobot format definition example

{
    "codebase_version": "v2.1",
    "robot_type": "custom_arm",
    "total_episodes": 20,
    "total_frames": 5134,
    "total_tasks": 20,
    "total_videos": 0,
    "total_chunks": 1,
    "chunks_size": 1000,
    "fps": 30,
    "splits": {
        "train": "0:20"
    },
    "data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
    "video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
    "features": {
        "observation.images.camera_01": {
            "dtype": "image",
            "shape": [480, 640, 3]
        },
        "observation.state": {
            "dtype": "float64",
            "shape": [37],
            "names": [
                "r_joint1", "r_joint2", "r_joint3", "r_joint4", "r_joint5", "r_joint6",
                "l_joint1", "l_joint2", "l_joint3", "l_joint4", "l_joint5", "l_joint6",
                "R_thumb_MCP_joint1", "R_thumb_MCP_joint2", "R_thumb_PIP_joint", "R_thumb_DIP_joint",
                "R_index_MCP_joint", "R_index_DIP_joint", "R_middle_MCP_joint", "R_middle_DIP_joint",
                "R_ring_MCP_joint", "R_ring_DIP_joint", "R_pinky_MCP_joint", "R_pinky_DIP_joint",
                "L_thumb_MCP_joint1", "L_thumb_MCP_joint2", "L_thumb_PIP_joint", "L_thumb_DIP_joint",
                "L_index_MCP_joint", "L_index_DIP_joint", "L_middle_MCP_joint", "L_middle_DIP_joint",
                "L_ring_MCP_joint", "L_ring_DIP_joint", "L_pinky_MCP_joint", "L_pinky_DIP_joint",
                "platform_joint"
            ]
        },
        "action": {
            "dtype": "float64",
            "shape": [12],
            "names": [
                "l_joint1", "l_joint2", "l_joint3", "l_joint4", "l_joint5", "l_joint6",
                "r_joint1", "r_joint2", "r_joint3", "r_joint4", "r_joint5", "r_joint6"
            ]
        },
        "observation.gripper": {
            "dtype": "float64",
            "shape": [2],
            "names": ["right_gripper", "left_gripper"]
        },
        "timestamp": { "dtype": "float32", "shape": [1], "names": null },
        "frame_index": { "dtype": "int64", "shape": [1], "names": null },
        "episode_index": { "dtype": "int64", "shape": [1], "names": null },
        "index": { "dtype": "int64", "shape": [1], "names": null },
        "task_index": { "dtype": "int64", "shape": [1], "names": null }
    }
}

Parameters and Fields

Directory Naming Parameters

The root directory names of human data and teleoperation data are assembled from the following fields. Field values come from the collection stage; if any segment is missing, the data cannot be ingested.

Field	Type	Required	Default	Value range	Description
date	string	Yes	None	Date, such as 20250115	Collection date
project	string	Yes	None	Project name	Owning project
scene	string	Yes	None	Scene name	Collection scene
task	string	Yes	None	Task name	Collection task
staff_id	string	Yes	None	Collector identifier	Collector
timestamp	string	Yes	None	Time, such as 180926	Collection time
robot_name	string	Yes	None	Robot identifier, such as RM_AIDAL	Robot name, used by teleoperation data
sequence_id	int	Yes	0	Greater than or equal to 0	Collection sequence number, used by teleoperation data

HDF5 Fields

Field	Type	Required	Default	Value range	Description
action	float32 array	Yes	None	Dimension equals the joint count	Joint control command
observation.state	float32 array	Yes	None	Dimension equals the joint count	Sensor observation values
observation.images.*	JPEG-compressed image	No	None	Height × width × 3	Images from each view
observation.gripper	float32 array	No	None	Dimension equals the gripper count	Gripper state
task	string	No	None	—	English natural language description
task_zh	string	No	None	—	Chinese natural language description
score	number	No	-1	0–1; -1 means not scored	Action quality score

Basis and Sources

Item	Value	Basis and source
Robot models	23	Models with a built-in visualization model that can be registered in robot management
Training export formats	2	This page covers HDF5 and LeRobot
Human data Topics	24	Excluding /mocap/touch_data for tactile gloves
Teleoperation data Topics	11	Three cameras and eight channels for state, command, pose, and TF

Limitations

Data formats are constrained by the existing boundaries below.

Item	Limit	Description
Data baseline format	ROS standard message model only	Non-ROS formats are converted to MCAP during import first, and the platform manages the converted MCAP
HDF5 ingestion sources	Agilex, Realman, Dobot, Limx, Unix	Only these sources have dedicated converters; other HDF5 recordings cannot be ingested directly
Directory naming	All segments required	6 segments for human data and 4 segments for teleoperation data; data missing any segment cannot be ingested
Image field encoding	HDF5 uses JPEG, LeRobot uses MP4 or JPG	Encoding is written according to the selected format on export; the platform does not retain a switchable original encoding
Chunking and sampling parameters	HDF5 chunk_size 1–100, hz 1–60	For parameter meanings, see HDF5 datasets
Export format versions	LeRobot limited to v3.0 and v2.1	For version differences, see LeRobot v2 and v3 format differences

Related Pages

Page	Purpose
Data export	All export target formats and export history
LeRobot datasets and training	Import, visualization, and training of LeRobot datasets
HDF5 datasets	Import and visualization of HDF5 data
Data QC	Automatic quality check rules for ROS recordings