Open-X Embodiment

The key convention of Open-X Embodiment (OXE) is as follows:
`episode --> timestep --> {observation, action, language_instruction}

Open X-Embodiment stores each dataset as a sequence of episodes using the RLDS episode format.

Reinforcement Learning Datasets (RLDS)

This format is a standardized schema and storage specification for sequential decision-making and robot learning data.

The dataset is a collection of independent episodes (or trajectories), serialized as TFRecord files and managed via TensorFlow Datasets (TFDS). Episodes in the dataset is an ordered sequence of steps. Each episode contains nested feature dictionaries, metadata (e.g. robot type, task description), and the steps themselves. Steps are the individual state transitions within an episode.

Observation Format

step["observation"] = {
    "image": ...,          # RGB camera image
    "depth": ...,          # optional depth image
    "state": ...,          # optional robot proprioception
}
step["language_instruction"] = "pick up the red block"

OpenVLA/OXE configs map dataset-specific camera names into common slots:

Slot	Meaning
primary	main external RGB camera
secondary	optional second external RGB camera
wrist	optional wrist / gripper camera
depth primary/secondary/wrist	same idea, but for depth

For proprioceptive state, OpenVLA standardizes to an 8D state vector when state is used. The encoding depends on StateEncoding:

Encoding	Convention (End-Effector, EEF)
POS_EULER	[EEF_x, EEF_y, EEF_z, roll, pitch, yaw, pad, gripper]
POS_QUAT	[EEF_x, EEF_y, EEF_z, quat_4, gripper]
JOINT	[joint_1 ... joint_7, gripper], padded if fewer joints
NONE	no proprioceptive state

Action Format

action = [
    delta_x,
    delta_y,
    delta_z,
    delta_roll,
    delta_pitch,
    delta_yaw,
    gripper_command,
]

This corresponds to ActionEncoding.EEF_POS: EEF delta XYZ + orientation command + gripper open/close. OpenVLA’s config also defines joint-space alternatives such as JOINT_POS, but most manipulation datasets in the OpenVLA OXE mixture use end-effector-space actions.

Many dataset transforms explicitly concatenate:

world_vector + rotation_delta + gripper

and standardize the gripper action so that:

1.0 = open0.0 = close

What to do for UR5e DemoRecorder

FYP plan is to synchronize joint positions, TCP pose, gripper state, and camera image at a fixed rate. Storing all the data generously:

{
    "timestamp": t,
    "image_primary": rgb,
    "joint_positions": q_actual,        # 6D for UR5e
    "tcp_pose_ur": [x, y, z, rx, ry, rz],# UR axis-angle raw pose
    "gripper_state": gripper,
    "language_instruction": instruction,
    "commanded_action": action_command,
}

Then the converter can produce OXE/OpenVLA-style data:

observation_state = [
	tcp_x, tcp_y, tcp_z,
	roll, pitch, yaw,
	0.0, # pad
	gripper_open
]

action = [
	delta_x,
	delta_y,
	delta_z,
	delta_roll,
	delta_pitch,
	delta_yaw,
	gripper_command
]