Fine-tuning the SmolVLA Model

SmolVLA is a Vision-Language-Action (VLA) policy from Hugging Face, and its base weights are lerobot/smolvla_base. It takes multi-view images, robot state, and optional language instructions as input and outputs continuous actions. The platform registers it under the model identifier smolvla, uses the lerobot training framework, and supports LeRobot v2 and v3 datasets.

Roles and Prerequisites

Roles

Role	Can create a training task	Description
Administrator	Yes	Not restricted by Module Permissions
Project Manager	Requires authorization	An administrator grants Create Training Job in Module Permissions
Annotator	Requires authorization	Same as above
Auditor	Requires authorization	Same as above
Collector	Requires authorization	Same as above

Viewing training tasks and metrics requires only View Training.

Prerequisites

Item	Requirement
Dataset version	LeRobot v2 or v3. The training service reads codebase_version in the dataset's meta/info.json to identify the version and choose the training image
Framework	lerobot
GPU memory	No fixed lower bound; determined by batch_size, image resolution, and the freeze strategy. The vision encoder is frozen by default
Dependency	The platform's model training service is enabled with at least one training location configured; the training image must include the LeRobot training entry point lerobot-train and video decoding dependencies
Dataset structure	The directory contains meta/info.json, data/, and videos/; image fields are consistent across episodes, and the dimensions of observation.state and action are fixed
Permission	Creating requires Create Training Job; viewing requires View Training

Training Task Creation and Parameters

Procedure

1. Go to Model → Training and select Create Training Job.
2. Choose a training location: a local GPU or a cloud training location enabled by an administrator.
3. Under Selected Model, select smolvla.
4. Under Training Dataset, choose the data source: Export Records, an uploaded LeRobot dataset, Download Link, or Hugging Face.
5. Set parameters in Training Parameters; the table below lists their meaning and default values.
6. Enter Notes (optional).
7. Select Create Training Job.

Training Parameters

The lerobot framework first applies general training parameters:

Parameter	Description	Default	Valid range
batch_size	Batch size	1	Integer, ≥ 1
steps	Training steps	10000	Integer, ≥ 1
seed	Random seed	1000	Integer
num_workers	DataLoader processes	4	Integer, ≥ 1
eval_freq	Evaluation interval	1000	Integer, ≥ 1 and not greater than steps
log_freq	Logging interval	100	Integer, ≥ 1 and not greater than steps
save_freq	Checkpoint save interval	5000	Integer, ≥ 1 and not greater than steps
save_checkpoint	Whether to save checkpoints	true	true / false

It then appends model parameters with the policy. prefix:

Parameter	Description	Default	Valid range
policy.n_obs_steps	Input observation steps	1	Integer
policy.chunk_size	Action prediction length	50	Integer
policy.n_action_steps	Action steps executed per call	50	Integer
policy.max_state_dim	State vector padding dimension	32	Integer
policy.max_action_dim	Action vector padding dimension	32	Integer
policy.resize_imgs_with_padding	Image preprocessing size	[512, 512]	Two-element tuple
policy.empty_cameras	Number of empty camera placeholders	0	Integer
policy.tokenizer_max_length	Maximum tokenizer length	48	Integer
policy.num_steps	Number of decoding steps	10	Integer
policy.use_cache	Whether to use the attention cache	true	true / false
policy.freeze_vision_encoder	Whether to freeze the vision encoder	true	true / false
policy.train_expert_only	Whether to train only the action expert	true	true / false
policy.train_state_proj	Whether to train the state projection layer	true	true / false
policy.vlm_model_name	Vision-language backbone model	HuggingFaceTB/SmolVLM2-500M-Video-Instruct	String
policy.attention_mode	Attention mode	cross_attn	cross_attn, self_attn
policy.pad_language_to	Language padding method	longest	longest, max_length
policy.num_vlm_layers	Number of vision-language backbone layers	16	Integer
policy.self_attn_every_n_layers	Interval for inserting self-attention layers	2	Integer
policy.expert_width_multiplier	Hidden width ratio of the action expert	0.75	Float
policy.optimizer_lr	Optimizer learning rate	1e-4	Float
policy.optimizer_grad_clip_norm	Gradient clipping threshold	1.0	Float
policy.scheduler_decay_steps	Learning rate decay steps	100000	Integer
policy.scheduler_decay_lr	Final learning rate after decay	1e-5	Float

Verification

Training Monitoring

The training details page parses metrics in the LeRobot log format and plots the curves:

Metric	Meaning	Criteria
step	Number of completed training steps	Increases step by step up to the configured steps
sample	Number of processed samples	Increases with step
episode	Number of processed episodes	Increases with step
epoch	Number of trained epochs	Increases with step
loss	Training loss	Decreases overall
gradient_norm	Gradient norm	Does not keep growing
learning_rate	Current learning rate	Changes according to the schedule
update_time_s	Time per update step	Stable, does not keep growing
data_time_s	Data loading time	Stable

Offline loss only reflects how well the model fits the training distribution. To evaluate policy performance, use the real-robot success rate under fixed initial states and varied object positions and lighting conditions.

Verification Checklist

The training service determines the final task state from the container exit code: exit code 0 is recorded as Success, and a nonzero code as Failed.

Check	Pass criteria
Task status	The training details page shows Success and the container exit code is 0
Output directory	checkpoints/ under the output directory
Checkpoints	Checkpoints lists the checkpoints, whose contents include pretrained_model/; the entry that last points to has type Final, and the rest have type Step
Metric curves	Training Metrics plots step, loss, gradient_norm, and learning_rate
Logs	Real-time Logs can be read without repeated errors

Checkpoints listed in Checkpoints can be downloaded, synced to object storage, or used to create an inference service; see Model Inference for the inference side.

Error Handling

Symptom	Possible cause	Action	Owner
Training fails immediately with a dataset-not-found error	The dataset lacks meta/info.json, or the selected source path is wrong	Select the correct Export Records entry or uploaded dataset again	Project Manager
Video decoding fails	The training image lacks ffmpeg or video decoding dependencies	Use the platform training image, or switch to exported MP4 video data	Algorithm Engineer
Out of GPU memory	batch_size or image resolution is too large	Lower batch_size; check policy.resize_imgs_with_padding	Algorithm Engineer
Loss decreases but real-robot performance is poor	Insufficient data coverage, or the task text does not match the actions	Add samples with varied object positions, lighting, and initial poses; verify the task text	Algorithm Engineer
Save interval greater than training steps error	save_freq exceeds steps	Reduce save_freq to no more than steps	Algorithm Engineer
Loss stops decreasing for a long time	Unsuitable learning rate or too little data	Check the number of episodes in the dataset; adjust policy.optimizer_lr	Algorithm Engineer
Training location unavailable	The platform has not enabled that training location, or its GPU is unavailable	Switch to another training location; ask an administrator to check the Training Service	Administrator

Related Pages

Page	Purpose
Model Training	Creating, monitoring, and quotas for training tasks
LeRobot datasets and training	Dataset export and training overview
LeRobot v2 and v3 format differences	Differences between the two data formats and migration
Data export	Generate a LeRobot training package