Data Upload
Feature Overview
IO data collection devices support one-click batch automatic upload, generally no manual upload is required. Unless it's data from external systems. You can contact IO for integration and adaptation to achieve automated upload processes as much as possible, eliminating the need for manual upload.
The platform supports multiple format data file uploads and online processing. The platform provides intelligent transcoding, format validation, progress monitoring and other functions to ensure data can smoothly enter the annotation workflow.
Main Features
Multi-format File Support
Robot Data Formats
The platform supports multiple robot data formats: MCAP files as the standard format for robot multimodal data, recommended for robot data; BAG files as the standard format for ROS robot data, supporting automatic conversion to MCAP; HDF5 files as scientific computing data format, supporting robot state data.
Media File Formats
Supports mainstream media file formats: video files including MP4, AVI, MOV, MKV, etc., audio files including MP3, WAV, AAC, OGG, etc., image files including JPG, PNG, BMP, TIFF, etc. These formats cover the vast majority of data collection scenarios.
Automatic Format Conversion
The system provides intelligent format conversion functionality, supporting automatic conversion of multiple formats to standard MCAP format:
Video to MCAP:
- Supports mainstream video formats including MP4, AVI, MOV, MKV, etc.
- Real-time conversion in browser, no server processing required
- Configurable conversion parameters:
- Image Quality: 20%, 50%, 70%, 100% (default 70%)
- Video Frame Rate: Auto, 10 FPS, 15 FPS, 30 FPS (default auto)
- Include Audio: Optional whether to retain audio tracks in video
- Conversion process displays real-time progress
- Converted MCAP files are automatically uploaded
Audio to MCAP:
- Supports mainstream audio formats including MP3, WAV, AAC, OGG, etc.
- Automatically extracts audio metadata such as sample rate, channel count, etc.
- Converts audio data to ROS standard AudioData message format
- Supports multi-channel audio processing
- Audio message frequency fixed at 10Hz to ensure synchronization with robot data
BAG to MCAP:
- Supports ROS1 and ROS2 BAG file formats
- Automatically extracts all topics and messages
- Maintains timestamp and message structure integrity
- Converted MCAP files can be used normally in the platform
Browser Compatibility Detection:
Video conversion functionality requires browser support for MediaStreamTrackProcessor API. Currently only the latest versions of Chrome and Edge browsers support this feature.
The platform automatically detects whether the browser supports video conversion:
- Compatible Browsers: Chrome 94+, Edge 94+ (fully supported)
- Incompatible Browsers: Firefox, Safari, etc. (will display clear error message)
- Auto Detection: When uploading video files, the system automatically detects browser compatibility
- User-friendly Prompt: If not supported, will prompt users to use Chrome or Edge browser
If your browser does not support video conversion, we recommend:
- Use the latest version of Chrome or Edge browser
- Or convert videos to MCAP format before uploading
- Contact technical support for other solutions
Project and Storage Management
Project Selection
The system displays all accessible projects, including personal spaces, shared projects, private projects and other different types. According to user permissions, accessible projects are displayed to ensure data security and access control.
Cloud Storage Selection
Supports multiple cloud storage services: Tencent Cloud COS (supporting multiple regions), Alibaba Cloud OSS, Huawei Cloud OBS, self-built MinIO storage, etc. Users can switch between different storage options and choose the most suitable storage solution.
Smart Upload Features
Drag and Drop Upload
Supports dragging files to the upload area, can upload multiple files simultaneously, and even supports uploading entire folders. This intuitive upload method greatly improves user experience.
Progress Monitoring
Real-time display of upload progress for each file, including current upload speed, remaining time and other information. When upload fails, the system will display specific error reasons to help users quickly locate problems.
Resume Upload
When network is interrupted, the system will automatically resume upload to ensure file upload integrity. It also has duplicate upload detection functionality to avoid uploading the same files repeatedly, saving time and bandwidth.
Soft-deleted Dataset Recovery:
The platform supports intelligent dataset recovery functionality. When you re-upload a previously deleted (soft-deleted) dataset:
- Auto Detection - The system automatically detects if a soft-deleted dataset with the same name exists
- Recovery Options - If a soft-deleted dataset is detected, you can choose:
- Recover Existing Dataset - Recover the soft-deleted dataset, preserving original annotations, task associations, and other information
- Create New Dataset - Ignore the soft-deleted dataset and create a new dataset record
- Data Protection - Recovery operations preserve all historical information of the dataset, including:
- Original annotation data
- Associated annotation tasks
- Dataset tags and metadata
- Access and operation logs
Dataset Recovery Recommendations:
- If data was accidentally deleted, we recommend selecting "Recover Existing Dataset" to preserve historical information
- If it's a new data file, you can choose "Create New Dataset"
- Recovered datasets are automatically removed from the trash
Data Preprocessing
Format Validation
The system automatically validates file formats, checks if files are corrupted, detects video/audio encoding formats, checks video resolution and frame rate, etc. These validations ensure uploaded data meets platform requirements.
Automatic Transcoding
Provides intelligent transcoding functionality to ensure data can be used normally in the platform:
Video Transcoding:
- Convert videos to standard MCAP format
- Support custom quality and frame rate settings
- Automatically handle video encoding formats
- Optimize file size while maintaining video quality
Audio Transcoding:
- Convert audio to MCAP format AudioData messages
- Automatically extract audio metadata (sample rate, channel count, format, etc.)
- Convert to ROS standard message format
- Maintain audio quality, support lossless conversion
Format Standardization:
- Automatically unify data from different sources to standard format
- Handle encoding differences and format compatibility issues
- Ensure data can be played and processed normally in the platform
Compression Optimization:
- Automatically compress large files to reduce storage space
- Maintain data quality while compressing
- Support selecting compression level
- Optimize transfer speed
Metadata Extraction
The system automatically extracts various metadata: video information (duration, resolution, frame rate, etc.), audio information (duration, sample rate, channels, etc.), robot data (sensor data information), timestamps (data collection time), etc. This metadata provides important information for subsequent data management.
Upload Management
Upload Queue
Provides comprehensive upload queue management functions: managing pending upload file queues, setting file upload priorities, pausing or resuming upload tasks, canceling unnecessary upload tasks. These functions allow users to flexibly control the upload process.
History Records
Records all upload history, providing complete upload task tracking and management functionality:
Upload Record Information:
- File Information - File name, size, format, upload time
- Processing Status - Pending, converting, uploading, completed, failed
- Progress Information - Real-time display of upload and conversion progress percentage
- Result Information - Created dataset ID and link (after successful upload)
- Error Information - If failed, displays detailed error reason
Upload Status Tracking:
- pending (Pending) - File selected, waiting to start processing
- processing (Processing) - Currently converting or uploading
- converting (Converting) - Video/audio files being converted to MCAP format
- uploading (Uploading) - File being uploaded to cloud storage
- completed (Completed) - File upload successful, dataset created
- error (Error) - Error occurred during processing
- cancelled (Cancelled) - User actively cancelled upload
Task Management:
- Cancel Upload - Can cancel ongoing upload tasks at any time
- Retry Upload - Failed tasks can be retried for upload
- View Dataset - After successful upload, can quickly jump to created dataset page
- Batch Operations - Support batch viewing and management of multiple upload tasks
Error Handling:
- Automatically displays error information to help quickly locate problems
- Common error prompts and handling suggestions
- Support viewing detailed error logs
- Failed tasks support one-click retry
Applicable Roles
Administrator
As a platform administrator, you can configure cloud storage and upload parameters, manage different cloud storage configurations, monitor upload status and storage usage, and set upload permissions for different users. These functions ensure the platform's upload service is stable and reliable.
Project Manager
Project managers can upload relevant data for projects, organize uploaded data by project, ensure upload data quality, and guide team members to upload data correctly. Through the data upload module, project managers can effectively manage project data resources.
Data Collector
Data collectors can upload collected raw data, batch upload data from collection tasks, convert collected data to standard formats, and update collection task status. These functions help data collectors efficiently complete data collection work.
Annotator
Annotators can upload data that needs annotation, upload annotation reference data, and upload annotation result data. These functions support annotators in completing various annotation tasks.