Data Upload
Feature Overview
IO data collection devices support one-click batch automatic upload, generally no manual upload is required. Unless it's data from external systems. You can contact IO for integration and adaptation to achieve automated upload processes as much as possible, eliminating the need for manual upload.
The platform supports multiple format data file uploads and online processing. The platform provides intelligent transcoding, format validation, progress monitoring and other functions to ensure data can smoothly enter the annotation workflow.
Main Features
Multi-format File Support
Robot Data Formats
The platform supports multiple robot data formats: MCAP files as the standard format for robot multimodal data, recommended for robot data; BAG files as the standard format for ROS robot data, supporting automatic conversion to MCAP; HDF5 files as scientific computing data format, supporting robot state data.
Media File Formats
Supports mainstream media file formats: video files including MP4, AVI, MOV, MKV, etc., audio files including MP3, WAV, AAC, OGG, etc., image files including JPG, PNG, BMP, TIFF, etc. These formats cover the vast majority of data collection scenarios.
Automatic Format Conversion
The system provides intelligent format conversion functions: converting video files to MCAP format in real-time, converting audio files to MCAP format, converting ROS BAG files to MCAP format, and supporting batch processing and format conversion of images. These conversion processes are transparent to users, ensuring data can smoothly enter subsequent workflows.
Project and Storage Management
Project Selection
The system displays all accessible projects, including personal spaces, shared projects, private projects and other different types. According to user permissions, accessible projects are displayed to ensure data security and access control.
Cloud Storage Selection
Supports multiple cloud storage services: Tencent Cloud COS (supporting multiple regions), Alibaba Cloud OSS, Huawei Cloud OBS, self-built MinIO storage, etc. Users can switch between different storage options and choose the most suitable storage solution.
Smart Upload Features
Drag and Drop Upload
Supports dragging files to the upload area, can upload multiple files simultaneously, and even supports uploading entire folders. This intuitive upload method greatly improves user experience.
Progress Monitoring
Real-time display of upload progress for each file, including current upload speed, remaining time and other information. When upload fails, the system will display specific error reasons to help users quickly locate problems.
Resume Upload
When network is interrupted, the system will automatically resume upload to ensure file upload integrity. It also has duplicate upload detection functionality to avoid uploading the same files repeatedly, saving time and bandwidth.
Data Preprocessing
Format Validation
The system automatically validates file formats, checks if files are corrupted, detects video/audio encoding formats, checks video resolution and frame rate, etc. These validations ensure uploaded data meets platform requirements.
Automatic Transcoding
Provides intelligent transcoding functions: converting videos to standard formats, converting audio to standard formats, automatically compressing large files while maintaining data quality. These processes ensure data can be used normally in the platform.
Metadata Extraction
The system automatically extracts various metadata: video information (duration, resolution, frame rate, etc.), audio information (duration, sample rate, channels, etc.), robot data (sensor data information), timestamps (data collection time), etc. This metadata provides important information for subsequent data management.
Upload Management
Upload Queue
Provides comprehensive upload queue management functions: managing pending upload file queues, setting file upload priorities, pausing or resuming upload tasks, canceling unnecessary upload tasks. These functions allow users to flexibly control the upload process.
History Records
Records all upload history, tracks file processing status, views error information for failed uploads, supports re-uploading failed files. These functions help users manage upload tasks and handle problems promptly.
Applicable Roles
Administrator
As a platform administrator, you can configure cloud storage and upload parameters, manage different cloud storage configurations, monitor upload status and storage usage, and set upload permissions for different users. These functions ensure the platform's upload service is stable and reliable.
Project Manager
Project managers can upload relevant data for projects, organize uploaded data by project, ensure upload data quality, and guide team members to upload data correctly. Through the data upload module, project managers can effectively manage project data resources.
Data Collector
Data collectors can upload collected raw data, batch upload data from collection tasks, convert collected data to standard formats, and update collection task status. These functions help data collectors efficiently complete data collection work.
Annotator
Annotators can upload data that needs annotation, upload annotation reference data, and upload annotation result data. These functions support annotators in completing various annotation tasks.