Skip to main content

Annotation Tasks

Feature Overview​

Annotation task management is the core workflow module of the IO data platform, providing complete task lifecycle management functionality. Through status grouping, progress tracking, quality control and team collaboration, it ensures annotation work proceeds efficiently and orderly.

Main Features​

Task Status Management​

Status Grouping​

The system divides tasks into five main statuses: pending start (tasks created but not yet started), in progress (tasks currently being annotated), pending review (annotation completed tasks waiting for review), review passed (annotation tasks that passed review), data submitted (data submitted for training). This status grouping allows you to clearly understand the current status of each task.

Status Flow​

Tasks go through the following status flow in their lifecycle: after task creation, enter pending start status, after starting annotation, change to in progress, after completing annotation, enter pending review, after review passes, change to review passed, if review fails, return to in progress for re-annotation, finally after data submission, change to data submitted. This process ensures orderly annotation work.

Task Creation and Assignment​

Task Creation​

When creating tasks, you need to select data to annotate on the data page, set task name, description, priority, specify annotators and reviewers, select project the task belongs to, and set task completion time. These configurations ensure tasks can be executed as expected.

Batch Operations​

The system supports rich batch operation functionality that greatly improves task management efficiency:

Batch Selection:

  • Support multi-select tasks in task list
  • Can filter by status, project and other conditions before batch operations
  • Bottom action bar appears after selection

Batch Delete:

  • Can batch delete selected tasks
  • Confirmation dialog shown before deletion, displaying number of tasks to be deleted
  • Delete operation also cleans related task association data
  • Support cancel operation

Batch Assignment (In Development):

  • Support batch reassignment of tasks to annotators
  • Support batch modification of task attributes
warning

Batch delete operations cannot be recovered, please operate with caution. Deleting tasks will not delete associated datasets, only remove the association relationship between tasks and datasets.

Task Detail Page​

The platform provides detailed detail pages for each annotation task, facilitating viewing and management of all aspects of tasks:

Page Structure:

  • Breadcrumb Navigation - Quick return to task list or project page
  • Tab Switching - Multiple function tabs organizing different content
  • Sidebar - Display task basic information and action buttons
  • Main Content Area - Display corresponding content based on active tab

Function Tabs:

  1. Annotation Data (dataset) - Default tab

    • Display list of all datasets associated with the task
    • Can view basic information of each dataset
    • Support dataset-task association management
    • Can add or remove datasets

  2. Batch Annotation (repeats) - Batch annotation workspace

    • Batch annotation mode to improve annotation efficiency
    • Sample selector for quick data switching
    • Annotation progress bar showing completion status
    • Sync metadata reminders

  3. Invalid Annotations (invalids) - Invalid annotation management

    • Display all annotations marked as invalid
    • Can view invalid reasons
    • Support re-annotation or deletion of invalid annotations

  4. Annotation Statistics (analysis) - Annotation data analysis

    • Display statistical information of all annotations in the task
    • Annotation description list and details
    • Support annotation data export

Sidebar Information:

  • Task basic information (name, status, creation time, etc.)
  • Task related personnel (annotators, reviewers)
  • Task statistics (dataset count, annotation count, etc.)
  • Task operations (edit, delete, etc.)

Progress Tracking and Monitoring​

Real-time Progress​

The system displays task completion percentage, estimated remaining completion time, annotation quality statistics, and annotation efficiency trend analysis in real-time. This information helps you timely understand task progress and make corresponding adjustments.

Task Detail Page Progress:

  • Can view detailed progress information on task detail page
  • Display annotation completion status of each dataset
  • Display annotator work progress
  • Real-time updates, automatically refreshes when page visibility changes

Progress Reports​

Provides various reports such as personal progress (individual annotator work progress), team progress (overall team work progress), project progress (project-level progress statistics), quality reports (annotation quality analysis reports), meeting different levels of management needs.

Quality Control System​

Review Mechanisms​

The system provides multiple review mechanisms: rule-based automatic quality checking, manual quality checking by reviewers, random sampling quality checking, full review of all annotations. These mechanisms ensure annotation quality meets requirements.

Quality Indicators​

Through indicators such as pass rate (proportion passing review on first attempt), accuracy (statistics of annotation accuracy), consistency (consistency between different annotators), completeness (completeness checking of annotations), comprehensively evaluate annotation quality.

Team Collaboration Functions​

Task Assignment​

Supports various assignment methods such as intelligent assignment (intelligent assignment based on annotator capabilities and workload), manual assignment (administrator manual assignment), reassignment (reassigning tasks to other annotators), task transfer (transferring tasks between different annotators), ensuring tasks can be reasonably assigned.

Communication and Collaboration​

Provides collaboration functions such as task comments (adding comments and feedback in tasks), problem reporting (annotators reporting encountered problems), solutions (reviewers providing solutions), experience sharing (team members sharing annotation experience), promoting team communication and knowledge sharing.

Data Export and Integration​

Annotation Result Export​

Supports export in standard formats such as LeRobot, HDF5, custom export format based on needs, batch export of annotation results from multiple tasks, and export of only new or modified annotations. These functions meet data export needs for different scenarios.

Training Data Preparation​

Provides functions such as data cleaning (automatic cleaning and preprocessing of annotation data), format conversion (converting to model training required format), quality validation (validating exported data quality), version management (managing different versions of training data), ensuring exported data can be directly used for model training.

Task Queue Management​

The platform provides comprehensive background task queue management system for managing background tasks such as data export and processing:

Queue Functions (Admin Permission):

Queue Control:

  • Pause Queue - Temporarily pause queue processing, stop executing new tasks
  • Resume Queue - Resume queue processing, continue executing waiting tasks
  • Queue Status - Real-time display of queue pause/running status

Queue Cleanup:

  • Empty Waiting Queue - Clear all waiting and delayed tasks (does not affect tasks in progress, completed, or failed)
  • Clean Historical Tasks - Clean tasks completed or failed more than 24 hours ago, freeing storage space
  • Batch Retry - Batch retry all failed tasks (process up to 1000 at a time)

Queue Monitoring:

  • View number of tasks in queue (waiting, in progress, completed, failed)
  • View task execution logs and error information
  • Monitor queue processing speed and performance
info

Task Queue Notes:

  • Task queues are mainly used for background task processing, such as data export, format conversion, etc.
  • Pausing queue does not affect tasks in progress, only prevents new tasks from starting
  • Cleanup operations permanently delete historical task records, please operate with caution
  • Batch retry can help recover tasks that failed due to temporary errors

Applicable Roles​

Administrator​

As a platform administrator, you can view overall status of all tasks, manage annotator and reviewer resources, monitor overall annotation quality, and configure task processes and rules. These functions ensure the platform's task management service is stable and efficient.

Project Manager​

Project managers can create annotation tasks for projects, track project annotation progress, monitor annotation quality status, and coordinate annotator and reviewer work. Through task management module, project managers can effectively control project annotation work.

Annotator​

Annotators can receive tasks assigned to them, execute specific annotation work, update task completion progress, and feedback problems encountered during annotation. These functions support annotators in efficiently completing annotation tasks.

Reviewer​

Reviewers can review tasks completed by annotators, evaluate annotation quality, provide feedback to annotators, and establish and update annotation standards. This role plays an important role in ensuring annotation quality.