Workflow Management

When data volume is large and processing workflows are repetitive, manual operations are time-consuming and error-prone.

Typical scenarios:

• Newly uploaded MCAP files need automatic preprocessing
• Data from specific projects need automatic annotation task creation
• Completed annotation data need automatic quality checks
• HDF5 format data need automatic conversion to MCAP format

Workflow management is designed to solve these problems. By configuring matching rules and action rules, it can automatically execute data processing, annotation assignment, quality checks and other operations, improving efficiency and reducing manual intervention.

Core Concepts

Matching Rules

Define data matching conditions to identify datasets that need processing.

Rule Types:

• Match by Project: Match data from specific projects
• Match by Tag: Match data with specific tags
• Match by Robot Type: Match data collected by specific robots
• Match by Data Format: Match data in specific formats (such as MCAP, HDF5)
• Match by Status: Match data in specific status (such as unannotated, annotated)

Rule Configuration:

• Rule name and description
• Matching conditions (AND/OR logic)
• Rule priority
• Rule enable status

Action Rules

Define operations to execute on matched data.

Operation Types:

• Data Preprocessing: Automatic renaming, format conversion, etc.
• Automatic Annotation Assignment: Automatically create annotation tasks and assign
• Quality Check: Automatically execute quality checks
• Tag Addition: Automatically add tags
• Project Assignment: Automatically assign to projects
• Format Conversion: HDF5 → MCAP, LeRobot → MCAP, etc.

Rule Configuration:

• Rule name and description
• Execution action
• Action parameters
• Rule priority

Workflows

Combine matching rules and action rules into complete workflows.

Workflow Structure:

• Workflow name and description
• Associated matching rules
• Associated action rules
• Workflow priority
• Workflow enable status

Execution Flow:

1. Data enters system or status changes
2. Workflow engine checks matching rules
3. If matched, execute associated action rules
4. Record execution logs

Quick Start: Create Your First Workflow

Step 1: Create Matching Rule

1. Go to Workflow Management page, switch to "Matching Rules" tab
2. Click "New Matching Rule"
3. Fill in rule information:
   • Rule Name: Clearly describe matching conditions (e.g., "Newly Uploaded MCAP Files")
   • Rule Description: Additional notes (optional)
4. Configure matching conditions:
   • Select matching fields (project, tag, format, etc.)
   • Set matching values
   • Can add multiple conditions, combine with AND/OR logic
5. Set priority and enable status
6. Save rule

Step 2: Create Action Rule

1. Switch to "Action Rules" tab
2. Click "New Action Rule"
3. Fill in rule information:
   • Rule Name: Clearly describe executed operation (e.g., "Automatic Preprocessing")
   • Rule Description: Additional notes (optional)
4. Configure execution action:
   • Select action type (data preprocessing, format conversion, etc.)
   • Configure action parameters
   • Can add multiple action steps
5. Set priority and enable status
6. Save rule

Step 3: Create Workflow

1. Switch to "Workflows" tab
2. Click "New Workflow"
3. Fill in workflow information:
   • Workflow Name: Clearly describe workflow purpose (e.g., "MCAP File Automatic Preprocessing")
   • Workflow Description: Additional notes (optional)
   • Project Scope: Select global or specific project
4. Select rules:
   • Select matching rule
   • Select action rule
5. Set priority and enable status
6. Save workflow

After workflow is created, the system will automatically monitor data status and automatically execute workflows when data matches rules.

Advanced Usage

How to Configure Complex Matching Conditions?

Multi-Condition Combination:

Matching rules support combination of multiple conditions:

• AND Logic: All conditions must be satisfied

  • Example: Project = "Project A" AND Format = "MCAP" AND Status = "Unannotated"

• OR Logic: Any condition satisfied is sufficient

  • Example: Tag = "High Quality" OR Tag = "Test Data"

Condition Types:

• Project Matching: Match data from specific projects
• Tag Matching: Match data with specific tags
• Robot Type Matching: Match data collected by specific robots
• Data Format Matching: Match data in specific formats
• Status Matching: Match data in specific status

How to Configure Multiple Action Steps?

Action Steps:

Action rules can contain multiple steps, executed in sequence:

1. Automatic Renaming: Rename datasets according to rules
2. Format Conversion: Convert data to standard format
3. Automatic Project Import: Import data to specified project
4. Run Custom Algorithm: Execute custom processing algorithms

Step Configuration:

Each step can configure independent parameters:

• Renaming rules: Set naming patterns and target formats
• Conversion parameters: Set conversion options and parameters
• Project selection: Select target project

How to Set Workflow Priority?

Priority Note:

• Smaller numbers mean higher priority
• When multiple workflows match, execute in priority order
• High priority workflows execute first

Setting Recommendations:

• Important workflows set higher priority (e.g., 1-10)
• General workflows set medium priority (e.g., 11-50)
• Optional workflows set lower priority (e.g., 51-100)

How to Test Workflows?

Rule Testing:

After creating rules, you can test if rules are correct:

1. Click "Test" in rule list
2. System will display list of matched datasets
3. Confirm if matching results meet expectations

Workflow Testing:

1. After creating workflow, can disable it first
2. Manually trigger test, view execution results
3. Enable workflow after confirming no issues

Use Cases

Scenario 1: Automatic Data Preprocessing

Requirement: Newly uploaded MCAP files need automatic preprocessing.

Configuration Steps:

1. Create matching rule: Match newly uploaded MCAP files
   • Condition: Format = "MCAP" AND Status = "Newly Uploaded"
2. Create action rule: Execute data preprocessing operation
   • Action: Data Preprocessing
3. Create workflow: Associate matching rule and action rule
4. Enable workflow: Newly uploaded MCAP files automatically trigger preprocessing

Scenario 2: Automatic Annotation Assignment

Requirement: Data from specific projects and unannotated need automatic annotation task creation.

Configuration Steps:

1. Create matching rule: Match data from specific projects and unannotated
   • Condition: Project = "Project A" AND Status = "Unannotated"
2. Create action rule: Automatically create annotation tasks and assign to annotators
   • Action: Automatic Annotation Assignment
   • Parameters: Specify annotators and reviewers
3. Create workflow: Associate matching rule and action rule
4. Enable workflow: Data meeting conditions automatically create annotation tasks

Scenario 3: Automatic Format Conversion

Requirement: HDF5 format data need automatic conversion to MCAP format.

Configuration Steps:

1. Create matching rule: Match HDF5 format data
   • Condition: Format = "HDF5"
2. Create action rule: Execute format conversion
   • Action: HDF5 → MCAP Conversion
   • Parameters: Select robot type (Agilex, Realman, etc.)
3. Create workflow: Associate matching rule and action rule
4. Enable workflow: HDF5 data automatically convert to MCAP

Workflow Management

How to View Workflow Execution Status?

Execution Logs:

On the Workflow Management page you can view:

• Execution time
• Workflow name
• Matched datasets
• Executed actions
• Execution result (success/failure)
• Error information (if failed)

Query Functions:

• Filter by workflow
• Filter by time range
• Filter by execution status
• View execution details

How to Manage Workflows?

Workflow Operations:

• Enable/Disable: Control whether workflow executes
• Edit: Modify workflow configuration
• Delete: Delete unnecessary workflows
• Copy: Create new workflow based on existing workflow

Workflow List:

• Display list of all workflows
• Display workflow scope (project or global)
• Display enable status and priority
• Display creation and update time

Common Questions

What to Do When Workflow Doesn't Execute?

Possible Causes:

1. Workflow Not Enabled: Check if workflow is enabled
2. Matching Rule Doesn't Match: Check if matching rule is correct
3. Priority Too Low: Other high priority workflows execute first
4. Action Rule Error: Check if action rule configuration is correct

Solution:

1. Check workflow enable status
2. Test matching rule, confirm it can correctly match data
3. View execution logs to understand failure reasons
4. Adjust workflow priority

How to Avoid Workflow Conflicts?

Conflict Scenarios:

• Multiple workflows match same data
• Workflow execution order uncertain
• Action rules affect each other

Solution:

1. Set Priority: Set clear priorities for workflows
2. Refine Matching Conditions: Use more precise matching conditions to avoid overlap
3. Test Verification: Test after creation, confirm no conflicts

What to Do When Workflow Execution Fails?

Handling Steps:

1. View execution logs to understand failure reasons
2. Check if action rule configuration is correct
3. Check if data meets requirements
4. Fix issues and re-execute

Common Errors:

• Data format doesn't meet requirements
• Action parameter configuration error
• Insufficient system resources
• Network or storage issues

Applicable Roles

Administrator

You can:

• Create and manage all workflows
• Configure matching rules and action rules
• Monitor workflow execution status
• Optimize workflow performance
• Handle workflow exceptions

Project Manager

You can:

• Create project-specific workflows
• Configure project-specific automation rules
• Monitor project workflow execution status
• Optimize project data processing workflows

Related Features

After completing workflow management, you may also need:

• Data Management: View data processed by workflows
• Annotation Tasks: View automatically created annotation tasks
• System Settings: Configure data pipeline related settings