Automated Video Data Labeler

The Data Labeling Bottleneck

In computer vision, video data is abundant — thousands of terabytes flow through IP-based camera networks daily. But labeled video data? That's the bottleneck. Manually scrubbing through hours of footage to find a specific event — a forklift violation, a safety breach, a product defect — is prohibitively expensive. Companies spend an estimated $25–$50 per hour on human annotators, leading to the rise of expensive services like AWS SageMaker Ground Truth and other data labeling vendors.

Breakthroughs in semantic video understanding allow us to invert this workflow entirely. Instead of manually hunting for events, we treat video as data that can be queried, clustered, and auto-labeled. This isn't just a time-saver — it's a force multiplier for deploying Vision Language Models (VLLMs) in production environments.

Core Features

Everything you need to go from raw footage to training-ready datasets.

How It Works

A three-step pipeline from raw video to training-ready dataset.

const task = await tl_client.tasks.create({
    indexId: indexId,
    videoUrl: videoURL,
    userMetadata: JSON.stringify({
        indexName: "Autonomous Driving",
        description: "Dashcam footage for perception model training"
    })
});

// Wait for TwelveLabs to finish processing
const completed = await tl_client.tasks.waitForDone(task.id, {
    sleepInterval: 5
});

console.log(`Video ${completed.videoId} indexed successfully`);

const prompt = `Analyze this video and generate annotations.
For each distinct event, provide:
- label: one of [${domainLabels.join(', ')}]
- start_timestamp: exact seconds when the event begins
- end_timestamp: exact seconds when the event ends
- description: brief description of what's happening

Return as JSON array.`;

const response = await fetch('/api/annotate', {
    method: 'POST',
    body: JSON.stringify({ videoId, prompt })
});

{
  "info": {
    "description": "Autonomous Driving Dataset",
    "date_created": "2026-02-15T00:00:00Z"
  },
  "videos": [
    { "id": 1, "file_name": "dashcam_001.mp4", "duration": 124.5 }
  ],
  "annotations": [
    {
      "id": 1,
      "video_id": 1,
      "category_id": 3,
      "start": 12.4,
      "end": 15.8,
      "description": "Vehicle executing left turn at intersection"
    }
  ],
  "categories": [
    { "id": 1, "name": "pedestrian_crossing" },
    { "id": 2, "name": "lane_change" },
    { "id": 3, "name": "car_turning_left" }
  ]
}

Understanding Your Data Through Embeddings

Every video indexed by TwelveLabs is represented as a 512-dimensional embedding vector generated by the Marengo 3.0 model. These vectors capture the semantic meaning of video content across visual, audio, and textual modalities.

The Embeddings tab in each index uses Principal Component Analysis (PCA) to project these high-dimensional vectors into 2D space. Videos that are semantically similar to each other appear as clusters — giving you an intuitive way to audit data quality, identify duplicates, and discover patterns before training.

// Build N×N gram matrix instead of dim×dim covariance
// This keeps computation O(N³) instead of O(dim³ = 512³)
const gram = Array.from({ length: N }, (_, i) =>
    Array.from({ length: N }, (_, j) =>
        centered[i].reduce((s, v, k) => s + v * centered[j][k], 0)
    )
);

// Power iteration to find top-2 eigenvectors
let v = Array.from({ length: N }, () => Math.random() - 0.5);
for (let iter = 0; iter < 100; iter++) {
    const next = gram.map(row =>
        row.reduce((s, g, j) => s + g * v[j], 0)
    );
    const norm = Math.sqrt(next.reduce((s, x) => s + x * x, 0));
    v = next.map(x => x / norm);
}

Why TwelveLabs?

TwelveLabs provides the foundational video understanding models that power every feature in this application.

From Curated Data to Business Impact

The output of this tool — structured, labeled datasets — is not just a file; it's an actionable asset that drives business intelligence and model performance.

Technology Stack