Krida

An engine for building interactive experiences with real-time camera-based tracking.

Status: Pre-1.0. The API may change between releases.

Overview

Implement Scene, pass a <video> element to Session, and receive per-frame tracking data in your updateTracker() callback. Session manages webcam setup, model loading, and cleanup. Tracking currently runs on MediaPipe models; additional backend support is planned.

Installation

npm install krida @mediapipe/tasks-vision

Note: MediaPipe model files and WASM bundle must be served. See sandbox/ for a CDN-based example.

Quick Start

import { Session, fitCanvasToVideo, type Scene, type TrackerResult } from "krida";

// Get references to the HTML elements.
const video = document.querySelector("video") as HTMLVideoElement;
const canvas = document.querySelector("canvas") as HTMLCanvasElement;
const ctx = canvas.getContext("2d")!;

// Keep the canvas drawing resolution matched to the camera's intrinsic size.
fitCanvasToVideo(canvas, video);

// Define a scene.
const pointerScene: Scene = {
  updateTracker(result: TrackerResult) {
    ctx.clearRect(0, 0, canvas.width, canvas.height);
    for (const hand of result.hand?.landmarks ?? []) {
      const tip = hand[8]; // index finger tip
      ctx.fillRect(tip.x * canvas.width, tip.y * canvas.height, 4, 4);
    }
  },
};

// Create and start a Krida session with the scene.
const session = new Session();
await session.start({
  video,
  scenes: [pointerScene],
  visionEngineOptions: {
    handLandmarkerEnabled: true,
    visionTaskFilesetPath: "/models/tasks-vision-wasm",
    handLandmarkerModelPath: "/models/hand_landmarker.task",
  },
});

Minimal HTML:

<video autoplay playsinline muted></video>
<canvas></canvas>

Note: Landmark coordinates are normalized to [0, 1], so multiply by your canvas dimensions when drawing.

Krida runs in the browser and uses getUserMedia(), so camera access requires a secure context such as https:// or http://localhost.

Core Concepts

Scene — the interface you implement. Provide an updateTracker(trackerResult) method called once per processed frame, and optional onStart / onStop lifecycle hooks. A session can run several scenes at once.

Session — top-level orchestrator. Acquires the webcam, initializes VisionEngine and FrameProcessor, forwards each frame's results to every managed scene, and manages cleanup on page hide/unload.

VisionEngine — runs tracking inference each frame, caches per-frame results, smooths landmark output via built-in filtering, and applies any configured gesture detectors.

FrameProcessor — passes video frames to the engine and hands each TrackerResult to its callback, in a loop or on demand via update(). Supports an FPS cap for the loop and an optional debug overlay canvas.

API Reference

Scene Interface

interface Scene {
  updateTracker(result: TrackerResult): void;
  onStart?(): void;
  onStop?(): void;
}

TrackerResult.hand contains per-hand landmarks arrays (normalized {x, y, z} points). TrackerResult.pose contains pose landmark data. Both may be undefined if the respective tracker is disabled.

updateTracker() fires at most once per camera frame; frames where the video has not advanced, or arrive faster than targetFPS, are skipped.

Session

new Session(options?: SessionOptions)
session.start(options: SessionStartOptions): Promise<void>
session.destroy(): void
session.addScene(...scenes: Scene[]): void
session.removeScene(scene: Scene): boolean
session.update(timestampMs?: number): TrackerResult | null
session.isActive: boolean

SessionOptions.autoCleanupOnPageLifecycle (default: true) automatically calls destroy() when the page is hidden or unloaded.

SessionStartOptions:

Use frameMode: "manual" when another render loop owns scheduling. Call session.update(timestampMs) from that loop; it processes at most one new video frame and returns null if the session is not active or no new video frame is available. It throws if the session is active but frameMode is not "manual".

timestampMs must be in milliseconds and monotonic. performance.now() and a requestAnimationFrame callback's timestamp work. A seconds-based elapsed-time value, such as three.js's Clock.getElapsedTime(), does not.

addScene() throws if the session is not active; pass scenes via start() instead of adding them beforehand. destroy() is safe to call multiple times and calls onStop() on every active scene.

With debugView: true, Krida creates its own absolutely-positioned overlay canvas and appends it to the video's parent element, so give that parent position: relative.

VisionEngine

Created internally by Session, or directly via VisionEngine.create(options).

Key options:

Defaults are exported as VisionEngineDefaults.

FrameProcessor

new FrameProcessor(visionEngine, options?: FrameProcessorOptions, monitor?: PerformanceMonitor | null)
processor.bind(video: HTMLVideoElement, onResult?: (result: TrackerResult) => void): void
processor.startLoop(): void
processor.update(timestampMs?: number): TrackerResult | null
processor.stopLoop(): void
processor.destroy(): void
processor.debugCanvas: HTMLCanvasElement | null   // setter only
processor.isLooping: boolean

FrameProcessorOptions:

Call bind() before startLoop(), or call update() to process frames from an external scheduler. stopLoop() keeps the binding, so startLoop() resumes the loop. destroy() stops the loop and releases the binding but leaves the VisionEngine intact, so a shared engine can outlive the processor.

Prefer debugView: true on session.start() unless you want to supply and position the overlay canvas yourself.

Gestures

Attach detectors to the engine, then read their state off the tracker result:

import { PinchDetector } from "krida";

await session.start({
  video,
  scenes: [pointerScene],
  visionEngineOptions: {
    handLandmarkerEnabled: true,
    handGestureDetectors: [new PinchDetector()],
  },
});

// Inside pointerScene:
updateTracker(result) {
  const pinch = result.hand?.gestures?.get("pinch");
  if (pinch?.justActivated) {
    console.log("pinched at", pinch.position);
  }
}

result.hand.gestures is a GestureMap: get(name), active(), justActivated(), justDeactivated(), and iterable as [name, state] pairs.

Each GestureState carries confidence, isActive, justActivated, justDeactivated, activeSince, and position.

Every detector accepts GestureOptions to tune its activation state machine: name, activateAt, deactivateAt (must not exceed activateAt), and holdFrames. Subclass GestureDetector to add your own.

PerformanceMonitor

new PerformanceMonitor(options?: PerformanceMonitorOptions)
monitor.snapshot(): PerformanceSnapshot

Pass a PerformanceMonitor instance to session.start(). Call snapshot() at any time to get metrics such as actualFPS, frameTime, handInference, poseInference, model init timings, and more.

fitCanvasToVideo

fitCanvasToVideo(canvas: HTMLCanvasElement, video: HTMLVideoElement, signal?: AbortSignal): void

Sets the canvas drawing resolution (canvas.width / canvas.height) to the video's intrinsic dimensions and keeps them in sync on resize. Does not affect the canvas CSS layout size. Pass an AbortSignal to detach the listener.

Examples

See sandbox/ for a minimal working implementation with hand-tracking, pose-tracking, debug overlay, performance monitor, and CDN-hosted models.

License

MIT