Learning Augmented Reality with MediaPipe

Augmented reality always looked like a technology that needed expensive hardware. It doesn't. A webcam and a JavaScript library are enough to put a virtual object in front of a person and have that person move it with their hand — which is exactly the kind of thing I wanted to understand rather than just watch in a video.

The live version is the hand-controlled cube, and it is a single client component: one <video> with the webcam feed, one transparent WebGL canvas stacked on top of it, and a requestAnimationFrame loop that runs between them.

#Why hand tracking is a good first AR problem

Most "AR" demos are really camera demos: they draw something on top of the video and call it a day. Hand tracking is more interesting because it closes a loop. The user moves, the software measures, the virtual object responds, and the response has to be stable enough that the loop feels direct.

The other reason is that it forces you to deal with two coordinate systems at once:

• the camera space, where the landmarks are measured, normalised to 0 — 1
• the scene space, where the virtual object lives, in Three.js world units

#The stack

Next.js with React, three for the scene, and @mediapipe/tasks-vision for the tracking. The current Tasks API replaces the old @mediapipe/hands wrapper: there is no Hands class and no onResults callback. You resolve the WASM runtime, create a HandLandmarker, and pull a result out of it once per frame.

// Camera.js
const vision = await FilesetResolver.forVisionTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision/wasm"
);
 
handLandmarker = await HandLandmarker.createFromOptions(vision, {
  baseOptions: {
    // Bundled in /public/models, so the model is served from our own origin.
    modelAssetPath: "/models/hand_landmarker.task",
  },
  runningMode: "VIDEO",
  numHands: 1,
});

Two deliberate choices there. runningMode: "VIDEO" tells the landmarker that frames arrive in order and carry timestamps, which is what makes detectForVideo valid. numHands: 1 keeps the model fast and stops one hand from flickering into two — the same reason the old API had maxNumHands.

The model file lives in public/models/hand_landmarker.task rather than on a CDN. Only the WASM runtime comes from jsDelivr; the weights are ours, which means the first load is a static asset on the same origin instead of a second remote dependency.

The landmark set is 21 points per hand, which is small enough to reason about. The ones I have actually wired up so far:

#Reading the landmarks

The detection call is synchronous and you own the timing. Everything happens inside a requestAnimationFrame loop that I throttle to 30 Hz, because there is no point running the model faster than the display can consume the result:

// Camera.js
const now = performance.now();
 
// Only run MediaPipe ~30 times per second
if (now - lastDetectionTime >= detectionInterval) {
  lastDetectionTime = now;
 
  const results = handLandmarker.detectForVideo(video, now);
 
  if (results.landmarks.length > 0) {
    const landmarks = results.landmarks[0];
 
    const indexFinger = landmarks[8]; // Index fingertip
    const thumbfinger = landmarks[4]; // Thumb tip
 
    // ...
  }
}

detectForVideo wants a monotonically increasing timestamp, so now from performance.now() is passed straight through. The render call and the cube interpolation stay outside that throttle block, so the scene still animates at the display refresh rate while the model runs at 30.

#Normalising the coordinates

Each landmark is { x, y, z } where x and y are normalised to the frame and z is a rough relative depth. That space is convenient for the model and awkward for the scene, so each point gets mapped into world units:

// Camera.js
// Normalised 0—1 → centred world units. Negated so movement follows the user.
const rawX = -(indexFinger.x - 0.5) * 6;
const rawY = -(indexFinger.y - 0.5) * 4;

The - 0.5 recentres the frame on the origin, the negation mirrors the axis, and the 6 and 4 are the half-extents of the reach box in world units: x spans -3 … 3, y spans -2 … 2. Those numbers are the 4:3 shape of the canvas scaled up by a fixed factor, which is why they are 6 and 4 rather than 2 and 2. With the camera at z = 5 and a 75° field of view, that box sits comfortably inside the frustum — the cube uses roughly the middle half of the frame and never clips the edges, which turns out to be a nicer feel than mapping the full frame.

Mirroring is handled twice, on purpose. The <video> element carries transform: scaleX(-1) so the feed looks like a mirror, and the - in the rawX expression mirrors the landmark back into that same convention. Change one and you have to change the other, or the cube moves the wrong way relative to your hand.

#Driving a cube

The scene itself is small: a PerspectiveCamera(75, 640 / 480, 0.1, 1000) at z = 5, a default BoxGeometry, and a MeshBasicMaterial in green on a renderer created with alpha: true. Because the material is unlit, the scene needs no lights at all — and because the canvas is transparent, the cube appears to float over the webcam feed. The canvas is absolutely positioned over the video with pointerEvents: "none", so it never steals a click.

// Camera.js
const targetPosition = new THREE.Vector3();
const smoothedHand = new THREE.Vector2();
 
let isPinching = false;
 
// ...
 
if (isPinching) {
  const rawX = -(indexFinger.x - 0.5) * 6;
  const rawY = -(indexFinger.y - 0.5) * 4;
 
  const inputSmoothing = 1;
  smoothedHand.x += (rawX - smoothedHand.x) * inputSmoothing;
  smoothedHand.y += (rawY - smoothedHand.y) * inputSmoothing;
 
  targetPosition.x = smoothedHand.x;
  targetPosition.y = smoothedHand.y;
}
 
// Runs every frame, not just on detection frames
cube.position.lerp(targetPosition, 0.30);

Two things are worth being honest about here.

The first is that inputSmoothing = 1, which makes those two lines a pass-through: smoothedHand lands exactly on rawX. The variable is a hook I left in place for tuning, but at its current value it does nothing, and the frame-to-frame smoothing the cube actually gets comes entirely from the lerp(…, 0.30). That single call is doing more for perceived quality than anything else in the file. Raw landmarks carry a small amount of noise every frame, and snapping the cube to the raw value turns that noise into visible vibration; interpolating about 30% of the way to the target each frame removes most of it without the drag-through-syrup feeling you get when the factor is too low. Because the target only updates on detection frames while the lerp runs on every frame, the cube converges smoothly between measurements instead of arriving in 30 little steps.

The second is that the cube only moves while isPinching is true. Pinching is a clutch, not a modifier — release and the cube stays exactly where you left it. That is what makes it possible to point at a target, pinch, drag, and let go, without a constant hand-relative offset fighting you.

#Pinch detection

The pinch test is a plain 2D distance in normalised space:

// Camera.js
const dx = thumbfinger.x - indexFinger.x;
const dy = thumbfinger.y - indexFinger.y;
const distance = Math.sqrt(dx * dx + dy * dy);
 
if (distance < 0.05) {
  console.log("Pinching!");
  isPinching = true;
} else {
  isPinching = false;
}

0.05 is about 5% of the frame width — a tight but reachable threshold for thumb and index tip meeting at the same point. Note what this measurement does not do: it is an absolute distance, so it is sensitive to how far the hand is from the camera. The textbook version normalises against hand size, using the wrist-to-middle-base distance as a scale reference:

// Not in the code yet — this is the version that survives zooming in and out.
const handScale = distance(landmarks[0], landmarks[9]);
const pinching = pinchDistance / handScale < 0.35;

That is the first thing on my list, because right now the gesture gets fussier the further back you sit. The console.log in that branch is also called at up to 30 times a second while pinching, which is exactly the kind of thing that makes the console unusable when you actually need it.

#Setup order and cleanup

The effect has a few ordering constraints that are easy to get wrong. The landmarker and the webcam stream resolve asynchronously, the detection loop can only start once frames exist, and any of those steps can finish after the component has unmounted. So each await is followed by a bail-out:

// Camera.js
let cancelled = false;
 
const vision = await FilesetResolver.forVisionTasks(/* ... */);
if (cancelled) return;
 
handLandmarker = await HandLandmarker.createFromOptions(vision, /* ... */);
if (cancelled) return;
 
const stream = await navigator.mediaDevices.getUserMedia({
  video: true,
  audio: false,
});
 
if (cancelled) {
  // Nothing is going to consume this stream — release the camera.
  stream.getTracks().forEach((track) => track.stop());
  return;
}
 
videoRef.current.srcObject = stream;
videoRef.current.addEventListener("loadeddata", detectHands);

The stream check matters most. Without it, unmounting while the permission prompt is open leaves the camera light on with no component to stop it. The loop itself starts on loadeddata, which guarantees videoRef.current has decoded a frame before detectForVideo is handed the element.

Teardown mirrors that: set the flag, cancel the animation frame, remove the loadeddata listener, stop every track, dispose the renderer, geometry and material, and remove the canvas from the overlay div.

#What tripped me up

• Mirroring, twice over. The scaleX(-1) on the video and the negation in the x mapping are two halves of one decision. Getting one without the other produces a cube that moves beautifully — in the wrong direction.
• The reach box is not the frame. 6 and 4 are a deliberate comfort zone, not an aspect-ratio correction. If the webcam hands you a 16:9 frame, objectFit: "cover" on a 4:3 element crops the left and right edges of what the model sees, so landmarks near the edge of the camera frame map to a cube position that is off the visible video entirely.
• Smoothing has to be earned. I named a variable inputSmoothing, set it to 1, and spent a while wondering why the cube was still jittery. The smoothing was in the lerp, one line further down.
• Guard clauses that kill the loop. detectHands returns early when the landmarker is not ready — and that return skips the requestAnimationFrame call, so the loop would never restart. Today the setup order means it cannot trigger, but reordering the setup would silently freeze the whole thing.
• The old scene is still in the repo. There is an unused mouse-driven version of the Three.js scene left over from the first experiment. Only the camera component is rendered now, and the leftover has a stray font import in it.
• Log the landmark confidence. I have not done this yet. Tasks Vision puts a handedness score in the results and I am ignoring it — but when a hand "disappears", the reason is almost always a threshold, and it is much easier to see that in a number than in a video.

Hand tracking is a measurement problem before it is a rendering problem. The graphics are the easy half.

#Next steps

Real occlusion is the obvious next step, so the cube can pass behind a finger instead of always drawing on top of it. After that: normalising the pinch by hand scale so the gesture works at any distance, using the wrist and middle landmarks for rotation, and two-hand interactions — going from one hand to two is mostly bookkeeping, but retrofitting it after the single-hand path is everywhere is worse. And a webcam-free fallback, so the page still makes sense to someone who does not want to turn on a camera.