A Halloween X-ray. Turn on the camera and see yourself as a skeleton, under a bed sheet, as a ghost or as a stick figure. The tracking runs in your browser, and the picture never leaves your device.
- Move
- the bones move along, fingers included
- Open your mouth
- and the jaw or the sheet’s mouth opens; blink and the eyes close
- Hands up
- without a face in view, that opens the jaw too
- No camera
- point somewhere and the skeleton looks and reaches; press and it opens its mouth
How it’s built
Four MediaPipe models look at the camera in two workers: your body and outline in one, your face and hands in the other. three.js draws what they see over the very frame they saw, mirrored, so the bones and the picture stay in step.
- Bones from Krea. Twelve 3D models, made with Krea from text or from a reference picture, and slimmed down to 3.4 MB. Each is an
InstancedMesh, placed from joint to joint with its front turned to you. - A skull that follows your face. The face model’s blendshapes open the jaw and move the lights in the sockets. Its transformation matrix turns the skull with your head. Every finger bone runs between two points of the hand model.
- A sheet that isn’t simulated. Physics needs depth, and the camera’s is too rough. So the top of the sheet lies on your outline, every column hangs down to a wavy hem, and the folds get deeper on the way. The hem follows you late, so it swings.
- A ghost from your outline. The selfie segmenter’s mask feeds a buffer that follows you late and drifts up. The light parts of the camera picture, blurred, glow brighter, so your face is just about there. It’s drawn as light that adds up with what’s behind it.
- Soft tissue. With the skeleton, the same outline glows faintly around the bones, like flesh on an X-ray. A blue rim in the shader does the rest.