Yusuf's Thoughts

MediaPipe Face Tracking

4 min read

Want to make a digital avatar that smiles when you do? This recipe shows you how to use MediaPipe to track your face and facial expressions in real time. We’ll use this data to drive fun, interactive visuals.

Before You Start

You will need to download the MediaPipe TouchDesigner Plugin by Torin Blankensmith. Download it here (get the release.zip).

1. Setup the Plugin

  1. Extract the Zip: Unzip the plugin and place the toxes/ folder right next to your .toe project file.
  2. Add to TD:
    • Press Tab and add a Base COMP. Name it mediapipe_face.
    • Inside, drag in MediaPipe.tox and Face Tracking.tox.
  3. Turn it on:
    • Click the MediaPipe node.
    • In the parameters, find Face Tracking and toggle it On.
    • Set your Webcam in the dropdown.

2. Understanding the Data

MediaPipe gives us two main types of data:

  1. Landmarks: 468 points on your face (eyes, nose, lips).
  2. Blend Shapes (Expressions): These are numbers from 0 to 1 that represent how much you are doing a specific expression (like mouthSmile or browInnerUp).

What is a Blend Shape?

Think of it as a slider. If your mouth is closed, mouthSmile is 0.0. If you’re grinning ear-to-ear, it’s 1.0.

3. Extracting Your Smile

Let’s grab your smile and use it to control something!

  1. Add a Select CHOP and connect it to the output of the Face Tracking node.
  2. In the Channel Names parameter, type: F1_mouthSmileLeft.
  3. Add a Math CHOP to make it more sensitive if needed.
  4. Add a Null CHOP and name it MY_SMILE.

4. Your First Project: The “Glowing Smile”

Let’s make a circle that grows and changes color when you smile.

  1. The Circle: Add a Circle TOP.
  2. Control the Size:
    • Make the MY_SMILE Null active (click the + icon).
    • Drag the channel onto the Radius parameter of the Circle TOP.
    • Select CHOP Bind.
  3. Control the Color:
    • Drag the same channel onto the Color R (Red) parameter.
    • Now the circle will turn redder the more you smile!

Troubleshooting

  • “The plugin is red/erroring.” — Make sure the toxes/ folder is in the same folder as your project file.
  • “It’s really slow.” — MediaPipe is heavy. Go to the MediaPipe COMP and make sure Face Tracking is the only model turned on.
  • “My face isn’t being detected.” — Ensure you have good lighting! If your room is dark, the AI will struggle to find your features.

Next Steps

  • Puppet an Avatar: Map your jaw opening (F1_jawOpen) to a 3D character’s mouth.
  • Eyebrow Control: Use F1_browInnerUp to trigger a burst of particles.
  • 3D Mesh: Check out the Face Mesh output to see a full 3D wireframe of your face.

Parameter Tuning & Behavior

ParameterBehavior
mouthSmile intensity0 to 1; drives how much the visual “reacts” to your happiness.
Math MultiplierHigher = makes the visual react to even a tiny smirk; Lower = requires a wide grin.
Lag / SmoothingHigher = visual reacts slowly and “fluidly”; Lower = visual is “twitchy” and instant.
Bloom IntensityHigher = smile becomes a blinding light; Lower = subtle, soft glow.

Network Architecture

To visualize how the face tracking data flows, here is the final network map:

[ VIDEO INPUT ]                  [ MEDIAPIPE PLUGIN ]
Webcam TOP ──────────────────▶ [ MediaPipe.tox ]
                                      │
                                      ▼
[ DATA DECODING ]              [ Face Tracking.tox ]
                                      │
                                      ▼
[ EXPRESSIONS ]                [ Select CHOP ] (F1_mouthSmile*)
                                      │
                                      ▼
[ SMOOTHING ]                  [ Lag / Filter CHOP ]
                                      │
                                      ▼
[ EXPORT ]                     [ Null CHOP (MY_SMILE) ]
                                      │
                                      ▼
[ VISUALS ]                    [ Circle TOP ] ◀──────────────┐
                               (Radius Bind)                 │
                                      │                      │
[ RENDERING ]                  [ Bloom TOP ] ──▶ [ HSV Adjust ] ──▶ [ OUT ]

Data Flow Explanation

  1. Plugin Layer: MediaPipe.tox is the engine. It runs the “Face Mesh” model in an embedded browser and sends the 468 landmark points into TouchDesigner.
  2. Blend Shapes: The Face Tracking.tox component decodes those 468 points into high-level “Blend Shapes” (e.g., mouthSmile, eyeBlink, jawOpen).
  3. Data Extraction: We use a Select CHOP to grab just the mouthSmileLeft channel. This gives us a 0.0 to 1.0 value representing the intensity of your smile.
  4. The Bridge: By Binding this CHOP value to the Radius of a Circle TOP, we bridge the physical world (your face) to the digital world (the circle).
  5. Polishing: The Bloom TOP adds a glow that intensifies as you smile wider, creating a direct visual feedback loop for the user.

(y) Return to Recipes & Projects | (y) Return to TouchDesigner | (y) Return to Home

Graph View