Nodaro DocsSpeaker Frames
Find where each speaker's face is, over time, on every camera of a multicam edit, and get a face-track file that Speaker View's tracked framing reads.
Available on Nodaro Cloud
The Speaker Frames node finds where each speaker's face is, over time, on every camera of an edit. It writes the result to a stored face-track file, with boxes per face on each camera's own clock. The Face tracks output carries a summary of that file. Wire it into the Face tracks input of Speaker View, or click Add face tracking there, and each speaker's crop follows their face.
Speaker Frames has no credit price at the moment. Every run is refused before anything is charged, with the message "Speaker Frames is not priced yet". The node, its settings panel and the Run button say so. You can place the node, connect it and set it up.
Speaker Frames runs on Nodaro Cloud. On a self-hosted install, connect the install to Nodaro Cloud. A connected install builds small detection copies of the cameras it samples and sends only those, never the original files. The master audio and any camera you untick are not sent at all. At the moment, a run from a connected install is refused with the message "Speaker Frames is not available on a connected install yet".
When to use it
- You want the framing of a podcast or interview render to follow each speaker's face.
- You have a multicam edit from Edit Plan or Camera Switch, and you need to know where each person is on every camera.
- You want face positions for one video, without an edit.
Quick start
Add the node
Press Tab on the canvas and choose Video › Cut & Assemble › Speaker Frames.
Connect the edit
Wire the Switched edit (EDL) output of Camera Switch into the EDL input. Wire the Transcript (named) output of Camera Switch into the Transcript input.
Choose the cameras
Open the settings panel. Under Cameras to track, untick any camera you do not need tracked, for example a close-up that a render shows full frame.
Inputs
| Input | Required | What to connect |
|---|---|---|
| EDL | One of EDL and Video | The edit, from Edit Plan or Camera Switch. A clip pack runs once: when you wire the Clips output of Edit Plan, every clip is folded into one run over all the footage the clips keep, never one run per clip. |
| Video | One of EDL and Video | One video, sampled whole, up to 180 minutes. |
| Transcript | No | The Transcript (named) output of Camera Switch, with its speaker names. The tracks are then labelled with the names the edit uses. Wire the same transcript that Camera Switch outputs. |
Connect exactly one of EDL and Video. With both connected, the run is refused before anything is charged.
Outputs
| Output | What it carries |
|---|---|
| Face tracks | A summary of the stored face-track file: its address, its SHA-256 hash and size, the sample rate, and for each camera its sampled stretches, scene cuts and tracks. Each track has an id, the speaker when one is known, and its number of boxes. The boxes themselves are in the stored file. |
On a connected self-hosted install, the face-track file is copied into the install's own storage after the run, and its hash is checked again. The output then points at that copy.
Settings
| Setting | What it does |
|---|---|
| Cameras to track | Every camera the connected edit samples, with its number of frames. Untick a camera you do not need tracked, and it is not sampled. A camera you unticked that is no longer in the edit is ignored when the node runs. |
The node samples only what the edit keeps: each kept stretch, plus 2 seconds on each side, at 2 frames per second, on every video camera the edit shows. The sample rate is fixed. A camera whose kept footage is longer than 180 minutes is refused before anything is charged.
Corrections to speaker labels
The node keeps corrections to the speaker of each track, by track id. A correction can also clear a track's label. You set the corrections in Edit framing… on Speaker View: it draws the tracked faces over each camera, and choosing who a face is saves the correction on this node. The settings panel, the summary on the canvas and Speaker View apply them. A correction keeps the transcript's own label, so after you rename a speaker in Camera Switch, the correction follows the new name. The Face tracks output carries the result as the run wrote it, without the corrections. A new run keeps the corrections whose tracks it still finds. The panel lists any correction whose track the new result no longer has, so a correction is never dropped without notice.
Credits
Speaker Frames has no credit price at the moment. Every run is refused with the message "Speaker Frames is not priced yet", and nothing is charged.
Tips
- Run Camera Switch first on a multicam shoot, and wire its named transcript, so the tracks carry the speaker names the edit uses.
- Untick the cameras you do not need. A close-up that a render shows full frame needs no tracking, and fewer cameras mean fewer frames to sample.
- Cut the edit before you track. The node samples only the footage the edit keeps, so a tighter edit is sampled faster.
Troubleshooting
| Message or problem | Cause and fix |
|---|---|
| "Speaker Frames is not priced yet" | Speaker Frames has no credit price at the moment, so every run is refused. |
| "Wire an EDL into “EDL”, or a video into “Video”" | Nothing is connected. Connect an edit or a video. |
| The run is refused with both inputs connected. | Connect only one of EDL and Video. |
| A camera is refused as too long. | The camera's kept footage is longer than 180 minutes. Keep less footage, or untick that camera. |
| The tracks are not labelled with the right names. | Wire the Transcript (named) output of Camera Switch, not the transcript of Transcribe. |
From the API
POST /v1/speaker-frames takes edl or videoUrl, exactly one of them, with optional transcript and excludeSourceIds. edl is one EDL, a clip pack (an array of EDLs) or a clip set, as objects or JSON strings. excludeSourceIds lists the cameras you unticked. The finished job's output_data.json is the summary of the face-track file.
Requests are refused before anything is charged:
| Error | Cause |
|---|---|
400 invalid_edl | The request is malformed, or the edit is not on the recording's clock. |
400 invalid_input | Both or neither of edl and videoUrl are sent, or excludeSourceIds names no camera of the edit. |
400 too_long | A camera is longer than 180 minutes. |
503 not_priced | Speaker Frames has no credit price at the moment. |
On a self-hosted install, the route answers 503 nodaro_connection_required without a Nodaro Cloud connection, or when Nodaro Cloud rejects the install's connection. Reconnect from Integrations. It answers 503 speaker_frames_relay_unavailable while Nodaro Cloud does not take runs from a connected install, or cannot be reached. If a detection copy the install built is larger than the 500 MB a relay can send, the job fails and names the camera. Keep less footage or untick that camera.
GET /v1/speaker-frames/capabilities answers { relay, source }: whether this server runs a Speaker Frames job relayed from a connected self-hosted install. source is server, nodaro.ai, nodaro.ai-unreachable, nodaro.ai-rejected or not-connected. See Run a single node and Jobs.
Frequently asked questions
What does the Speaker Frames node do?
Speaker Frames finds where each speaker's face is, over time, on every camera of an edit. It stores the boxes in a face-track file, and its Face tracks output carries a summary of that file for Speaker View's tracked framing.
Can I run Speaker Frames now?
Not yet. Speaker Frames has no credit price at the moment, so every run is refused with the message "Speaker Frames is not priced yet" before anything is charged. You can place and set up the node.
Does Speaker Frames analyze the whole recording?
No. With an EDL connected, it samples only the footage the edit keeps, plus 2 seconds on each side, at 2 frames per second. A 90-minute episode cut down to 60 minutes is sampled for about 60 minutes.
Does a clip pack run Speaker Frames once per clip?
No. When you connect the Clips output of Edit Plan, every clip is folded into one run over all the footage the clips keep.
Which transcript should I connect?
Connect the Transcript (named) output of Camera Switch, the same transcript Camera Switch outputs. The face tracks are then labelled with the speaker names the edit uses.
Related
Speaker View
Camera Switch
Edit Plan
Apply EDL
Transcribe
Last updated on
Camera Switch
Put each cut of a multicam podcast or interview edit on the camera of whoever is speaking, from a transcript with speaker labels. The sound never changes.
Speaker View
Render a multicam edit with its speakers on screen, side by side, stacked, in a grid or picture-in-picture, with an emphasis on whoever is talking.