Live agent mode

Documentation Index

Fetch the complete documentation index at: https://docs.bowerlabs.ai/llms.txt

Use this file to discover all available pages before exploring further.

Live agent mode is a continuous, low-latency audio and video session with Bird. Unlike the standard chat interface — where you type a question and wait for a response — live mode is conversational. You can talk through an experiment, describe what’s on an instrument display, or ask follow-up questions without stopping to type. Live mode is available on both mobile and desktop. With video enabled, Bird can see what your camera sees (or your screen) and respond to visual context: “What’s the reading on that display?” or “Does this sample look contaminated to you?” are reasonable things to ask.

​

When to use it

• You’re mid-experiment and want to talk through your observations
• You need to describe something visual that would be difficult to type
• You want Bird to track key observations while you work
• You want a real-time second opinion before moving to the next step
• You want hands-free audio + video clips of a procedure as it happens

​

How to start a live session

​

Video awareness

With video enabled, Bird processes your camera feed in real time. It can:

• Read values from instrument displays you point the camera at
• Describe what it sees when asked
• Flag visual anomalies if you ask it to watch for them

Bird will not act on visual information unprompted — it responds when you ask.

​

Capturing notes during a session

You can ask Bird to capture what it sees or hears at any point during a live session:

• “Take a snapshot” — Bird captures the current camera frame and saves it as an attachment in your workspace.
• “Create a note from what we just discussed” — Bird creates a note with the key points from your conversation.
• You can also capture a photo manually during the session.

​

Recording video clips

You can ask Bird to record short audio + video clips of your bench work hands-free. While you’re in a live session, just say what you want and Bird records it. To start recording, say something like:

• “Record this”
• “Start recording”
• “Capture a clip of this”
• “Film this for me”

Bird confirms (“Recording now.”) and a red REC indicator appears on screen for as long as the recording is active. To stop, say:

• “Stop recording”
• “That’s enough, you can stop”
• “End the recording”

Bird confirms and the clip is saved to your workspace’s attachments as MP4, where it plays back inline. Bird is tuned to be conservative and only start recording when you explicitly ask. Phrases that mention “record” but mean something else — “recall the protocol”, “record my thoughts in my notebook”, “what’s the record temperature” — won’t trigger a recording.

​

Standby mode

When you need to pause — to have a private conversation, take a call, or step away — put Bird in standby. In standby, no audio, video, or any other data is sent from your device. The live stream is fully paused. Your microphone and camera are not active.

• To enter standby: Tap the standby button, or say “Standby Bird”
• To wake Bird: Say “Hello Bird”, or tap the standby button again to resume manually

The wake word “Hello Bird” is detected locally on your device — it does not require streaming audio to a server. Only once the wake word is recognised does the live stream resume. That means you can leave Bird active without keeping it always listening — keep your phone in your pocket with headphones, leave it on the lab bench, or leave a desktop session running while you take a call. Standby is useful for long lab sessions where you want Bird available but not always active.

​

Execution mode (running a protocol)

When you start a live session and ask Bird to run one of your saved protocols, the live agent enters a structured execution mode. Bird parses the protocol body into a stepwise checklist, walks you through it step by step, and tracks your progress so you can keep both hands on the bench.

​

Starting a protocol

• “Run the <name> protocol”
• “Start the <name> protocol”
• “Let’s do the <name> protocol”

If multiple protocols match what Bird heard, it will list them and ask which one — voice transcription can flip a letter (“workstation” vs “workspace”), so Bird is deliberately generous when searching.

​

The step strip

A translucent strip appears at the bottom of the live screen showing:

• Done steps — soft / ticked, behind you
• Current step — large, bold, accented in brand colour
• Upcoming steps — visible but de-emphasised

The current step auto-scrolls into view as you advance. You can scroll up or down to peek at done or upcoming work; the next advance re-centres.

​

Advancing through steps

Bird marks the current step done when you tell it you’re finished. Natural phrasings all work:

• “Done” / “That’s done” / “Next”
• “Move on” / “Finished that”

Your verbal confirmation is always an unconditional override — Bird won’t ask you to confirm or wait to visually verify. You said it, it’s done. You can also skip a step:

• “Skip this one” / “We don’t need this step”

​

Pausing and resuming

• “Pause for a sec” / “Hold on” — Bird stops the running clock and stays quiet until you resume
• “Resume” / “Where were we?” — Bird announces the current step again and continues
• “That’s done” / “We’re finished” (after the last step) — Bird ends the run and returns to general live mode

​

Timers

Inside a protocol you can set a wall-clock timer for the current step:

• “Remind me in 30 minutes to take the gel out”
• “Set a 5-minute timer”
• “Buzz me when the incubation is done in 10 minutes”

Bird reads the duration back so you can catch a misheard time, then quietly waits. When the timer fires, Bird speaks the reminder. Timers are scoped to the current step — moving on clears them. If you want a multi-step reminder, set it on the step it should fire during.

​

Reporting deviations

If something goes wrong or you do something differently than the protocol says, tell Bird. Examples:

• “I skipped the centrifuge step”
• “I used 50 microlitres, not 100”
• “I waited 10 minutes instead of 30”
• “That tube touched the bench”

Bird captures the deviation with the right category (timing, parameter, sterile breach, etc.) and the active step id, so you have a record of what actually happened — useful for troubleshooting a failed run later.

​

Bird AI annotations

Bird AI annotations are the lab-learnings Bird captures while you run a protocol live, so future runs of the same protocol get smarter. You don’t manage them by hand — you talk, Bird saves the right things.

​

What gets captured

• A specific value you state (“we use 50 µL of master mix here, not 100”, “set the temperature to 42 °C”)
• A correction to the protocol body (“we always do this cold”, “skip step 3”, “anneal at 58 °C, not 60”)
• A sterile-technique reminder (“change pipette tips between each reagent”, “vortex the tube before each addition”) — the cross-contamination class of mistake that ruins more PCRs than any other
• A named tool, reagent, or setting the protocol left open
• A deviation or near-miss (also kept as a separate record — see above)
• Anything you explicitly tell Bird to remember: “remember that…”, “make a note that…”, “for next time…”

You don’t have to use special phrasing. The first time you say “we use 50 µL of master mix, not the 100 in the protocol” during step 2, Bird saves that against step 2 of that protocol. The next time anyone in your workspace runs it, Bird already knows.

​

How they’re used next time

When you run the same protocol again, Bird loads its accumulated annotations as soft hints. As you reach each step, Bird weaves the relevant note into the announcement — “Heads up — your lab usually uses 50 µL here, not 100. Want to stick with that?” — rather than reading the bullets verbatim. If you repeat the same correction, Bird recognises it and reinforces the note (the bullet shows [seen N×] — higher N means more lab consensus). Reinforcement gives the model confidence to surface that note proactively in future runs.

​

Where to view and edit them

On the Protocol detail page, open the kebab menu (⋮) and choose Bird AI annotations. The dialog shows everything Bird has captured for that protocol, grouped by step. Every bullet has inline edit and delete controls. Use them when Bird captured something incorrectly (a misheard quantity, a typo, the wrong reagent) or when an old note no longer applies. Edits update the visible bullet text and the seen-count is preserved — re-saying the corrected note in a future run reinforces the same bullet. Deletes are immediate and can’t be undone from the modal. Any workspace member can edit or delete any bullet, including ones marked as admin-locked — the lock is a UI marker for who originally authored the bullet, not an authorization gate. Viewers (read-only members) can’t edit or delete.

​

What annotations are not

• Not a transcript — Bird isn’t recording every word you say. Only intent-bearing facts get saved.
• Not the protocol — annotations live in a companion artifact, not the protocol body. The canonical protocol stays unchanged so admins keep control of the SOP.
• Not visible to people outside the workspace — annotations inherit the parent protocol’s access controls.

​

Visual watchers — “every time you see X, say Y”

You can ask Bird to react to specific visual cues during any live session. Tell Bird what to watch for and what to say when it sees the thing, and Bird will keep an eye on the camera in the background. This works in any mode — you don’t need to be running a protocol.

​

How to set one up

Just say it in plain words:

• “Every time you see a thumbs up, say ‘cool, man.’”
• “Let me know when the gel turns pink.”
• “Watch for the colour change and tell me.”
• “Alert me if my hand touches the bench.”

Bird confirms it’s watching and the system starts checking camera frames every few seconds. When it spots a high-confidence match, Bird speaks the response you chose.

​

Limits

• Up to 3 watchers per session. Past the limit, Bird tells you and asks you to clear one first. Voice-clear with “forget about the thumbs up” or “stop watching for the gel”.
• About a 15-second cooldown per watcher. If the cue stays on camera, Bird won’t repeat itself every time. Wait a bit before the next firing.
• Watchers reset on disconnect. End the live session, and the watchers go with it. Re-register on the next session if you want them back.
• Each watcher costs a small amount. Bird uses a separate AI model to check frames, so there’s a per-session cost when watchers are active. Don’t leave watchers running all day — clear them when you’re done.

​

What works well

Concrete, visually unambiguous cues:

• ✅ “a researcher’s thumbs-up gesture”
• ✅ “the centrifuge display reaching 10000 rpm”
• ✅ “the gel turning pink”

What doesn’t work as well — vague targets or things you can’t see:

• ❌ “good things” — too abstract; the model can’t ground “good”.
• ❌ “when I’m done” — depends on your intent, not a visual.
• ❌ “the temperature reaching 37°C” if the readout isn’t on camera.

​

Asking Bird what it’s watching for

Just ask — “what are you watching for?” or “what cues did I set up?”. Bird lists the active watchers and how many times each has fired.

​

What’s saved

During a live session, notes, snapshots, and video clips are only saved when you explicitly ask Bird to capture them. Bird AI annotations are the exception — those are captured automatically inside execution mode so the lab-learning loop works without conscious effort. Visual watchers are also automatic in spirit — they fire whenever Bird sees the cue — but they live only for the duration of the live session. Nothing gets saved to your workspace from a watcher firing; Bird just speaks the response and continues. All captured notes, snapshots, and clips are searchable immediately and play back inline in the attachment viewer. Annotations are accessible from the Protocol detail kebab menu.