Live agent mode

Live agent mode is a continuous, low-latency audio and video session with Bower. Unlike the standard chat interface — where you type a question and wait for a response — live mode is conversational. You can talk through an experiment, describe what’s on an instrument display, or ask follow-up questions without stopping to type. Live mode is available on both mobile and desktop. With video enabled, Bower can see what your camera sees (or your screen) and respond to visual context: “What’s the reading on that display?” or “Does this sample look contaminated to you?” are reasonable things to ask.

​

When to use it

• You’re mid-experiment and want to talk through your observations
• You need to describe something visual that would be difficult to type
• You want Bower to track key observations while you work
• You want a real-time second opinion before moving to the next step
• You want hands-free audio + video clips of a procedure as it happens

​

How to start a live session

​

Video awareness

With video enabled, Bower processes your camera feed in real time. It can:

• Read values from instrument displays you point the camera at
• Describe what it sees when asked
• Flag visual anomalies if you ask it to watch for them

Bower will not act on visual information unprompted — it responds when you ask.

​

Capturing notes during a session

You can ask Bower to capture what it sees or hears at any point during a live session:

• “Take a snapshot” — Bower captures the current camera frame and saves it as an attachment in your workspace.
• “Create a note from what we just discussed” — Bower creates a note with the key points from your conversation.
• You can also capture a photo manually during the session.

​

Recording video clips

You can ask Bower to record short audio + video clips of your bench work hands-free. While you’re in a live session, just say what you want and Bower records it. To start recording, say something like:

• “Record this”
• “Start recording”
• “Capture a clip of this”
• “Film this for me”

Bower confirms (“Recording now.”) and a red REC indicator appears on screen for as long as the recording is active. To stop, say:

• “Stop recording”
• “That’s enough, you can stop”
• “End the recording”

Bower confirms and the clip is saved to your workspace’s attachments as MP4, where it plays back inline. Bower is tuned to be conservative and only start recording when you explicitly ask. Phrases that mention “record” but mean something else — “recall the protocol”, “record my thoughts in my notebook”, “what’s the record temperature” — won’t trigger a recording.

​

Standby mode

When you need to pause — to have a private conversation, take a call, or step away — put Bower in standby. In standby, no audio, video, or any other data is sent from your device. The live stream is fully paused. Your microphone and camera are not active.

• To enter standby: Tap the standby button, or say “Standby Bower”
• To wake Bower: Say “Hello Bower”, or tap the standby button again to resume manually

The wake word “Hello Bower” is detected locally on your device — it does not require streaming audio to a server. Only once the wake word is recognized does the live stream resume. That means you can leave Bower active without keeping it always listening — keep your phone in your pocket with headphones, leave it on the lab bench, or leave a desktop session running while you take a call. Standby is useful for long lab sessions where you want Bower available but not always active.

​

Execution mode (running a protocol)

When you start a live session and ask Bower to run one of your saved protocols, the live agent enters a structured execution mode. Bower parses the protocol body into a stepwise checklist, walks you through it step by step, and tracks your progress so you can keep both hands on the bench.

​

Starting a protocol

• “Run the <name> protocol”
• “Start the <name> protocol”
• “Let’s do the <name> protocol”

If multiple protocols match what Bower heard, it will list them and ask which one — voice transcription can flip a letter (“workstation” vs “workspace”), so Bower is deliberately generous when searching.

​

The step strip

A translucent strip appears at the bottom of the live screen showing:

• Done steps — soft / ticked, behind you
• Current step — large, bold, accented in brand color
• Upcoming steps — visible but de-emphasised

The current step auto-scrolls into view as you advance. You can scroll up or down to peek at done or upcoming work; the next advance re-centers.

​

Advancing through steps

Bower marks the current step done when you tell it you’re finished. Natural phrasings all work:

• “Done” / “That’s done” / “Next”
• “Move on” / “Finished that”

Your verbal confirmation is always an unconditional override — Bower won’t ask you to confirm or wait to visually verify. You said it, it’s done. You can also skip a step:

• “Skip this one” / “We don’t need this step”

​

Jumping to a specific step

When you don’t want the next step but a particular one — by number or by what it’s about — just say so:

• “Jump to step 5” / “Go back to the staining step” / “Let’s do the gas setup now”

Because landing on the wrong step mid-procedure is disruptive, a jump is confirmed before it happens. Bower works out which step you meant and reads its title back as a quick check — “Jump to ‘Set up gas lines’ — that right?” — and waits. Say “yes” / “do it” / “go ahead” and Bower makes the jump. Say no, or name a different step, and it simply moves on to that instead — nothing jumps until you confirm. Marking a step done, skipping, and going back all stay instant — only a jump asks for confirmation. If Bower isn’t sure which step you meant, it asks you to say which one rather than guessing.

​

Pausing and resuming

• “Pause for a sec” / “Hold on” — Bower stops the running clock and stays quiet until you resume
• “Resume” / “Where were we?” — Bower announces the current step again and continues
• “That’s done” / “We’re finished” (after the last step) — Bower ends the run and returns to general live mode

​

Timers

Inside a protocol you can set a wall-clock timer for the current step:

• “Remind me in 30 minutes to take the gel out”
• “Set a 5-minute timer”
• “Buzz me when the incubation is done in 10 minutes”

Bower reads the duration back so you can catch a misheard time, then quietly waits. When the timer fires, Bower speaks the reminder. Timers are scoped to the current step — moving on clears them. If you want a multi-step reminder, set it on the step it should fire during.

​

Reporting deviations

If something goes wrong or you do something differently than the protocol says, tell Bower. Examples:

• “I skipped the centrifuge step”
• “I used 50 microliters, not 100”
• “I waited 10 minutes instead of 30”
• “That tube touched the bench”

Bower captures the deviation with the right category (timing, parameter, sterile breach, etc.) and the active step id, so you have a record of what actually happened — useful for troubleshooting a failed run later.

​

Remembering something for next time

When you tell Bower something worth keeping during a protocol run, such as “remember that we use 50 µL here” or “make a note that the buffer was cold”, Bower writes it into a note anchored to that run and tells you so: “Got it, putting that in a note for this run.” The note appears in your Capture Stream straight away and is searchable like any other note. Bower does not carry these notes into future runs. It won’t read a tip back to you when you reach the same step next time, so if a change should apply to every run, edit the protocol itself. If you ask for a note without saying what it should contain (“take a note”), Bower starts a voice note instead and records what you dictate next.

​

Visual watchers — “every time you see X, say Y”

You can ask Bower to react to specific visual cues during any live session. Tell Bower what to watch for and what to say when it sees the thing, and Bower will keep an eye on the camera in the background. This works in any mode — you don’t need to be running a protocol.

​

How to set one up

Just say it in plain words:

• “Every time you see a thumbs up, say ‘cool, man.’”
• “Let me know when the gel turns pink.”
• “Watch for the color change and tell me.”
• “Alert me if my hand touches the bench.”

Bower confirms it’s watching and the system starts checking camera frames every few seconds. When it spots a high-confidence match, Bower speaks the response you chose.

​

Limits

• Up to 3 watchers per session. Past the limit, Bower tells you and asks you to clear one first. Voice-clear with “forget about the thumbs up” or “stop watching for the gel”.
• About a 15-second cooldown per watcher. If the cue stays on camera, Bower won’t repeat itself every time. Wait a bit before the next firing.
• Watchers reset on disconnect. End the live session, and the watchers go with it. Re-register on the next session if you want them back.
• Each watcher costs a small amount. Bower uses a separate AI model to check frames, so there’s a per-session cost when watchers are active. Don’t leave watchers running all day — clear them when you’re done.

​

What works well

Concrete, visually unambiguous cues:

• ✅ “a researcher’s thumbs-up gesture”
• ✅ “the centrifuge display reaching 10000 rpm”
• ✅ “the gel turning pink”

What doesn’t work as well — vague targets or things you can’t see:

• ❌ “good things” — too abstract; the model can’t ground “good”.
• ❌ “when I’m done” — depends on your intent, not a visual.
• ❌ “the temperature reaching 37°C” if the readout isn’t on camera.

​

Asking Bower what it’s watching for

Just ask — “what are you watching for?” or “what cues did I set up?”. Bower lists the active watchers and how many times each has fired.

​

What’s saved

During a live session, notes, snapshots, and video clips are only saved when you explicitly ask Bower to capture them. Visual watchers are also automatic in spirit — they fire whenever Bower sees the cue — but they live only for the duration of the live session. Nothing gets saved to your workspace from a watcher firing; Bower just speaks the response and continues. All captured notes, snapshots, and clips are searchable immediately and play back inline in the attachment viewer.