A live call co-pilot built in eight days on an open-source base

The program runs free 45-minute discovery calls with Australian small and medium businesses. In each call the operator has to confirm the business is eligible and find a pain point the owner can put a number on. Before hanging up, they also need to book a follow-up technical session. Sales people call that last part booking a meeting from a meeting.

Eligibility comes down to five things: an ABN, fewer than 200 employees, turnover under $50M, the state the business is in, and its industry.

The written spec is dated 2 May 2026 and it is ambitious. Between the spec and the roadmap you find a call agenda, a teleprompter, research on each business before the call, an ABN lookup, a live eligibility tracker, prompt cards, a score after every call, an emailed report and an admin dashboard. The project's own roadmap estimated 8 to 10 weeks for all of it. We had about a week, so the question was what to borrow.

I picked Natively because the slow, fiddly parts were already written. It had Rust audio capture through napi, using ScreenCaptureKit and CoreAudio Process Tap on macOS and WASAPI on Windows, with voice activity detection and resampling built in. It also had streaming speech-to-text adapters, a prompt library of about 2,100 lines, local document search on sqlite-vec and electron-updater. Our architecture notes put what we kept at about 70% of the infrastructure.

Some of it had to go first. Natively is pitched as an open-source alternative to Cluely, and its build hides itself from screen capture. We removed the stealth mode and the process disguise. This app is meant to be seen on the operator's screen. The paid and donation layer went too, along with GA4 analytics that phoned home, and a pile of features the program had no use for, including local Ollama models, a resume profile engine and a DRM license module. Since the import, the diff stands at 192 files changed, about 14,500 lines added and 31,100 removed.

Then we swapped out the AI plumbing. Every per-provider SDK became one OpenRouter gateway. Live coaching runs on anthropic/claude-haiku-4.5 at a temperature of 0.2 to 0.3, and the after-call summary runs on anthropic/claude-sonnet-4.6. Transcription moved to ElevenLabs scribe_v2_realtime, with one WebSocket per audio stream. Seven other speech-to-text providers can still be selected.

The decision I would defend is the fork itself, with its cost said out loud. AGPL-3.0 means this app can be used inside the program and nowhere else. If anyone wants to sell it, the coaching layer needs a clean-room rewrite without the inherited code. The README and the build strategy both say so. When the build ended, the program's written acceptance of the license was still marked as awaiting, and that is the last status we have on record.

Audio is captured on the operator's own machine. The planning notes priced a meeting bot service, Recall.ai, at $0.65 to $0.76 an hour against $0.26 an hour for Deepgram streaming. Those were estimates. We never measured real running cost, and prompt caching is still on the pending list.

The commit log shows the shape of the week: 22 commits on May 11, 17 on May 12, three on May 13, then 8, 20, 24 and 17 over May 15 to 18. Versions 0.1.0 to 0.1.2 were tagged on May 18. In all there are 114 commits on main, one author.

The worst bug threw no error. When I rebranded the dev build of the Electron app, macOS invalidated its Screen Recording permission. CoreAudio kept running and handed back audio that was all zeros, so the business owner's side of the call never appeared. Nothing crashed. Nothing logged a failure. Dev builds now use ScreenCaptureKit, and a detector flags the stream after 12 seconds of zero-filled audio. CoreAudio Process Tap is still the production default, and it still fails when AirPods are both the mic and the speaker.

The second bug lived in the Rust audio code. Silence suppression dropped quiet frames, which squeezed each stream's timeline by roughly 5 to 10 times. With the operator and the owner on separate streams, the two stopped lining up, and the operator's question could land 10 to 30 seconds after the owner's answer to it. I patched the ordering in the renderer first. That was a workaround, so I went back and fixed the frames at the source in Rust.

Then came echo. Without headphones, the operator's mic picks up the owner's voice from the speakers, and the app transcribed it a second time as the operator. The duplicate arrived 1 to 3 seconds before any deduplication could catch it. The fix is a gate: while system audio peaks above 1000, mic audio is dropped, and the gate stays shut for another 400 ms.

ElevenLabs also closes a socket after about 30 seconds of quiet. Long pauses killed transcription until I added a reconnect.

Back to the ABN. The instruction not to repeat did nothing, so the fix went into code around the model. The Director keeps a list of what it has already suggested, checks each new suggestion against 37 regex topic patterns, and retries once if the topic overlaps. A design note says operators found this far more useful. That is one anecdote.

The second model problem was memory. The inherited code gave the model a window of 60 to 180 seconds of transcript. That broke the Book move. At minute 30, the model could not see the pain the owner had put a number on at minute 10, and that number is the whole case for booking the next session. I widened the context to the full meeting. The trade-off is more tokens on every request, and because we never measured cost, I cannot tell you how much more it spends.

The window sits next to the video call. The top half is the live transcript, each line labeled by stream. Partial text shows up about 300 to 800 ms after someone starts talking, and the line settles once the speaker pauses for 0.8 seconds.

Below it, the model posts whispers of 15 words or fewer. Each carries one of seven tags: SILENCE, NUDGE, WARN, BUYING-SIGNAL, OBJECTION, FIELD-MISSING or QUALIFY. They follow a 12-stage playbook laid over the 45 minutes, with a time window for each stage. A few triggers are hard-coded in the prompt. Pass minute 35 without a booking and the operator gets a WARN. If one of the five eligibility details has not come up, a FIELD-MISSING nudge says which.

When the operator wants something on demand, there are six buttons. Guide me returns a three-step plan. Fact, Story, Warm, Specifics and Book each ask for a different kind of next move, and any suggestion can be regenerated in three styles. Consultation versions of Clarify, Recap, Follow-up Questions, Speak and Chat sit alongside, and Recap lists eligibility details as bullets. There is also a client briefing panel. Behind all of this are 12 consultation-specific prompt constants in a prompts file that is now 3,734 lines long.

After the call, Sonnet writes a meeting summary, and a Usage tab logs which buttons the operator pressed.

There is no call score. Scoring is on the roadmap and has not been started. There is no structured eligibility tracker either; eligibility shows up only as FIELD-MISSING nudges and Recap bullets. Pre-call research, ABN lookup, prompt cards, the emailed report and the admin dashboard are all unstarted, and calendar matching is switched off. Transcripts are not saved after the call yet.

Latency was never measured. The spec targets prompt cards in under 5 seconds and the roadmap wants the transcript in under 2. Upstream, using the Whisper API, took 3 to 9 seconds from speech to suggestion. I do not have a number for ours.

There are zero test files. The builds are unsigned because there was no $99 Apple developer account, so installing needs an xattr workaround. The helper we put inside the DMG to do that was itself blocked by Gatekeeper. Shared API keys are baked into the installer and anyone determined can pull them out of the asar. Credentials sit in a plaintext JSON file with 0600 permissions, to stop keychain prompts. CI covers macOS and Windows, but only macOS builds were ever shipped, and the planned Electron upgrade from 33 to 41 did not happen.

We also built a consent popup and then reverted it after a product decision. What remains is a verbal disclosure that the operator is coached to give. There is no pilot data and no outcome data.

I would pay the $99 for the Apple account on day one and skip the xattr instructions and the fight with Gatekeeper. I would also put a small proxy in front of OpenRouter and ElevenLabs so no key ships inside the installer. For a handful of trusted staff we accepted the risk. I would not hand the app to anyone outside that group as it is.

I should have written tests for the audio timeline before anything else. The out-of-order transcript would have shown up in a test that fed in two streams with known timestamps, and I would have found it before the renderer workaround.

Version 0.1.4 was tagged on May 25, after a fix to the updater. The README still says MVP and internal use only. The intended audience is about ten of the program's staff, and we have no data yet on how it holds up on real calls.