Agent Play: When Push-to-Talk Finally Felt Natural

The original north star did not change: walk up to an agent, press P, and talk like it belongs there.

A while ago, we had the architecture.

We knew the user moment we wanted: move close to an agent, press P, speak, and hear a response that feels immediate and alive in the world.

Thanks for reading Isaac's Pub! Subscribe for free to receive new posts and support my work.

But as with most realtime features, the hard part was never the idea. It was everything around it: session control, short-lived credentials, connection state, retries, and the usual WebRTC edge cases that quietly consume entire days.

The phase where things were “working,” but heavy

Early on, we proved the flow in pieces.
Voice reached the system. Responses came back. The concept held.

But every step carried operational weight:

• credential timing had to be perfect,
• session setup paths duplicated under retry conditions,
• UI state and transport state could drift,
• and each failure mode needed defensive handling.

It was progress, but it was expensive progress.

The turning point: OpenAI Agents realtime primitives

Then we leaned into the OpenAI Agents realtime model more directly.

That gave us something we were missing: a simple, explicit interface for agent + session lifecycle.

Instead of wrestling the lower layers every time, we could center the flow around:

• a clear agent definition (identity + instructions),
• a clear session lifecycle (connect, active, close),
• and a tighter boundary between orchestration logic and transport concerns.

That shift has effectively 5x’d our implementation speed.

Not because complexity disappeared from the domain, but because the interface stopped forcing us to think about everything at once. We could ship behavior faster, test changes faster, and iterate without reopening the full WebRTC problem space each time.

What “less WebRTC stress” actually means

This is not a claim that realtime systems become trivial.
It means the most painful parts got compressed behind better abstractions.

Practically, that looked like:

• fewer bespoke connection pathways,
• cleaner retry behavior for ephemeral token expiry,
• clearer state transitions in the UI (connecting, connected, failed, disconnected),
• and one place to reason about session setup instead of many.

In other words: less firefighting, more product progress.

Why this matters for PTT specifically

Push-to-Talk is deceptively simple.

To users, it should feel like one action.
To engineers, it is a choreography across identity, event routing, ephemeral auth, realtime transport, and turn management.

The OpenAI Agents interface helped us keep that choreography disciplined:

• request credentials just-in-time,
• establish session predictably,
• enforce voice instructions (including selected language),
• and recover gracefully when the network or token lifetime disagrees.

That reliability is what makes PTT feel natural, not fragile.

The big next.

This milestone is not “we added voice.”

It is that PTT moved from a technically possible feature to a repeatable product capability we can scale:

• private player-to-agent conversations now,
• richer multi-party conversational spaces next,
• all built on the same core patterns of session orchestration, event correlation, and controlled realtime state.

The original north star did not change:
walk up to an agent, press P, and talk like it belongs there.

What changed is how quickly we can now build toward that future.

Thanks for reading Isaac's Pub! Subscribe for free to receive new posts and support my work.