{"slug":"agent-assist","title":"Agent Assist — Real-Time Prompts and Knowledge Surfacing to Live Agents","tags":["agent-assist","real-time","live-coaching","knowledge-base","ai-assistant"],"agent_summary":"Agent assist surfaces real-time prompts, knowledge base answers, and CRM context to live human agents during an active call. Pipeline: live transcription (Deepgram/AssemblyAI streaming) → LLM analysis on partial transcript → push suggestions to agent UI via WebSocket. Latency target: under 500ms from caller utterance to suggestion display. Distinct from agent coaching/whisper (which is supervisor voice).","trigger_phrases":["agent assist","real time agent help","live call coaching","next best action voice","knowledge surfacing call","agent screen pop"],"runnable":false,"markdown":"\n# Agent Assist\n\nAgent assist is the real-time UI layer that helps live human agents during active calls. As the caller speaks, transcription runs live, an LLM analyzes the partial transcript, and the agent's screen surfaces:\n\n- Suggested responses\n- Relevant knowledge base articles\n- CRM context for the caller\n- Next-best-action prompts\n- Compliance prompts (\"remember to disclose recording\")\n- Warnings (\"this caller has opted out of marketing\")\n\nDistinct from **agent coaching** (supervisor whispers to agent — see [warm transfer](/topic/warm-transfer) for the coaching mechanism) — agent assist is software, not another human.\n\n## Architecture\n\n```\nLive call audio ─┬─→ Caller stream ┐\n │ ├─→ Live transcription (Deepgram / AssemblyAI streaming)\n └─→ Agent stream ┘\n ↓\n Streaming transcript (partial + final segments)\n ↓\n LLM analysis (Claude Haiku / Sonnet on partial)\n ↓\n Push suggestion via WebSocket to agent UI\n ↓\n Agent UI renders suggestion card\n```\n\nTotal caller-utterance-to-display latency: target under 500ms.\n\n## SignalWire integration: Tap for live audio\n\nSWML `tap` verb streams call audio to a WebSocket in real time:\n\n```yaml\nversion: 1.0.0\nsections:\n main:\n - answer: {}\n - tap:\n uri: wss://your.api/agent-assist?call_id=${call.id}&agent_id=${agent_id}\n direction: both\n codec: PCMU\n - connect:\n to: sip:agent@pbx.example.com\n```\n\nYour WebSocket endpoint receives the raw audio frames. From there:\n\n1. Forward audio to transcription provider (Deepgram streaming, AssemblyAI streaming).\n2. Receive partial + final transcript segments.\n3. Feed transcript chunks to an LLM for analysis.\n4. Push suggestions to the agent UI WebSocket.\n\n## Transcription provider — choose for latency\n\nFor agent assist, **latency is the metric that matters**. Compare:\n\n| Provider | Streaming latency (utterance to text) | Pricing |\n|---|---|---|\n| Deepgram Nova-2 streaming | 200-300ms | $0.0043/min |\n| AssemblyAI Streaming | 400-600ms | $0.0086/min (streaming tier) |\n| Google Cloud Speech v2 | 300-500ms | $0.024/min |\n| Microsoft Azure Speech | 300-400ms | $0.01/min |\n\nDeepgram is currently fastest for English; benchmark for your specific audio.\n\n## Live LLM analysis — pattern\n\nDon't run a full LLM call on every transcript chunk — too expensive, too slow. Use a sliding-window pattern:\n\n```python\nclass AgentAssist:\n def __init__(self):\n self.transcript_window = []\n self.last_analysis_at = 0\n self.MIN_INTERVAL_SEC = 2 # don't reanalyze more often than this\n self.suggestions_sent = set() # dedupe\n \n async def on_transcript_segment(self, segment):\n self.transcript_window.append(segment)\n \n # Keep only the last 60 seconds\n cutoff = time.time() - 60\n self.transcript_window = [s for s in self.transcript_window if s.ts > cutoff]\n \n if time.time() - self.last_analysis_at < self.MIN_INTERVAL_SEC:\n return\n \n self.last_analysis_at = time.time()\n await self.analyze_and_suggest()\n \n async def analyze_and_suggest(self):\n recent_text = \"\\n\".join(s.text for s in self.transcript_window)\n \n response = await claude.messages.create(\n model=\"claude-haiku-4-5\", # fast, cheap\n max_tokens=300,\n system=\"\"\"You are an agent assistant. Read the live call transcript and output ONE most-helpful suggestion if appropriate, otherwise empty JSON.\n\nOutput format:\n{\"suggestion_type\": \"knowledge|response|warning|action|none\", \"title\": \"...\", \"body\": \"...\", \"kb_article_id\": \"...\"}\"\"\",\n messages=[{\"role\": \"user\", \"content\": recent_text}]\n )\n \n suggestion = json.loads(response.content[0].text)\n if suggestion[\"suggestion_type\"] == \"none\":\n return\n \n # Dedupe — don't show the same suggestion twice in 60 sec\n sug_key = f\"{suggestion['suggestion_type']}:{suggestion['title']}\"\n if sug_key in self.suggestions_sent:\n return\n self.suggestions_sent.add(sug_key)\n \n await self.push_to_agent_ui(suggestion)\n```\n\nKey points:\n\n- Use Claude Haiku or GPT-4o-mini (fast tier) — Opus is too slow.\n- Cap analysis frequency (don't run on every 100ms chunk).\n- Use a transcript window (last 60 seconds), not full call.\n- Dedupe suggestions so the agent UI doesn't churn.\n\n## Suggestion types\n\nFive common categories:\n\n### 1. Knowledge base answers\n\nCaller asked a question. Agent assist surfaces the answer from the company KB.\n\n```json\n{\n \"suggestion_type\": \"knowledge\",\n \"title\": \"Refund Policy\",\n \"body\": \"Refunds are honored within 30 days of purchase. After 30 days, store credit only.\",\n \"kb_article_id\": \"policy-refunds-2024\"\n}\n```\n\n### 2. Suggested response\n\nCaller is stalling on a price objection. Surface a script.\n\n```json\n{\n \"suggestion_type\": \"response\",\n \"title\": \"Price objection handler\",\n \"body\": \"Acknowledge the cost concern. Offer the 12-month payment plan: 'I hear that. What if we broke it into 12 monthly payments of $X?'\"\n}\n```\n\n### 3. Warning / compliance\n\nCaller mentioned a competitor — remind the agent not to disparage. Caller asked about medical advice — surface the disclaimer.\n\n```json\n{\n \"suggestion_type\": \"warning\",\n \"title\": \"Compliance Reminder\",\n \"body\": \"Caller is in California. Confirm recording consent verbally before continuing.\",\n \"severity\": \"high\"\n}\n```\n\n### 4. Next-best-action\n\nBased on context, suggest the next concrete step.\n\n```json\n{\n \"suggestion_type\": \"action\",\n \"title\": \"Schedule Appointment\",\n \"body\": \"Caller mentioned urgency. Offer same-day service. Tap to open booking widget.\",\n \"deep_link\": \"/booking?customer_id=12345&service=plumbing&priority=urgent\"\n}\n```\n\n### 5. CRM context surface\n\nPull caller history and surface relevant facts.\n\n```json\n{\n \"suggestion_type\": \"context\",\n \"title\": \"Returning customer\",\n \"body\": \"Bob Smith, customer since 2022. Last service: 2025-12-04. Open ticket: #4521. CLTV: $4,200.\",\n \"deep_link\": \"/crm/contact/12345\"\n}\n```\n\n## Knowledge base retrieval\n\nSurfacing the right KB article is a retrieval problem. Two approaches:\n\n### Approach A: Pre-embedded KB, semantic search on transcript\n\nPre-embed the KB articles. On each transcript window, embed the recent text and search:\n\n```python\n# Pre-deploy: embed all KB articles\nfor article in kb_articles:\n embedding = openai.embeddings.create(input=article.text, model=\"text-embedding-3-small\")\n db.upsert(\"kb_embeddings\", article.id, embedding=embedding.data[0].embedding)\n\n# Runtime: embed transcript window, search\nquery_embedding = openai.embeddings.create(input=recent_text).data[0].embedding\nmatches = db.execute(\"\"\"\n SELECT id, title, body, 1 - (embedding <=> $1) AS similarity\n FROM kb_embeddings\n ORDER BY embedding <=> $1\n LIMIT 5\n\"\"\", query_embedding)\n```\n\nMatch similarity > 0.8 = surface the article.\n\n### Approach B: LLM as retrieval router\n\nLet the LLM identify the KB topic and look up directly:\n\n```python\nresponse = await claude.messages.create(\n model=\"claude-haiku-4-5\",\n system=\"If the caller is asking about a topic covered by our KB, return the topic slug. Otherwise return 'none'. Topics: refund-policy, shipping-times, account-setup, password-reset, ...\",\n messages=[{\"role\": \"user\", \"content\": recent_text}]\n)\ntopic = response.content[0].text\nif topic != \"none\":\n article = db.get_kb_article(topic)\n```\n\nApproach B is cheaper and more accurate for closed-domain KBs. Approach A scales to open-ended KBs.\n\n## Agent UI rendering\n\nThe agent UI receives suggestions over WebSocket. Render as cards:\n\n```jsx\nfunction AgentAssistPanel({ callId }) {\n const [suggestions, setSuggestions] = useState([]);\n \n useEffect(() => {\n const ws = new WebSocket(`wss://api.example.com/agent-assist?call=${callId}`);\n ws.onmessage = (e) => {\n const sug = JSON.parse(e.data);\n setSuggestions(prev => [...prev.slice(-4), sug]);\n };\n return () => ws.close();\n }, [callId]);\n \n return (\n
\n {suggestions.map(sug => (\n \n ))}\n
\n );\n}\n```\n\nEach card has:\n- Color-coded badge (knowledge / response / warning / action / context)\n- Title and body\n- \"Acknowledge\" or \"Dismiss\" buttons\n- Tap-to-copy or tap-to-open behaviors\n\n## Privacy and compliance\n\nReal-time transcription handles sensitive data live. Considerations:\n\n- **PII redaction** — transcription providers can redact phone numbers, SSNs, credit card numbers before delivering text to your system.\n- **PHI** — if covered entity, vendors must have BAA. AssemblyAI and Deepgram offer BAA on enterprise tier.\n- **Recording retention** — agent assist may not need persistent transcripts. Discard the streaming buffer after the call ends.\n- **EU GDPR Article 22** — automated decision-making about agents (performance scoring via assist) requires transparency. Inform agents what's being analyzed.\n\n## Cost model\n\nFor an 8-hour shift with 50 calls averaging 5 minutes:\n\n| Component | Per minute | Daily per agent |\n|---|---|---|\n| Streaming transcription (Deepgram) | $0.0043 | $1.08 (250 minutes) |\n| LLM analysis (Claude Haiku, ~10 analyses per call) | $0.001 × 10 = $0.01 per call | $0.50 |\n| Total | | $1.58 per agent per day |\n\nFor 50 agents: $79/day, $2,400/month for an always-on agent assist platform.\n\n## ROI signals\n\nAgent assist platforms typically deliver:\n\n- 15-25% reduction in average handle time (faster resolution via KB surfacing)\n- 30-50% improvement in first-call resolution (right answer first time)\n- 10-15% improvement in CSAT scores\n- 50%+ reduction in new-agent ramp time (KB surfacing scaffolds knowledge)\n\nThe hard part is integrating the KB and tuning the suggestion frequency. Too many suggestions = agent ignores them. Too few = no value.\n\n## Common pitfalls\n\n- **Latency too high** — agent ignores suggestions that arrive 5 seconds late. Optimize the transcription + LLM chain ruthlessly.\n- **Over-suggestion fatigue** — agent UI shows 20 cards per call, all ignored. Throttle to max 1 suggestion per 30 seconds; rank by relevance.\n- **Generic LLM hallucinations in knowledge surface** — Claude makes up policy details. Always cite the KB article and let the agent verify before reciting.\n- **Privacy creep** — recording every agent's screen activity for \"training\" goes too far. Be explicit about what's logged.\n- **No measurement** — without A/B test of assist on/off, can't prove ROI. Build the experimental framework into the product.\n\n## Related patterns\n\n- [Sentiment analysis pipeline](/topic/sentiment-analysis-pipeline) — post-call equivalent\n- [AssemblyAI transcription](/topic/assemblyai-transcription)\n- [SignalWire Call Intelligence](/topic/signalwire-call-intelligence)\n- [Warm transfer](/topic/warm-transfer) — coaching is the human version of agent assist\n\n## References\n\n- Deepgram streaming API documentation\n- AssemblyAI Real-Time Transcription API\n- SignalWire SWML `tap` verb — real-time audio bridge\n- Anthropic Claude — tool use and streaming\n","html":"

Agent Assist

\n

Agent assist is the real-time UI layer that helps live human agents during active calls. As the caller speaks, transcription runs live, an LLM analyzes the partial transcript, and the agent's screen surfaces:

\n\n

Distinct from agent coaching (supervisor whispers to agent — see warm transfer for the coaching mechanism) — agent assist is software, not another human.

\n

Architecture

\n
Live call audio ─┬─→ Caller stream  ┐\n                 │                     ├─→ Live transcription (Deepgram / AssemblyAI streaming)\n                 └─→ Agent stream  ┘\n                                                 ↓\n                                  Streaming transcript (partial + final segments)\n                                                 ↓\n                              LLM analysis (Claude Haiku / Sonnet on partial)\n                                                 ↓\n                              Push suggestion via WebSocket to agent UI\n                                                 ↓\n                              Agent UI renders suggestion card\n
\n

Total caller-utterance-to-display latency: target under 500ms.

\n

SignalWire integration: Tap for live audio

\n

SWML tap verb streams call audio to a WebSocket in real time:

\n
version: 1.0.0\nsections:\n  main:\n    - answer: {}\n    - tap:\n        uri: wss://your.api/agent-assist?call_id=${call.id}&agent_id=${agent_id}\n        direction: both\n        codec: PCMU\n    - connect:\n        to: sip:agent@pbx.example.com\n
\n

Your WebSocket endpoint receives the raw audio frames. From there:

\n
    \n
  1. Forward audio to transcription provider (Deepgram streaming, AssemblyAI streaming).
    2. \n
  2. Receive partial + final transcript segments.
    3. \n
  3. Feed transcript chunks to an LLM for analysis.
    4. \n
  4. Push suggestions to the agent UI WebSocket.
    5. \n
\n

Transcription provider — choose for latency

\n

For agent assist, latency is the metric that matters. Compare:

\n

| Provider | Streaming latency (utterance to text) | Pricing |\n|---|---|---|\n| Deepgram Nova-2 streaming | 200-300ms | $0.0043/min |\n| AssemblyAI Streaming | 400-600ms | $0.0086/min (streaming tier) |\n| Google Cloud Speech v2 | 300-500ms | $0.024/min |\n| Microsoft Azure Speech | 300-400ms | $0.01/min |

\n

Deepgram is currently fastest for English; benchmark for your specific audio.

\n

Live LLM analysis — pattern

\n

Don't run a full LLM call on every transcript chunk — too expensive, too slow. Use a sliding-window pattern:

\n
class AgentAssist:\n    def __init__(self):\n        self.transcript_window = []\n        self.last_analysis_at = 0\n        self.MIN_INTERVAL_SEC = 2  # don't reanalyze more often than this\n        self.suggestions_sent = set()  # dedupe\n    \n    async def on_transcript_segment(self, segment):\n        self.transcript_window.append(segment)\n        \n        # Keep only the last 60 seconds\n        cutoff = time.time() - 60\n        self.transcript_window = [s for s in self.transcript_window if s.ts > cutoff]\n        \n        if time.time() - self.last_analysis_at < self.MIN_INTERVAL_SEC:\n            return\n        \n        self.last_analysis_at = time.time()\n        await self.analyze_and_suggest()\n    \n    async def analyze_and_suggest(self):\n        recent_text = \"\\n\".join(s.text for s in self.transcript_window)\n        \n        response = await claude.messages.create(\n            model=\"claude-haiku-4-5\",  # fast, cheap\n            max_tokens=300,\n            system=\"\"\"You are an agent assistant. Read the live call transcript and output ONE most-helpful suggestion if appropriate, otherwise empty JSON.\n\nOutput format:\n{\"suggestion_type\": \"knowledge|response|warning|action|none\", \"title\": \"...\", \"body\": \"...\", \"kb_article_id\": \"...\"}\"\"\",\n            messages=[{\"role\": \"user\", \"content\": recent_text}]\n        )\n        \n        suggestion = json.loads(response.content[0].text)\n        if suggestion[\"suggestion_type\"] == \"none\":\n            return\n        \n        # Dedupe — don't show the same suggestion twice in 60 sec\n        sug_key = f\"{suggestion['suggestion_type']}:{suggestion['title']}\"\n        if sug_key in self.suggestions_sent:\n            return\n        self.suggestions_sent.add(sug_key)\n        \n        await self.push_to_agent_ui(suggestion)\n
\n

Key points:

\n\n

Suggestion types

\n

Five common categories:

\n

1. Knowledge base answers

\n

Caller asked a question. Agent assist surfaces the answer from the company KB.

\n
{\n  \"suggestion_type\": \"knowledge\",\n  \"title\": \"Refund Policy\",\n  \"body\": \"Refunds are honored within 30 days of purchase. After 30 days, store credit only.\",\n  \"kb_article_id\": \"policy-refunds-2024\"\n}\n
\n

2. Suggested response

\n

Caller is stalling on a price objection. Surface a script.

\n
{\n  \"suggestion_type\": \"response\",\n  \"title\": \"Price objection handler\",\n  \"body\": \"Acknowledge the cost concern. Offer the 12-month payment plan: 'I hear that. What if we broke it into 12 monthly payments of $X?'\"\n}\n
\n

3. Warning / compliance

\n

Caller mentioned a competitor — remind the agent not to disparage. Caller asked about medical advice — surface the disclaimer.

\n
{\n  \"suggestion_type\": \"warning\",\n  \"title\": \"Compliance Reminder\",\n  \"body\": \"Caller is in California. Confirm recording consent verbally before continuing.\",\n  \"severity\": \"high\"\n}\n
\n

4. Next-best-action

\n

Based on context, suggest the next concrete step.

\n
{\n  \"suggestion_type\": \"action\",\n  \"title\": \"Schedule Appointment\",\n  \"body\": \"Caller mentioned urgency. Offer same-day service. Tap to open booking widget.\",\n  \"deep_link\": \"/booking?customer_id=12345&service=plumbing&priority=urgent\"\n}\n
\n

5. CRM context surface

\n

Pull caller history and surface relevant facts.

\n
{\n  \"suggestion_type\": \"context\",\n  \"title\": \"Returning customer\",\n  \"body\": \"Bob Smith, customer since 2022. Last service: 2025-12-04. Open ticket: #4521. CLTV: $4,200.\",\n  \"deep_link\": \"/crm/contact/12345\"\n}\n
\n

Knowledge base retrieval

\n

Surfacing the right KB article is a retrieval problem. Two approaches:

\n

Approach A: Pre-embedded KB, semantic search on transcript

\n

Pre-embed the KB articles. On each transcript window, embed the recent text and search:

\n
# Pre-deploy: embed all KB articles\nfor article in kb_articles:\n    embedding = openai.embeddings.create(input=article.text, model=\"text-embedding-3-small\")\n    db.upsert(\"kb_embeddings\", article.id, embedding=embedding.data[0].embedding)\n\n# Runtime: embed transcript window, search\nquery_embedding = openai.embeddings.create(input=recent_text).data[0].embedding\nmatches = db.execute(\"\"\"\n    SELECT id, title, body, 1 - (embedding <=> $1) AS similarity\n    FROM kb_embeddings\n    ORDER BY embedding <=> $1\n    LIMIT 5\n\"\"\", query_embedding)\n
\n

Match similarity > 0.8 = surface the article.

\n

Approach B: LLM as retrieval router

\n

Let the LLM identify the KB topic and look up directly:

\n
response = await claude.messages.create(\n    model=\"claude-haiku-4-5\",\n    system=\"If the caller is asking about a topic covered by our KB, return the topic slug. Otherwise return 'none'. Topics: refund-policy, shipping-times, account-setup, password-reset, ...\",\n    messages=[{\"role\": \"user\", \"content\": recent_text}]\n)\ntopic = response.content[0].text\nif topic != \"none\":\n    article = db.get_kb_article(topic)\n
\n

Approach B is cheaper and more accurate for closed-domain KBs. Approach A scales to open-ended KBs.

\n

Agent UI rendering

\n

The agent UI receives suggestions over WebSocket. Render as cards:

\n
function AgentAssistPanel({ callId }) {\n    const [suggestions, setSuggestions] = useState([]);\n    \n    useEffect(() => {\n        const ws = new WebSocket(`wss://api.example.com/agent-assist?call=${callId}`);\n        ws.onmessage = (e) => {\n            const sug = JSON.parse(e.data);\n            setSuggestions(prev => [...prev.slice(-4), sug]);\n        };\n        return () => ws.close();\n    }, [callId]);\n    \n    return (\n        <div className=\"suggestions-stack\">\n            {suggestions.map(sug => (\n                <SuggestionCard key={sug.id} suggestion={sug} />\n            ))}\n        </div>\n    );\n}\n
\n

Each card has:

\n\n

Privacy and compliance

\n

Real-time transcription handles sensitive data live. Considerations:

\n\n

Cost model

\n

For an 8-hour shift with 50 calls averaging 5 minutes:

\n

| Component | Per minute | Daily per agent |\n|---|---|---|\n| Streaming transcription (Deepgram) | $0.0043 | $1.08 (250 minutes) |\n| LLM analysis (Claude Haiku, ~10 analyses per call) | $0.001 × 10 = $0.01 per call | $0.50 |\n| Total | | $1.58 per agent per day |

\n

For 50 agents: $79/day, $2,400/month for an always-on agent assist platform.

\n

ROI signals

\n

Agent assist platforms typically deliver:

\n\n

The hard part is integrating the KB and tuning the suggestion frequency. Too many suggestions = agent ignores them. Too few = no value.

\n

Common pitfalls

\n\n

Related patterns

\n\n

References

\n\n"}