Agent Assist

Agent assist is the real-time UI layer that helps live human agents during active calls. As the caller speaks, transcription runs live, an LLM analyzes the partial transcript, and the agent's screen surfaces:

• Suggested responses
• Relevant knowledge base articles
• CRM context for the caller
• Next-best-action prompts
• Compliance prompts ("remember to disclose recording")
• Warnings ("this caller has opted out of marketing")

Distinct from agent coaching (supervisor whispers to agent — see warm transfer for the coaching mechanism) — agent assist is software, not another human.

Architecture

Live call audio ─┬─→ Caller stream  ┐
                 │                     ├─→ Live transcription (Deepgram / AssemblyAI streaming)
                 └─→ Agent stream  ┘
                                                 ↓
                                  Streaming transcript (partial + final segments)
                                                 ↓
                              LLM analysis (Claude Haiku / Sonnet on partial)
                                                 ↓
                              Push suggestion via WebSocket to agent UI
                                                 ↓
                              Agent UI renders suggestion card

Total caller-utterance-to-display latency: target under 500ms.

SignalWire integration: Tap for live audio

SWML tap verb streams call audio to a WebSocket in real time:

version: 1.0.0
sections:
  main:
    - answer: {}
    - tap:
        uri: wss://your.api/agent-assist?call_id=${call.id}&agent_id=${agent_id}
        direction: both
        codec: PCMU
    - connect:
        to: sip:agent@pbx.example.com

Your WebSocket endpoint receives the raw audio frames. From there:

1. Forward audio to transcription provider (Deepgram streaming, AssemblyAI streaming).
2. Receive partial + final transcript segments.
3. Feed transcript chunks to an LLM for analysis.
4. Push suggestions to the agent UI WebSocket.

Transcription provider — choose for latency

For agent assist, latency is the metric that matters. Compare:

| Provider | Streaming latency (utterance to text) | Pricing | |---|---|---| | Deepgram Nova-2 streaming | 200-300ms | $0.0043/min | | AssemblyAI Streaming | 400-600ms | $0.0086/min (streaming tier) | | Google Cloud Speech v2 | 300-500ms | $0.024/min | | Microsoft Azure Speech | 300-400ms | $0.01/min |

Deepgram is currently fastest for English; benchmark for your specific audio.

Live LLM analysis — pattern

Don't run a full LLM call on every transcript chunk — too expensive, too slow. Use a sliding-window pattern:

class AgentAssist:
    def __init__(self):
        self.transcript_window = []
        self.last_analysis_at = 0
        self.MIN_INTERVAL_SEC = 2  # don't reanalyze more often than this
        self.suggestions_sent = set()  # dedupe
    
    async def on_transcript_segment(self, segment):
        self.transcript_window.append(segment)
        
        # Keep only the last 60 seconds
        cutoff = time.time() - 60
        self.transcript_window = [s for s in self.transcript_window if s.ts > cutoff]
        
        if time.time() - self.last_analysis_at < self.MIN_INTERVAL_SEC:
            return
        
        self.last_analysis_at = time.time()
        await self.analyze_and_suggest()
    
    async def analyze_and_suggest(self):
        recent_text = "\n".join(s.text for s in self.transcript_window)
        
        response = await claude.messages.create(
            model="claude-haiku-4-5",  # fast, cheap
            max_tokens=300,
            system="""You are an agent assistant. Read the live call transcript and output ONE most-helpful suggestion if appropriate, otherwise empty JSON.

Output format:
{"suggestion_type": "knowledge|response|warning|action|none", "title": "...", "body": "...", "kb_article_id": "..."}""",
            messages=[{"role": "user", "content": recent_text}]
        )
        
        suggestion = json.loads(response.content[0].text)
        if suggestion["suggestion_type"] == "none":
            return
        
        # Dedupe — don't show the same suggestion twice in 60 sec
        sug_key = f"{suggestion['suggestion_type']}:{suggestion['title']}"
        if sug_key in self.suggestions_sent:
            return
        self.suggestions_sent.add(sug_key)
        
        await self.push_to_agent_ui(suggestion)

Key points:

• Use Claude Haiku or GPT-4o-mini (fast tier) — Opus is too slow.
• Cap analysis frequency (don't run on every 100ms chunk).
• Use a transcript window (last 60 seconds), not full call.
• Dedupe suggestions so the agent UI doesn't churn.

Suggestion types

Five common categories:

1. Knowledge base answers

Caller asked a question. Agent assist surfaces the answer from the company KB.

{
  "suggestion_type": "knowledge",
  "title": "Refund Policy",
  "body": "Refunds are honored within 30 days of purchase. After 30 days, store credit only.",
  "kb_article_id": "policy-refunds-2024"
}

2. Suggested response

Caller is stalling on a price objection. Surface a script.

{
  "suggestion_type": "response",
  "title": "Price objection handler",
  "body": "Acknowledge the cost concern. Offer the 12-month payment plan: 'I hear that. What if we broke it into 12 monthly payments of $X?'"
}

3. Warning / compliance

Caller mentioned a competitor — remind the agent not to disparage. Caller asked about medical advice — surface the disclaimer.

{
  "suggestion_type": "warning",
  "title": "Compliance Reminder",
  "body": "Caller is in California. Confirm recording consent verbally before continuing.",
  "severity": "high"
}

4. Next-best-action

Based on context, suggest the next concrete step.

{
  "suggestion_type": "action",
  "title": "Schedule Appointment",
  "body": "Caller mentioned urgency. Offer same-day service. Tap to open booking widget.",
  "deep_link": "/booking?customer_id=12345&service=plumbing&priority=urgent"
}

5. CRM context surface

Pull caller history and surface relevant facts.

{
  "suggestion_type": "context",
  "title": "Returning customer",
  "body": "Bob Smith, customer since 2022. Last service: 2025-12-04. Open ticket: #4521. CLTV: $4,200.",
  "deep_link": "/crm/contact/12345"
}

Knowledge base retrieval

Surfacing the right KB article is a retrieval problem. Two approaches:

Approach A: Pre-embedded KB, semantic search on transcript

Pre-embed the KB articles. On each transcript window, embed the recent text and search:

# Pre-deploy: embed all KB articles
for article in kb_articles:
    embedding = openai.embeddings.create(input=article.text, model="text-embedding-3-small")
    db.upsert("kb_embeddings", article.id, embedding=embedding.data[0].embedding)

# Runtime: embed transcript window, search
query_embedding = openai.embeddings.create(input=recent_text).data[0].embedding
matches = db.execute("""
    SELECT id, title, body, 1 - (embedding <=> $1) AS similarity
    FROM kb_embeddings
    ORDER BY embedding <=> $1
    LIMIT 5
""", query_embedding)

Match similarity > 0.8 = surface the article.

Approach B: LLM as retrieval router

Let the LLM identify the KB topic and look up directly:

response = await claude.messages.create(
    model="claude-haiku-4-5",
    system="If the caller is asking about a topic covered by our KB, return the topic slug. Otherwise return 'none'. Topics: refund-policy, shipping-times, account-setup, password-reset, ...",
    messages=[{"role": "user", "content": recent_text}]
)
topic = response.content[0].text
if topic != "none":
    article = db.get_kb_article(topic)

Approach B is cheaper and more accurate for closed-domain KBs. Approach A scales to open-ended KBs.

Agent UI rendering

The agent UI receives suggestions over WebSocket. Render as cards:

function AgentAssistPanel({ callId }) {
    const [suggestions, setSuggestions] = useState([]);
    
    useEffect(() => {
        const ws = new WebSocket(`wss://api.example.com/agent-assist?call=${callId}`);
        ws.onmessage = (e) => {
            const sug = JSON.parse(e.data);
            setSuggestions(prev => [...prev.slice(-4), sug]);
        };
        return () => ws.close();
    }, [callId]);
    
    return (
        <div className="suggestions-stack">
            {suggestions.map(sug => (
                <SuggestionCard key={sug.id} suggestion={sug} />
            ))}
        </div>
    );
}

Each card has:

• Color-coded badge (knowledge / response / warning / action / context)
• Title and body
• "Acknowledge" or "Dismiss" buttons
• Tap-to-copy or tap-to-open behaviors

Privacy and compliance

Real-time transcription handles sensitive data live. Considerations:

• PII redaction — transcription providers can redact phone numbers, SSNs, credit card numbers before delivering text to your system.
• PHI — if covered entity, vendors must have BAA. AssemblyAI and Deepgram offer BAA on enterprise tier.
• Recording retention — agent assist may not need persistent transcripts. Discard the streaming buffer after the call ends.
• EU GDPR Article 22 — automated decision-making about agents (performance scoring via assist) requires transparency. Inform agents what's being analyzed.

Cost model

For an 8-hour shift with 50 calls averaging 5 minutes:

| Component | Per minute | Daily per agent | |---|---|---| | Streaming transcription (Deepgram) | $0.0043 | $1.08 (250 minutes) | | LLM analysis (Claude Haiku, ~10 analyses per call) | $0.001 × 10 = $0.01 per call | $0.50 | | Total | | $1.58 per agent per day |

For 50 agents: $79/day, $2,400/month for an always-on agent assist platform.

ROI signals

Agent assist platforms typically deliver:

• 15-25% reduction in average handle time (faster resolution via KB surfacing)
• 30-50% improvement in first-call resolution (right answer first time)
• 10-15% improvement in CSAT scores
• 50%+ reduction in new-agent ramp time (KB surfacing scaffolds knowledge)

The hard part is integrating the KB and tuning the suggestion frequency. Too many suggestions = agent ignores them. Too few = no value.

Common pitfalls

• Latency too high — agent ignores suggestions that arrive 5 seconds late. Optimize the transcription + LLM chain ruthlessly.
• Over-suggestion fatigue — agent UI shows 20 cards per call, all ignored. Throttle to max 1 suggestion per 30 seconds; rank by relevance.
• Generic LLM hallucinations in knowledge surface — Claude makes up policy details. Always cite the KB article and let the agent verify before reciting.
• Privacy creep — recording every agent's screen activity for "training" goes too far. Be explicit about what's logged.
• No measurement — without A/B test of assist on/off, can't prove ROI. Build the experimental framework into the product.

Related patterns

• Sentiment analysis pipeline — post-call equivalent
• AssemblyAI transcription
• SignalWire Call Intelligence
• Warm transfer — coaching is the human version of agent assist

References

• Deepgram streaming API documentation
• AssemblyAI Real-Time Transcription API
• SignalWire SWML tap verb — real-time audio bridge
• Anthropic Claude — tool use and streaming