Custom LLM

Integrating a Custom LLM with Millis AI Voice Agent

Basic

This guide describes how to integrate your own LLM chatbot with a Millis AI voice agent. By connecting your custom LLM, you can power the voice agent with your chatbot’s capabilities, providing a seamless voice interaction experience based on your model’s responses.

Prerequisites

• Create your Voice Agent on the Playground
• Setup a websocket server on your end.

Set Up Your WebSocket Endpoint:

• When an outbound or inbound call is initiated with your voice agent, the Millis AI server will establish a connection to your specified WebSocket URL.
• Your endpoint should be capable of both receiving messages from and sending messages to the Millis AI server. Here’s a Sample Code.
• Here’s how the interaction flows after connection established:

1. Initiate a call:

Millis AI server will send start_call event to tell your server when the conversation starts.

{
  "type": "start_call",
  "data": {
    "stream_id": <millis_stream_id>,
    "agent_id": <millis_agent_id>,
    "call_sid": <twilio_call_sid>, // optional, None if conversation is not via Twilio phone call
    "session_id": <millis_session_id> // unique id of the call session
  }
}

2. Listen to user’s message:

Millis AI streams the user’s spoken message, including the full conversation transcript, to your LLM.

{
  "type": "stream_request",
  "data": {
    "stream_id": <stream_id>,
    "transcript": [<chat_history>]
  }
}

3. Generate LLM Responses:

Your LLM processes the transcript and streams back the response. Indicate the end of a message stream with end_of_stream.

{
  "type": "stream_response",
  "data": {
    "stream_id": request['stream_id'],
    "content": "text",
    "end_of_stream": False / True
  }
}

• When your LLM generates a response, attach the stream_id from the original request so that we can keep track of which response corresponds to which request.
• For the first message that your server sends after receiving the start_call event, use the stream_id from the start_call event.

4. Handle advanced interaction:

Millis AI manages the conversation flow, including interruption detection and end-of-turn signals. You will be notified of these events:

{
  "type": "interrupt",
  "data": {
    "stream_id": <stream_id>
  }
}

Connect your Voice Agent to your Custom LLM:

In your voice agent’s configuration on the Millis AI platform, specify your WebSocket endpoint.