# Create Agent Source: https://docs.millis.ai/api-reference/agents/create-agent https://api-west.millis.ai/openapi.json post /agents # Delete Agent Source: https://docs.millis.ai/api-reference/agents/delete-agent https://api-west.millis.ai/openapi.json delete /agents/{agent_id} # Duplicate Agent Source: https://docs.millis.ai/api-reference/agents/duplicate-agent https://api-west.millis.ai/openapi.json post /agents/{agent_id}/duplicate # Get Agent Source: https://docs.millis.ai/api-reference/agents/get-agent https://api-west.millis.ai/openapi.json get /agents/{agent_id} # Get Agents Source: https://docs.millis.ai/api-reference/agents/get-agents https://api-west.millis.ai/openapi.json get /agents # Get Call Histories Source: https://docs.millis.ai/api-reference/agents/get-call-histories https://api-west.millis.ai/openapi.json get /agents/{agent_id}/call-histories # Set Agent Status Source: https://docs.millis.ai/api-reference/agents/set-agent-status https://api-west.millis.ai/openapi.json post /agents/{agent_id}/status # Set Embed Config Source: https://docs.millis.ai/api-reference/agents/set-embed-config https://api-west.millis.ai/openapi.json post /agents/{agent_id}/embed # Update Agent Source: https://docs.millis.ai/api-reference/agents/update-agent https://api-west.millis.ai/openapi.json put /agents/{agent_id} # Delete Call Recording Source: https://docs.millis.ai/api-reference/call-logs/delete-call-recording https://api-west.millis.ai/openapi.json delete /call-logs/{session_id}/recording # Get Call History Source: https://docs.millis.ai/api-reference/call-logs/get-call-history https://api-west.millis.ai/openapi.json get /call-logs/{session_id} # Get Call Log Metrics Source: https://docs.millis.ai/api-reference/call-logs/get-call-log-metrics https://api-west.millis.ai/openapi.json get /call-logs/metrics # List Call Histories Source: https://docs.millis.ai/api-reference/call-logs/list-call-histories https://api-west.millis.ai/openapi.json get /call-logs # Register Call Source: https://docs.millis.ai/api-reference/call-management/register-call https://api-west.millis.ai/openapi.json post /register_call # Register Sip Call Source: https://docs.millis.ai/api-reference/call-management/register-sip-call https://api-west.millis.ai/openapi.json post /register_sip_call # Start Outbound Call Source: https://docs.millis.ai/api-reference/call-management/start-outbound-call https://api-west.millis.ai/openapi.json post /start_outbound_call # Terminate Call Session Source: https://docs.millis.ai/api-reference/call-management/terminate-call-session https://api-west.millis.ai/openapi.json post /sessions/{session_id}/terminate # Create Campaign Source: https://docs.millis.ai/api-reference/campaigns/create-campaign https://api-west.millis.ai/openapi.json post /campaigns # Delete Campaign Source: https://docs.millis.ai/api-reference/campaigns/delete-campaign https://api-west.millis.ai/openapi.json delete /campaigns/{campaign_id} # Delete Campaign Record Source: https://docs.millis.ai/api-reference/campaigns/delete-campaign-record https://api-west.millis.ai/openapi.json delete /campaigns/{campaign_id}/records/{phone} # Edit Campaign Info Source: https://docs.millis.ai/api-reference/campaigns/edit-campaign-info https://api-west.millis.ai/openapi.json put /campaigns/{campaign_id}/info # Export Campaign Sessions Source: https://docs.millis.ai/api-reference/campaigns/export-campaign-sessions https://api-west.millis.ai/openapi.json get /campaigns/{campaign_id}/export Export all sessions linked to a specific campaign as CSV. # Get All Campaigns Source: https://docs.millis.ai/api-reference/campaigns/get-all-campaigns https://api-west.millis.ai/openapi.json get /campaigns # Get Campaign Source: https://docs.millis.ai/api-reference/campaigns/get-campaign https://api-west.millis.ai/openapi.json get /campaigns/{campaign_id} # Get Campaign Info Source: https://docs.millis.ai/api-reference/campaigns/get-campaign-info https://api-west.millis.ai/openapi.json get /campaigns/{campaign_id}/info # Set Caller Source: https://docs.millis.ai/api-reference/campaigns/set-caller https://api-west.millis.ai/openapi.json post /campaigns/{campaign_id}/set_caller # Start Campaign Source: https://docs.millis.ai/api-reference/campaigns/start-campaign https://api-west.millis.ai/openapi.json post /campaigns/{campaign_id}/start # Stop Campaign Source: https://docs.millis.ai/api-reference/campaigns/stop-campaign https://api-west.millis.ai/openapi.json post /campaigns/{campaign_id}/stop # Upload Records Source: https://docs.millis.ai/api-reference/campaigns/upload-records https://api-west.millis.ai/openapi.json post /campaigns/{campaign_id}/records # Create a WebRTC offer Source: https://docs.millis.ai/api-reference/connectivity/create-a-webrtc-offer https://api-west.millis.ai/openapi.json post /webrtc/offer Create a WebRTC offer for a call. # Create Sip Source: https://docs.millis.ai/api-reference/connectivity/create-sip https://api-west.millis.ai/openapi.json post /sip # Delete Sip Source: https://docs.millis.ai/api-reference/connectivity/delete-sip https://api-west.millis.ai/openapi.json delete /sip/{call_id} # Chat Completions Source: https://docs.millis.ai/api-reference/conversational-ai/chat-completions https://api-west.millis.ai/openapi.json post /chat/completions # Create Agent Source: https://docs.millis.ai/api-reference/endpoint/agents/create-agent post /agents # Delete Agent Source: https://docs.millis.ai/api-reference/endpoint/agents/delete-agent delete /agents/{agent_id} # Duplicate Agent Source: https://docs.millis.ai/api-reference/endpoint/agents/duplicate-agent post /agents/{agent_id}/duplicate # Get Agent Source: https://docs.millis.ai/api-reference/endpoint/agents/get-agent get /agents/{agent_id} # Get Agents Source: https://docs.millis.ai/api-reference/endpoint/agents/get-agents get /agents # Get Call Histories Source: https://docs.millis.ai/api-reference/endpoint/agents/get-call-histories get /agents/{agent_id}/call-histories # Set Embed Config Source: https://docs.millis.ai/api-reference/endpoint/agents/set-embed-config post /agents/{agent_id}/embed # Update Agent Source: https://docs.millis.ai/api-reference/endpoint/agents/update-agent put /agents/{agent_id} # Delete Call Recording Source: https://docs.millis.ai/api-reference/endpoint/call-logs/delete-call-recording delete /call-logs/{session_id}/recording # Get Call History Source: https://docs.millis.ai/api-reference/endpoint/call-logs/get-call-history get /call-logs/{session_id} # List Call Histories Source: https://docs.millis.ai/api-reference/endpoint/call-logs/list-call-histories get /call-logs # Register Call Source: https://docs.millis.ai/api-reference/endpoint/call-management/register-call post /register_call # Register Sip Call Source: https://docs.millis.ai/api-reference/endpoint/call-management/register-sip-call post /register_sip_call # Start Outbound Call Source: https://docs.millis.ai/api-reference/endpoint/call-management/start-outbound-call post /start_outbound_call # Terminate Call Session Source: https://docs.millis.ai/api-reference/endpoint/call-management/terminate-call-session post /sessions/{session_id}/terminate # Create Campaign Source: https://docs.millis.ai/api-reference/endpoint/campaigns/create-campaign post /campaigns # Delete Campaign Source: https://docs.millis.ai/api-reference/endpoint/campaigns/delete-campaign delete /campaigns/{campaign_id} # Delete Campaign Record Source: https://docs.millis.ai/api-reference/endpoint/campaigns/delete-campaign-record delete /campaigns/{campaign_id}/records/{phone} # Edit Campaign Info Source: https://docs.millis.ai/api-reference/endpoint/campaigns/edit-campaign-info put /campaigns/{campaign_id}/info # Get All Campaigns Source: https://docs.millis.ai/api-reference/endpoint/campaigns/get-all-campaigns get /campaigns # Get Campaign Source: https://docs.millis.ai/api-reference/endpoint/campaigns/get-campaign get /campaigns/{campaign_id} # Get Campaign Info Source: https://docs.millis.ai/api-reference/endpoint/campaigns/get-campaign-info get /campaigns/{campaign_id}/info # Set Caller Source: https://docs.millis.ai/api-reference/endpoint/campaigns/set-caller post /campaigns/{campaign_id}/set_caller # Start Campaign Source: https://docs.millis.ai/api-reference/endpoint/campaigns/start-campaign post /campaigns/{campaign_id}/start # Stop Campaign Source: https://docs.millis.ai/api-reference/endpoint/campaigns/stop-campaign post /campaigns/{campaign_id}/stop # Upload Records Source: https://docs.millis.ai/api-reference/endpoint/campaigns/upload-records post /campaigns/{campaign_id}/records # Create a WebRTC offer Source: https://docs.millis.ai/api-reference/endpoint/connectivity/create-a-webrtc-offer post /webrtc/offer Create a WebRTC offer for a call. # Create Sip Source: https://docs.millis.ai/api-reference/endpoint/connectivity/create-sip post /sip # Delete Sip Source: https://docs.millis.ai/api-reference/endpoint/connectivity/delete-sip delete /sip/{call_id} # Chat Completions Source: https://docs.millis.ai/api-reference/endpoint/conversational-ai/chat-completions post /chat/completions # Create File Source: https://docs.millis.ai/api-reference/endpoint/knowledge-base/create-file post /knowledge/create_file # Delete File Source: https://docs.millis.ai/api-reference/endpoint/knowledge-base/delete-file post /knowledge/delete_file # Generate Presigned Url Source: https://docs.millis.ai/api-reference/endpoint/knowledge-base/generate-presigned-url post /knowledge/generate_presigned_url # List Files Source: https://docs.millis.ai/api-reference/endpoint/knowledge-base/list-files get /knowledge/list_files # Set Agent Files Source: https://docs.millis.ai/api-reference/endpoint/knowledge-base/set-agent-files post /knowledge/set_agent_files # Delete Phone Source: https://docs.millis.ai/api-reference/endpoint/phones/delete-phone delete /phones/{phone} # Get Phone Source: https://docs.millis.ai/api-reference/endpoint/phones/get-phone get /phones/{phone} # Get Phones Source: https://docs.millis.ai/api-reference/endpoint/phones/get-phones get /phones # Import Phone Number Source: https://docs.millis.ai/api-reference/endpoint/phones/import-phone-number post /phones/import # Purchase Phone Number Source: https://docs.millis.ai/api-reference/endpoint/phones/purchase-phone-number post /phones/purchase # Set Agent Source: https://docs.millis.ai/api-reference/endpoint/phones/set-agent post /phones/{phone}/set_agent # Set Agent Config Override Source: https://docs.millis.ai/api-reference/endpoint/phones/set-agent-config-override post /phones/{phone}/agent-config-override Set agent config override for a specific phone number # Set Phone Agent Source: https://docs.millis.ai/api-reference/endpoint/phones/set-phone-agent post /set_phone_agent # Update Phone Tags Source: https://docs.millis.ai/api-reference/endpoint/phones/update-phone-tags put /phones/{phone}/tags Update all tags for a phone number (replaces existing tags) # Get User Info Source: https://docs.millis.ai/api-reference/endpoint/user/get-user-info get /user/info # Get Custom Voices Source: https://docs.millis.ai/api-reference/endpoint/voices/get-custom-voices get /voices/custom # Get Voices Source: https://docs.millis.ai/api-reference/endpoint/voices/get-voices get /voices # Create File Source: https://docs.millis.ai/api-reference/knowledge-base/create-file https://api-west.millis.ai/openapi.json post /knowledge/create_file # Delete File Source: https://docs.millis.ai/api-reference/knowledge-base/delete-file https://api-west.millis.ai/openapi.json post /knowledge/delete_file # Generate Presigned Url Source: https://docs.millis.ai/api-reference/knowledge-base/generate-presigned-url https://api-west.millis.ai/openapi.json post /knowledge/generate_presigned_url # List Files Source: https://docs.millis.ai/api-reference/knowledge-base/list-files https://api-west.millis.ai/openapi.json get /knowledge/list_files # Set Agent Files Source: https://docs.millis.ai/api-reference/knowledge-base/set-agent-files https://api-west.millis.ai/openapi.json post /knowledge/set_agent_files # Delete Phone Source: https://docs.millis.ai/api-reference/phones/delete-phone https://api-west.millis.ai/openapi.json delete /phones/{phone} # Get Phone Source: https://docs.millis.ai/api-reference/phones/get-phone https://api-west.millis.ai/openapi.json get /phones/{phone} # Get Phones Source: https://docs.millis.ai/api-reference/phones/get-phones https://api-west.millis.ai/openapi.json get /phones # Import Phone Number Source: https://docs.millis.ai/api-reference/phones/import-phone-number https://api-west.millis.ai/openapi.json post /phones/import # Purchase Phone Number Source: https://docs.millis.ai/api-reference/phones/purchase-phone-number https://api-west.millis.ai/openapi.json post /phones/purchase # Set Agent Source: https://docs.millis.ai/api-reference/phones/set-agent https://api-west.millis.ai/openapi.json post /phones/{phone}/set_agent # Set Agent Config Override Source: https://docs.millis.ai/api-reference/phones/set-agent-config-override https://api-west.millis.ai/openapi.json post /phones/{phone}/agent-config-override Set agent config override for a specific phone number # Set Phone Agent Source: https://docs.millis.ai/api-reference/phones/set-phone-agent https://api-west.millis.ai/openapi.json post /set_phone_agent # Update Phone Tags Source: https://docs.millis.ai/api-reference/phones/update-phone-tags https://api-west.millis.ai/openapi.json put /phones/{phone}/tags Update all tags for a phone number (replaces existing tags) # Get User Info Source: https://docs.millis.ai/api-reference/user/get-user-info https://api-west.millis.ai/openapi.json get /user/info # Set Credential Source: https://docs.millis.ai/api-reference/user/set-credential https://api-west.millis.ai/openapi.json post /user/credentials # Get Custom Voices Source: https://docs.millis.ai/api-reference/voices/get-custom-voices https://api-west.millis.ai/openapi.json get /voices/custom # Get Voices Source: https://docs.millis.ai/api-reference/voices/get-voices https://api-west.millis.ai/openapi.json get /voices # Agent Source: https://docs.millis.ai/core-concepts/agent Voice Agents are the core components of the Millis AI platform. These agents can be customized to perform a variety of tasks, from answering questions to guiding users through complex processes, all via natural spoken language. ### Agent Configurations It's important to configure your voice agent to ensure it operates effectively within your specific context. Here are the main aspects you can customize: ```json theme={null} { prompt: "You're a helpful assistant.", // Example prompt voice: { provider: "elevenlabs", // Voice provider voice_id: "voice-id" // Replace 'voice-id' with the ID of the desired voice }, language: "", // optional - use language code such as en, es tools: [ { name: "get_user_data", description: "", webhook: "https://...", header: { "Content-Type": "application/json", "Authorization": "" }, params: [ { name: "", type: "string" | "number" | "boolean", description: "", required: true } ] } ], // Replace with actual function calls you need custom_llm_websocket: "wss://...", // optional - enable custom llm llm: "", // optional - choose llm model. Ex: gpt-4o, llama-3-70b } ``` ### What You Can Customize #### Prompt: The system prompt is where you can provide specific instructions or information that the agent needs to remember and follow. This sets the initial context for your voice agent, guiding its responses and interactions. #### LLM (Large Language Model): (Optional) If not set, the default Millis AI model is used. * **Model**: Specifies the GPT model that your agent will operate on. We support OpenAI's latest model, GPT-4o, as well as open-source models like Meta Llama 3. * **Provider**: The provider who provides inference for the model. #### Voice Settings: * **provider**: The service provider for the text-to-speech service. This config determines the quality of your agent's voice. * **voice\_d**: The specific voice character from the chosen provider's catalog, allowing you to customize how your agent sounds. #### Language: Defines the operational language of the agent. If not specified, English is used by default. #### Custom LLM WebSocket: (Optional) If you prefer using your own custom LLM, specify a WebSocket URL to enable this connection. #### Tools: A list of function calls the agent can execute to perform tasks or retrieve information during interactions. This includes API webhooks and other integrations. ### LLM Model Choices You can select the AI model for your voice agent based on your needs: * **Default Millis AI Model**: Automatically used if no specific LLM model is provided. This model is best optimized for low latency. * **Popular Models from Providers**: Like OpenAI's GPT-4o, known for the best language processing capabilities but with a trade-off in latency. * **Custom Model via WebSocket**: Integrate your uniquely developed or tailored LLM to give your agent specialized abilities. You have full control over the agent's capabilities. # Agent Config Source: https://docs.millis.ai/core-concepts/agent-config Overview of all configuration options available for setting up and customizing agents in Millis. Learn how to manage voice settings, conversation flows, metadata inclusion, voicemail handling, and more. # Functions for Agents Source: https://docs.millis.ai/core-concepts/functions Functions are additional capabilities that you can integrate into your voice agents to enhance their utility and interaction dynamics. ## Function Types: #### API Webhooks These allow the agent to perform actions or retrieve information during a conversation by calling external APIs. This is useful for tasks like booking appointments, fetching user-specific data, or updating records in real-time. #### Web Form Trigger Implement a function where the agent can prompt users to fill out a web form during a conversation. This is particularly useful for gathering detailed information or when textual input is more practical than voice. For example: Email, phone number, name, etc. ## Understanding Function Structure ### Webhook Function Webhook functions include the following components: ```typescript theme={null} interface FunctionCall { // Main structure for defining a function. name: string; // Function name, formatted as a valid identifier (no spaces, begins with a letter) description: string; // Detailed description to help the agent understand when to use the function webhook: string; // The URL of the webhook to which the request will be sent header: object; // Any necessary headers for the webhook request params: Param[]; // Array of `Param` objects, defining the parameters needed by the function } interface Param { name: string; // Valid parameter name, no spaces, starts with a letter, can use underscores or camelCase type: ParamType; // The data type of the parameter (e.g., string, number, boolean) description: string; // Detailed description of the parameter required: boolean; // Whether this parameter is mandatory } ``` * **Function and Parameter Naming**: Ensure that the function name and parameter names are formatted as valid identifiers. They should have no spaces, begin with a letter, and can include underscores or use camelCase, such as "get\_email" or "getEmail". * **Descriptions** Provide comprehensive details in your function and parameter descriptions to help the agent understand what the function is for and when to use it. ### Web Form Function Web form functions allow your voice agent to trigger web forms on browser during conversation for data collection or user input. This is particularly useful for gathering detailed information or when textual input is more practical than voice. For example: Email, phone number, name, etc. ```json theme={null} export interface WebFormFunction { name: string; // Function name, formatted as a valid identifier description: string; // Detailed description to help the agent understand the function's purpose type: "web_form"; // Specify this as 'web_form' for web form functions data: { // Define the parameter to collect from the web form param: Param; }; } ``` ## Example ```json theme={null} // Webhook { name: "getUserInfo", description: "Retrieves user information from the database using their email.", webhook: "https://api.example.com/user", header: { "Content-Type": "application/json", "Authorization": "Bearer your_access_token" }, params: [ { name: "email", type: "string", description: "The user's email that you collect during the conversation.", required: true } ] }; // Web Form { name: "open_email_form", description: "Trigger a web form for user to enter contact details.", type: "web_form", data: { param: { name: "email", type: "string", description: "Collect user's email info.", required: true } } }; ``` Once you have defined your function, you can integrate it into your agent’s configuration. Add the function to the tools array in your agent config. ```json theme={null} { "tools": [ { name: "getUserInfo", // the rest of the function definition } ] } ``` # Attach metadata to call Source: https://docs.millis.ai/core-concepts/metadata Attach metadata to your call sessions for personalized conversations Millis AI allows you to attach metadata to your call sessions when using the Web SDK or starting outbound calls. Metadata can include any user-specific data, such as the caller's name, user ID, or other relevant information. The system provides multiple ways to utilize this metadata during the session. ## How to Add Metadata ### 1. Via URL Parameters (Embedded Widget) When using the embeddable call widget, you can add metadata directly through URL parameters: ```url theme={null} https://app.millis.ai/agents/embedded?id=XXXX&k=YYYYYY&userName=John&userType=premium ``` Any query parameters added to the widget URL will automatically become metadata for that session. ### 2. Using the Web SDK You can add metadata to a call using the Web SDK by providing the metadata as the second parameter in the `start` method. The third parameter, `include_metadata_in_prompt`, determines whether the metadata should be included in the agent’s system prompt. ```javascript theme={null} msClient.start(, metadata?: object, include_metadata_in_prompt?: boolean); ``` * The first parameter is the `agentId` or a temporary agent configuration. * The second parameter is the metadata you want to attach to the session. * The third parameter is `include_metadata_in_prompt`, which controls whether the metadata is used by the agent during the conversation. Learn more about our Web SDK [here](/integration/web-sdk). ### 3. Using the native Websocket connection You can also attach metadata by initiating a session through a WebSocket connection. Include the metadata in the `initiate` method payload. ```json theme={null} { "method": "initiate", "data": { "agent": { "agent_id": "your_agent_id", Or replace with agent_config: for dynamic configuration }, "public_key": "your_public_key", "metadata": { "key": "value" }, "include_metadata_in_prompt": true } } ``` Learn more about building native apps using websocket [here](/integration/native-apps-with-websocket). ### 4. Using the Outbound API To add metadata using the outbound API, you need to include it in the request body when calling the `start_outbound_call` API. You can also specify whether to include the metadata in the agent’s prompt. ```json theme={null} { "from_phone": "one of your agent's phone number", "to_phone": "receiver's phone number", "metadata": { "key": "value" }, "include_metadata_in_prompt": true } ``` * **`from_phone`**: One of your agent's phone numbers. * **`to_phone`**: The phone number of the call recipient. * **`metadata`**: Optional. Any extra data you want to attach to the session. * **`include_metadata_in_prompt`**: Optional. Set to `true` to include metadata in the agent's prompt. Defaults to `false`. Learn more about Outbound Call [here](/integration/outbound-call). ## How to retrieve and use Metadata The metadata will stay associated with that session throughout its lifecycle. This metadata will also be included in the following: * **Prefetch Data Webhook**: Metadata is forwarded during the session's prefetch data webhook which you can use to retrieve personalized data. * **End of Call Webhook**: Metadata is passed along at the conclusion of the call, allowing you to track and identify sessions. Learn more about the webhooks [here](/core-concepts/webhooks). ### Use Metadata as Variables Any metadata you attach to a call session can be used as dynamic variables throughout your call flow. Variables can be referenced in agent prompts, messages, webhook parameters, and function calls using the `{variableName}` syntax. For example, if you add metadata like: ```json theme={null} { "userName": "John", "userType": "premium" } ``` You can reference these values using `{userName}` or `{userType}` in various places during the call. For detailed information about using variables, including syntax and examples, see our [Variables documentation](/core-concepts/variables). # Running Voice Agents in Different Regions Source: https://docs.millis.ai/core-concepts/regions Millis AI now supports running voice agents in various regions, including the EU, to help reduce latency for calls in those regions and ensure compliance with EU laws. This documentation will guide you through the process of selecting and configuring regions for your voice agents. ### Selecting Regions There are two primary ways to set the region for your voice agents in Millis: 1. Set Region for Your Phone Number When importing phone numbers from Twilio or Vonage, you can choose the desired region. This setting will ensure that calls are routed through the selected region, optimizing latency and compliance. 2. Set Region Endpoint When Starting Your Agent from Web SDK You can specify the region endpoint when initializing your agent using the Millis Web SDK. Use the following code to set the region: ```javascript theme={null} const msClient = Millis.createClient({ publicKey: '', endPoint: '' }); ``` Replace `` with your actual Millis public key and `` with the endpoint of the desired region. ### List of Available Endpoints * **us-west**: `wss://api-west.millis.ai/millis` * **eu-west**: `wss://api-eu-west.millis.ai/millis` More endpoints will be available soon. # Session Continuation Source: https://docs.millis.ai/core-concepts/session-continuation Allow agent to continue past conversations with users Millis AI’s Session Continuation feature enables agents to leverage previous interaction data, allowing users to continue conversations seamlessly from prior sessions. By passing a `session_id`, agents can access past context, enhancing engagement and providing a personalized experience for users. *** ## Important Note on Data Opt-Out **Warning**: Session Continuation is only available if **Data Opt-Out is disabled**. When Data Opt-Out is enabled, Millis does not retain call history, so agent can't retrieve data from previous sessions. Ensure that Data Opt-Out is disabled if you require session continuation for your users. *** ## How to Pass `session_id` for Session Continuation ### 1. **Web SDK Integration** To enable session continuation using the Millis Web SDK, use the `msClient.start` method: ```javascript theme={null} msClient.start({ agent: { agent_id: agentId }, session_continuation: { session_id: "" } }); ``` Make sure you upgrade your web sdk to v1.0.15 to have this option. *** ### 2. **Native Integration via WebSocket** To continue a session via WebSocket, include the `session_id` in the `initiate` event: ```json theme={null} { "method": "initiate", "data": { "agent": { "agent_id": "your_agent_id", // or for dynamic configuration "agent_config": "" }, "public_key": "your_public_key", "metadata": { "key": "value" }, "include_metadata_in_prompt": true, "session_continuation": { "session_id": "" } } } ``` *** ### 3. **Outbound Call API Integration** For outbound calls, the `session_id` is included in the request body when calling the `start_outbound_call` API. #### Example Request ```json theme={null} { "from_phone": "your_agent_phone_number", "to_phone": "receiver_phone_number", "metadata": { "key": "value" }, "include_metadata_in_prompt": true, "session_continuation": { "session_id": "" } } ``` *** ## Example Scenarios 1. **Follow-Up Customer Support Calls**: A returning customer can pick up from a previous conversation by including the `session_id`, reducing the need to re-explain their issue. 2. **Ongoing Campaigns**: In multi-stage campaigns, `session_id` helps track each caller's journey, creating a more cohesive experience. 3. **Consultations**: Advisors can use session continuation to reference past discussions, fostering a more personalized relationship. *** ## Future Enhancements * **User Identification:** Upcoming features will allow sessions to continue based on `user_id`, `phone`, or similar identifiers, adding memory across sessions without requiring a `session_id`. # Dynamic Variables Source: https://docs.millis.ai/core-concepts/variables Dynamic Variables in Call Sessions # What Are Variables? Variables are pieces of data that can be dynamically inserted during call sessions. They can now be embedded within different parts of a call flow, including prompts, agent messages, webhook parameters, and function calls. This flexibility allows developers to build richer and more responsive interactions for their agents. In every call, there are predefined variables like `FromPhone` and `ToPhone` (representing the originating and receiving phone numbers for phone calls). Additionally, you can utilize any key-value pairs from the metadata that you provide for the call. For example, if you include custom information like `customerID`, `appointmentTime`, or any other data that you may want to reference throughout the call, these can now be easily used. # How to Use Variables Millis supports a simple syntax for using variables. Just wrap the variable name in curly braces like `{}`, and our system will automatically replace it with the corresponding value during the call. Here are a few examples: * To fetch the caller info by number: Use `{FromPhone}` as the description of the function's param.” * To greet the caller by name via metadata: Set agent's greeting line to be “Hi `{userName}! How can I assist you today?`.” * Webhook parameters: You can add variables as parameters in a webhook, such as `phone_number={FromPhone}`. ### Default and Dynamic Variables Millis offers both default variables and dynamic variables for customizing call sessions: * **Default Variables** (Special Variables): These are system-provided variables such as `FromPhone` and `ToPhone`, which represent the caller's phone number and the receiving phone number, respectively, when the call is made via a phone network. * **Dynamic Variables**: These are custom key-value pairs that you provide as metadata when starting a call. You can use any metadata as a variable within prompts, messages, function calls, or webhook parameters. ### Examples of How to Use Variables * **For Function Calls**: Put `{}` in the description of the parameter. For instance, if the parameter name is `phone_number`, you can set the parameter description to `{FromPhone}` to automatically include the caller's phone number. Use variable in param * **For Agent Messages**: Add `{userName: }` to metadata, then set the agent's greeting line to: `Hi {userName}! How can I help you?`. This can also be applied to the agent's prompt. # Webhooks Source: https://docs.millis.ai/core-concepts/webhooks Millis AI Webhooks Documentation Millis AI provides flexible webhooks to enhance integration capabilities. This documentation explains the purpose, format, and fields for the **Prefetch Data Webhook** and **End-of-Call Webhook**. *** ## **1. Prefetch Data Webhook** **Configuration Key in Agent Config**: `extra_prompt_webhook` The Prefetch Data Webhook is called before the conversation begins. It enables real-time customization and event notifications. ### Configuration The configuration can be provided in one of the following formats: * **String**: The URL of the webhook endpoint. * **Object**: ```json theme={null} { "url": "endpoint", "headers": { "key": "value" } } ``` ### **Use Cases** 1. **Call Notification**: Notify your system about an incoming call by receiving a webhook event. 2. **Metadata Override**: Dynamically override or add session metadata based on external systems. 3. **Extra Prompt**: Provide additional context to the agent's system prompt. ### **Request** * **Method**: `GET` * **Query Parameters**: * `session_id` (string): Unique session identifier. * `agent_id` (string): ID of the agent handling the session. * `from` (string): Caller’s phone number (if applicable). * `to` (string): Receiver’s phone number (if applicable). * Additional session metadata as key-value pairs (if available). ### **Expected Response** * **Content-Type**: `application/json`. * **Response Fields**: * **`metadata` (object)**: Overrides or extends the current session metadata. Existing keys are updated, and new keys are added. * **`extra_prompt` (string)**: Additional text appended to the agent’s system prompt. ### **Response Example** ```json theme={null} { "metadata": { "customer_tier": "premium", "location": "California" }, "extra_prompt": "The caller is a premium customer from California." } ``` *** ## **2. End-of-Call Webhook** **Configuration Key in Agent Config**: `session_data_webhook` The End-of-Call Webhook is triggered after a session concludes. It provides detailed information about the session for logging, analytics, or post-call processing. ### Configuration The configuration can be provided in one of the following formats: * **String**: The URL of the webhook endpoint. * **Object**: ```json theme={null} { "url": "endpoint", "headers": { "key": "value" } } ``` ### **Request** * **Method**: `POST` * **Payload** (JSON): * **`chat` (string)**: JSON string representing the chat history (user and agent messages). * **`function_calls` (array)**: List of external functions invoked during the session. * **`ts` (timestamp)**: Session start time. * **`duration` (integer)**: Session duration in seconds. * **`agent_config` (object)**: Serialized agent configuration. * **`agent_id` (string)**: ID of the agent managing the session. * **`call_id` (string)**: Unique identifier for the call session. * **`chars_used` (integer)**: Number of characters processed during the session. * **`session_id` (string)**: Unique session identifier. * **`cost_breakdown` (array)**: List of cost components for different services. ```json theme={null} [ { "type": "stt", "provider": "millis", "credit": 0.0043 }, { "type": "tts", "provider": "elevenlabs", "credit": 0.03 } ] ``` * **`voip` (object)**: Telephony details. ```json theme={null} { "provider": "twilio", "from": "+1234567890", "to": "+0987654321" } ``` * **`recording` (object)**: Call recording details. ```json theme={null} { "recording_url": "https://your-recording-url.com/recording123" } ``` * **`metadata` (object)**: Final metadata for the session. * **`call_status` (string)**: Status of the call. Possible values: * `user-ended` * `api-ended` * `voicemail-message` * `voicemail-hangup` * `agent-ended` * `timeout` * `error` * `chat_completion` * **`error_message` (string)**: Description of any errors encountered during the session. ### **Example** **Request URL**: ```http theme={null} POST https://your-webhook-url.com/session-data ``` **Payload**: ```json theme={null} { "chat": "[{\"role\": \"user\", \"content\": \"Hi, I'd like to book a flight.\"}, {\"role\": \"assistant\", \"content\": \"Sure! Can you provide the destination and date?\"}]", "function_calls": [{'name': 'check_status', 'params': {'check': 'true', 'order_id': 6689, 'session_id': '-OA1XVnwRGpaameIakJb'}}], "ts": 1729840680.886249, "duration": 300, "agent_config": {...}, "agent_id": "agent001", "call_id": "call123", "chars_used": 1024, "session_id": "session123", "cost_breakdown": [ { "type": "stt", "provider": "millis", "credit": 0.05 } ], "voip": { "provider": "twilio", "from": "+1234567890", "to": "+0987654321" }, "recording": { "recording_url": "https://your-recording-url.com/recording123" }, "metadata": { "customer_tier": "gold" }, "call_status": "user-ended", "error_message": null } ``` *** ## **Notes** 1. **Prefetch Webhook**: * Use this webhook to notify your system about calls, adjust metadata dynamically, or provide extra prompts to enhance conversational context. 2. **End-of-Call Webhook**: * Designed for detailed post-call analytics and tracking. * Ensure secure storage of sensitive information like call recordings and metadata. 3. **Security**: * Use HTTPS for all webhooks. * Authenticate requests with headers or API keys as needed. These webhooks provide a robust way to integrate Millis AI into your existing systems, offering flexibility and advanced customization for voice agent interactions. # Custom LLM Source: https://docs.millis.ai/integration/custom-llm Integrating a Custom LLM with Millis AI Voice Agent ## Basic This guide describes how to integrate your own LLM chatbot with a Millis AI voice agent. By connecting your custom LLM, you can power the voice agent with your chatbot's capabilities, providing a seamless voice interaction experience based on your model's responses. ## Prerequisites * Create your Voice Agent on the [Playground](https://app.millis.ai/agents) * Setup a websocket server on your end. ## Set Up Your WebSocket Endpoint: * When an outbound or inbound call is initiated with your voice agent, the Millis AI server will establish a connection to your specified WebSocket URL. * Your endpoint should be capable of both receiving messages from and sending messages to the Millis AI server. Here's a [Sample Code](/tutorials/setup-custom-llm-websocket). * Here's how the interaction flows after connection established: ### 1. Initiate a call: Millis AI server will send `start_call` event to tell your server when the conversation starts. ```json theme={null} { "type": "start_call", "data": { "stream_id": , "agent_id": , "call_sid": , // optional, None if conversation is not via phone call "session_id": , // unique id of the call session "metadata": {your own metadata}, // The metadata that you add in msClient.start function "voip": { "from": , "to": } // optional, None if conversation is not via phone call } } ``` ### 2. Listen to user's message: Millis AI streams the user's spoken message, including the full conversation transcript, to your LLM. ```json theme={null} { "type": "stream_request", "data": { "stream_id": , "transcript": [] } } ``` ### 3. Generate LLM Responses: Your LLM processes the transcript and streams back the response. Indicate the end of a message stream with `end_of_stream`. ```json theme={null} { "type": "stream_response", "data": { "stream_id": request['stream_id'], "content": "text", "flush": true/false, // optional "pause": , // optional "end_of_stream": true/false } } ``` * `flush`: Set this to `true` to instruct the agent to immediately generate audio based on the current response. If `false`, the agent will buffer the response and generate audio only when it receives a complete sentence. * `pause`: Set this to a number of milliseconds to instruct the agent to pause for that long after saying the response before saying the next response. - When your LLM generates a response, attach the `stream_id` from the original request so that we can keep track of which response corresponds to which request. - For the first message that your server sends after receiving the `start_call` event, use the `stream_id` from the `start_call` event. ## Control the Call from Your Custom LLM: Your custom LLM can send specific messages to control the flow of the call. Instead of sending `stream_response`, you can send the following types: ### 1. End the Call To terminate the call: ```json theme={null} { "type": "end_call", "data": { "stream_id": } } ``` ### 2. Transfer the Call To transfer the call to another destination (e.g., phone number): ```json theme={null} { "type": "transfer_call", "data": { "stream_id": , "destination": "+1234567890" } } ``` **Parameters:** * `stream_id`: The unique identifier for the stream. * `destination`: The phone number or endpoint to transfer the call to. ## Handle advanced interaction: Millis AI manages the conversation flow, including interruption detection and end-of-turn signals. You will be notified of these events: ### `partial_transcript` **Description:** Sent to provide a partial transcript of the conversation. The transcript can be either final or partial. **Message Structure:** ```json theme={null} { "type": "partial_transcript", "data": { "session_id": "string", "transcript": "string", "is_final": "boolean" } } ``` **Parameters:** * `session_id`: The unique identifier for the session. * `transcript`: The partial or complete transcript text. * `is_final`: Boolean indicating whether the transcript is final. ### `playback_finished` **Description:** Sent when the playback of agent's audio stream has finished. **Message Structure:** ```json theme={null} { "type": "playback_finished", "data": { "session_id": "string", "stream_id": "integer" } } ``` **Parameters:** * `session_id`: The unique identifier for the session. * `stream_id`: The unique identifier for the stream. ### `interrupt` **Description:** Sent when user interrupts agent's stream. **Message Structure:** ```json theme={null} { "type": "interrupt", "stream_id": "integer" } ``` **Parameters:** * `stream_id`: The unique identifier for the stream. *** ## Connect your Voice Agent to your Custom LLM: In your voice agent’s configuration on the Millis AI platform, specify your WebSocket endpoint. Custom LLM Step 1 Custom LLM Step 2 # Embeddable Call Widget Source: https://docs.millis.ai/integration/embeddable-call-widget Embedding a Voice Agent Call Widget into Your Web Application Millis AI offers a simple and effective way to integrate voice interaction into your web applications through our embeddable call widget. This widget allows users to interact with your voice agent directly from your website, providing a seamless user experience. The widget includes a button to start and stop interactions and features an animation of an audiogram to visually represent the audio interaction similar to our demo page. ## Steps to Embed the Voice Agent Call Widget #### Step 1: Create your voice agent #### Step 2. Obtain the widget embeddable code * Navigate to the voice agent you want to embed. * Click on the "Actions" button on the top right and select 'Embed to public site'. Embed Widget Step 1 * Copy the provided HTML code. Embed Widget Step 2 #### Step 3: Embed the widget into your web application With the HTML code, you can place it anywhere in your web app to embed the widget. Here's how to do it on Webflow: * Navigate to the designated area and add a "Code Embed" component. Embed Widget Step 3 * Paste the HTML code provided above, then click 'Save'. Embed Widget Step 4 ## Adding Metadata via URL Parameters You can customize each widget session by adding URL parameters that will be passed as metadata. This allows you to provide context-specific information to your voice agent. ### Example Usage You can add metadata parameters like this: ```html theme={null} https://app.millis.ai/agents/embedded?id=XXXX&k=YYYYYY&userName=John&userType=premium ``` These parameters will be automatically converted into metadata for the session, allowing your agent to access this information during the conversation. You can reference this metadata using [variables](/core-concepts/variables) in your agent's prompts or function calls. ### Common Use Cases * Adding user identification information * Passing context about the page or section where the widget is embedded * Providing custom configuration parameters for the conversation Remember that any metadata added via URL parameters will be visible in the URL. Don't include sensitive information this way. Learn more about how metadata works in Millis AI [here](/core-concepts/metadata). # Inbound call Source: https://docs.millis.ai/integration/inbound-call Let Millis AI handle your inbound calls in twilio ## Basic Learn how to setup your twilio phone number so such that the call gets handle by a voice agent from Millis AI specified by you. #### Prerequisites Obtain your public\_key from your [Millis AI Account](https://app.millis.ai/settings/keys). Setup a voice agent and obatain the agent\_id from your [Millis AI Dashboard](https://app.millis.ai/agents). #### Method 1 : TwiML Bin Create a TwiML Bin with the following configuration. Replace public\_key and agent\_id with your own. ``` ``` 1. In your phone number setting, select Webhook, TwiML Bin, Function, Studio Flow, Proxy Service 2. Select Twiml Bin and the name of your TwiML Bin for `A call comes in` section. #### Method 2 : Use your own endpoint 1. In your phone number setting, select Webhook, TwiML Bin, Function, Studio Flow, Proxy Service 2. Select Webhook and your own backend endpoint for `A call comes in` 3. Handle the webhook as you like and return the following as your response. ``` ``` # Build Native Apps with Websocket Source: https://docs.millis.ai/integration/native-apps-with-websocket Using Millis Platform via WebSocket to build voice agents on desktop and mobile This tutorial guides you through the process of integrating the Millis AI platform directly via WebSocket to build voice agents for desktop or mobile apps. Users can capture audio natively and send it to Millis via WebSocket, receiving voice responses in real-time. ## Requirements * Create your Voice Agent on the [Playground](https://app.millis.ai/agents) * Use native APIs on desktop or mobile to capture and playback audio. * Establish websocket connection to Millis server. ## Overview * **WebSocket Endpoint:** wss\://api-west.millis.ai:8080/millis * **Sample Rate:** 16000 Hz * **Encoding:** PCM * **Channels:** 1 * **Chunk Size:** Any ## Step-by-Step Guide ### 1. Establishing a WebSocket Connection Begin by establishing a connection with the Millis AI WebSocket endpoint. Here's an example code in javascript. ```javascript theme={null} let ws = new WebSocket("wss://api-west.millis.ai:8080/millis"); ws.binaryType = "arraybuffer"; ``` ### 2. Sending the Initiate Message Once connected, send an initiate message to start the interaction. ```javascript theme={null} ws.onopen = () => { let initiateMessage = { method: "initiate", data: { agent: { agent_id: "your_agent_id", // Or replace with agent_config: for dynamic configuration } public_key: "your_public_key", metadata: { key: value }, // Optional: extra data attached to the call include_metadata_in_prompt: true/false // Optional: option to include the metadata in the agent's system prompt } }; ws.send(JSON.stringify(initiateMessage)); }; ``` ```javascript theme={null} ws.onopen = () => { let initiateMessage = { method: "initiate", data: { agent: { agent_config: { prompt: "", voice: { provider: "elevenlabs", voice_id "..." } } } public_key: "your_public_key", metadata: { key: value }, // Optional: extra data attached to the call include_metadata_in_prompt: true/false // Optional: option to include the metadata in the agent's system prompt } }; ws.send(JSON.stringify(initiateMessage)); }; ``` Millis will respond with the message `{"method": "onready"}` indicating readiness. ### 3. Capturing and Sending Audio Capture audio on your device and send it as an ArrayBuffer to Millis. Make sure it's an `Uint8Array`. ```javascript theme={null} function sendAudioPacket(audioData) { let audioBuffer = new Uint8Array(audioData); ws.send(audioBuffer); } ``` **Note**: Audio packets should be in PCM format, 16000 Hz sample rate, and mono (1 channel). ### 4. Receiving and Playing Audio Responses Millis will send audio responses as ArrayBuffers with the same format and sample rate. You need to buffer and play these on your side. ```javascript theme={null} ws.onmessage = (event) => { if (event.data instanceof ArrayBuffer) { // ArrayBuffer received, handle as audio packets let audioResponse = new Uint8Array(event.data); // Buffer and play the audio response } else { // String received, handle as normal events let message = JSON.parse(event.data); handleIncomingMessage(message); } }; ``` ArrayBuffer data will be the audio packets, while string data indicates normal events that you need to process accordingly. ### 5. Keeping the Connection Alive Send a `{"method": "ping"}` message every 1000 packets to keep the connection alive. ### 6. Handling Incoming Events from Millis Millis may send various events to manage the session and interaction. Here is the logic behind each message: * **pause**: Millis detected some voice activity from the client. The agent decides to temporarily pause talking and observe the next voice activity. In this case, you should still keep and buffer incoming audio packets but not play them. * **unpause**: If Millis detects that it's not the human trying to talk over or interrupt, the agent will continue talking. In this case, you should continue playing audio packets in the buffer. * **clear**: Millis detected human's voice, indicating human interruption intent. The agent will reset and stay silent to let the human continue talking. In this case, clear all audio buffers and stop playback. * **ontranscript**: Real-time transcript of the client's audio. * **onresponsetext**: Real-time transcript of the agent's response. * **onsessionended**: For any reason Millis decides to end the session, you will receive this event. * **start\_answering**: The agent decides to start answering the human's query. * **ai\_action**: For debug purposes. During the conversation, Millis AI intelligently decides to take some action. Listen to this event to understand what the agent is trying to do. **Example:** ```javascript theme={null} function handleIncomingMessage(message) { switch (message.method) { case "pause": // Pause playback and buffer incoming audio packets break; case "unpause": // Resume playback of buffered audio packets break; case "clear": // Clear audio buffer and stop playback break; case "ontranscript": console.log("Client's audio transcript:", message.data); break; case "onresponsetext": console.log("Agent's response transcript:", message.data); break; case "onsessionended": console.log("Session ended."); ws.close(); break; case "start_answering": console.log("Agent starts answering the query."); break; case "ai_action": console.log("AI Action:", message.data); break; } } ``` ### 7. Closing the Connection Simply close the WebSocket connection to stop the conversation. # Outbound call Source: https://docs.millis.ai/integration/outbound-call Trigger an outbound call to a specific number ## Basic This API allows you to initiate outbound phone calls from a specified Millis AI voice agent to any given phone number. #### Prerequisites Obtain your API key from your [Millis AI Account](https://app.millis.ai/settings/keys). #### API Details * Method: POST * URL: ```console theme={null} https://api-west.millis.ai/start_outbound_call ``` * Headers: ```console theme={null} Content-Type: application/json Authorization: YOUR_API_KEY ``` * Body: ```json theme={null} { "from_phone": "one of your agent's phone number", "to_phone": "receiver's phone number", "metadata": { key: value }, // Optional: any extra data to attach to the session include_metadata_in_prompt: true/false // Optional: option to include the metadata in the agent's system prompt } ``` Ensure the phone number includes the full international dialing format (e.g., +15555555555), with no dashes or spaces. (Only US phone nubers are supported) Retrieve your voice agent ID from the Agent Details page on the playground. title ### Example Request using curl ```console theme={null} curl -X POST https://api-west.millis.ai:8080/start_outbound_call \ -H "Content-Type: application/json" \ -H "Authorization: YOUR_API_KEY" \ -d '{"from_phone": "+1234567890", "to_phone": "+1987654321"}' ``` # SIP Trunking Source: https://docs.millis.ai/integration/sip Connecting Phone Numbers to Millis via SIP # Overview Millis provides seamless integration for connecting your phone system to its AI-powered voice agents using SIP. This guide walks you through the process of rerouting phone calls to Millis via SIP. # Step-by-Step Process ## 1. Make a POST Request to `/register_sip_call` To initiate a call, send a POST request to the `/register_sip_call` API endpoint. Depending on your location, select either the **EU-West** or **US-West** region for lower latency: * **EU-West:** `https://api-eu-west.millis.ai/register_sip_call` * **US-West:** `https://api-west.millis.ai/register_sip_call` ### Request Body You must include the necessary parameters in your request body, with the option to customize agent behavior and include metadata if needed. **Request Body Example:** ```json theme={null} { "agent_id": "your_agent_id", "agent_config": { config options for the agent }, "metadata": { "key": "value" }, "include_metadata_in_prompt": true } ``` **Field Details:** * `agent_id`: *(Optional)* The ID of the Millis AI agent that will handle the call. * `agent_config`: *(Optional)* Configuration options for the agent, allowing you to customize behavior. * If both `agent_id` and `agent_config` are provided, the parameters in `agent_config` will override the original parameters for the agent tied to `agent_id`. * You can also provide just `agent_config` for a temporary configuration, which will be used to construct an agent to handle the call. * `metadata`: *(Optional)* Any additional information to attach to the session, giving the agent context to enhance its response. To learn more about how metadata works, visit the [Metadata Guide](/core-concepts/metadata). * `include_metadata_in_prompt`: *(Optional)* Boolean value indicating if the metadata should be included in the agent's conversational prompt. # 2. Receive the SIP URI After making the POST request, you will receive a response containing a `call_id` and a `sip_uri`. This `sip_uri` is the address you will use to route your phone calls to Millis. **Response Example:** ```json theme={null} { "call_id": "unique_call_id", "sip": "sip:call-uniqueid@" } ``` # 3. Reroute Your Call Use the provided `sip_uri` to reroute the call from your phone system to Millis. Your phone system will forward the call audio to Millis, where the voice agent can interact with the caller. # Web SDK Source: https://docs.millis.ai/integration/web-sdk Integrate Millis AI's voice agent capabilities directly into your web applications and browser extensions. ## Installation Install the SDK with npm: ``` npm install @millisai/web-sdk ``` ## Usage Here’s how to quickly set up a voice agent in your web application: ### 1. Import the SDK: ```js theme={null} import Millis from '@millisai/web-sdk'; ``` ### 2. Initialize the Client: ```js theme={null} const msClient = Millis.createClient({publicKey: 'your_public_key', endPoint?: 'region-based-endpoint'}); ``` Obtain your public key from your [Millis AI Playground](https://app.millis.ai/settings/keys) Learn more about which `endPoint` to use [HERE](https://millisai.mintlify.app/core-concepts/regions). ### 3. Start a Conversation: Starting from version 1.0.15, use the following format to initiate a call: ```js theme={null} msClient.start({ agent: { agent_id: agentId, // Optionally pass agent_id agent_config: {} // Optionally pass agent_config }, metadata: {}, // Optional metadata for personalized context include_metadata_in_prompt: true/false, // Optional flag to include metadata in prompt session_continuation: { // Optional session ID for continuation session_id: "" } }); ``` #### Using a Predefined Agent First, create a voice agent from the [Playground](https://app.millis.ai/agents). Then, start a conversation with your agent using the following code: ```js theme={null} msClient.start({ agent: { agent_id: } }); ``` Replace `agent-id` with the ID of your agent obtained from the Playground. The `metadata` is optional. You can pass any additional data to the session, which we will forward to your custom LLM and function webhooks. If you provide `metadata`, you can make it available to the agent by setting `include_metadata_in_prompt` to `true`. This will include the metadata in the agent's system prompt, allowing the agent to use the data during the conversation. #### Dynamically Creating a Temporary Voice Agent You can also dynamically create a temporary voice agent with custom configurations using the code below: ```js theme={null} msClient.start({ agent: { agent_config: { prompt: "You're a helpful assistant.", // Example prompt voice: { provider: "elevenlabs", // Voice provider voice_id: "voice-id" // Replace 'voice-id' with the ID of the desired voice }, language: "", // optional - use language code such as en, es tools: [ { name: "get_user_data", description: "", webhook: "https://...", header: { "Content-Type": "application/json", "Authorization": "" }, params: [ { name: "", type: "string" | "number" | "boolean", description: "", required: true } ] } ], // Replace with actual function calls you need custom_llm_websocket: "wss://...", // optional - enable custom llm llm: "", // optional - choose llm model. Ex: gpt-4o, llama-3-70b } } }); ``` To obtain the voice\_id, use this API to acquire the complete list of voices: [https://api-west.millis.ai:8080/voices](https://api-west.millis.ai:8080/voices) #### Overriding Agent configuration When both `agent_id` and `agent_config` are provided, the session will use the configuration associated with `agent_id` but will override it with any settings provided in `agent_config`. This option allows for minor modifications to the agent’s default configuration on a per-session basis. #### Additional Parameters ##### Metadata * **`metadata`**: Optional field to pass any additional information that may personalize the conversation. It can be used by the agent if `include_metadata_in_prompt` is set to `true`. ##### Including Metadata in Prompt * **`include_metadata_in_prompt`**: Boolean flag (`true` or `false`). If `true`, the metadata provided will be included in the prompt to give context to the agent. ##### Session Continuation * **`session_continuation`**: Provide `session_id` from a previous session to enable continuity in conversation. This allows the agent to reference previous interactions. ### 4. Stop a Conversation: ```js theme={null} msClient.stop(); ``` ### 5. Setup event listener: ```js theme={null} msClient.on("onopen", () => { // When the client connected to the server }); msClient.on("onready", () => { // When the conversation is ready }); msClient.on("onaudio", (audio: Uint8Array) => { // Incoming audio chunk }); msClient.on("analyzer", (analyzer: AnalyserNode) => { // AnalyserNode that you can use for audio animation }); msClient.on("onclose", (event) => { // When the connection is closed }); msClient.on("onerror", (error) => { // An error occurred }); ``` ## Event List ### `onopen` **Description:** Emitted when the WebSocket connection is successfully opened. **Callback Signature:** ```typescript theme={null} 'onopen': () => void; ``` ### `onready` **Description:** Emitted when the client is ready to start processing audio or other tasks. **Callback Signature:** ```typescript theme={null} 'onready': () => void; ``` ### `onsessionended` **Description:** Emitted when a session has ended. **Callback Signature:** ```typescript theme={null} 'onsessionended': () => void; ``` ### `onaudio` **Description:** Emitted when audio data is received. **Callback Signature:** ```typescript theme={null} 'onaudio': (audio: Uint8Array) => void; ``` **Parameters:** * `audio` - The received audio data in `Uint8Array` format. ### `onresponsetext` **Description:** Emitted when agent's response text is received. **Callback Signature:** ```typescript theme={null} 'onresponsetext': (text: string, payload: { is_final?: boolean }) => void; ``` **Parameters:** * `text` - The received response text. * `payload` - An object containing additional information. * `is_final` (optional) - A boolean indicating if the response text is final. ### `ontranscript` **Description:** Emitted when user's transcript text is received. **Callback Signature:** ```typescript theme={null} 'ontranscript': (text: string, payload: { is_final?: boolean }) => void; ``` **Parameters:** * `text` - The received transcript text. * `payload` - An object containing additional information. * `is_final` (optional) - A boolean indicating if the transcript text is final. ### `onfunction` **Description:** Emitted when agent triggered a function call. **Callback Signature:** ```typescript theme={null} 'onfunction': (text: string, payload: { name: string, params: object }) => void; ``` **Parameters:** * `text` - Empty. * `payload` - Information about the triggered function. * `name` - The function name. * `params` - The params being used in the function call. ### `analyzer` **Description:** Emitted with an `AnalyserNode` for agent's audio analysis. **Callback Signature:** ```typescript theme={null} 'analyzer': (analyzer: AnalyserNode) => void; ``` **Parameters:** * `analyzer` - The `AnalyserNode` used for audio analysis. ### `useraudioready` **Description:** Emitted when user audio is ready for processing. **Callback Signature:** ```typescript theme={null} 'useraudioready': (data: { analyser: AnalyserNode, stream: MediaStream }) => void; ``` **Parameters:** * `data` - An object containing audio-related information. * `analyser` - The `AnalyserNode` for user's audio analysis. * `stream` - The `MediaStream` containing the user's audio data. ### `onlatency` **Description:** Emitted to report latency information for debugging purpose. **Callback Signature:** ```typescript theme={null} 'onlatency': (latency: number) => void; ``` **Parameters:** * `latency` - The measured latency in milliseconds. ### `onclose` **Description:** Emitted when the WebSocket connection is closed. **Callback Signature:** ```typescript theme={null} 'onclose': (event: CloseEvent) => void; ``` **Parameters:** * `event` - The `CloseEvent` containing details about the WebSocket closure. ### `onerror` **Description:** Emitted when an error occurs in the WebSocket connection. **Callback Signature:** ```typescript theme={null} 'onerror': (error: Event) => void; ``` **Parameters:** * `error` - The `Event` containing details about the error. *** ### Example Usage Here's an example of how to listen to these events in the `Client` class: ```typescript theme={null} const client = new Client(config); client.on('onopen', () => { console.log('WebSocket connection opened.'); }); client.on('onready', () => { console.log('Client is ready.'); }); client.on('onsessionended', () => { console.log('Session ended.'); }); client.on('onaudio', (audio) => { console.log('Audio received:', audio); }); client.on('onresponsetext', (text, payload) => { console.log('Response text:', text, 'Payload:', payload); }); client.on('ontranscript', (text, payload) => { console.log('Transcript:', text, 'Payload:', payload); }); client.on('analyzer', (analyzer) => { console.log('Analyzer node:', analyzer); }); client.on('useraudioready', (data) => { console.log('User audio ready:', data); }); client.on('onlatency', (latency) => { console.log('Latency:', latency); }); client.on('onclose', (event) => { console.log('WebSocket connection closed:', event); }); client.on('onerror', (error) => { console.error('WebSocket error:', error); }); ``` ## Support If you encounter any issues or have questions, please reach out to us directly at [thach@millis.ai](mailto:thach@millis.ai). # WebRTC Source: https://docs.millis.ai/integration/webrtc Connect audio sources directly to Millis agents via WebRTC ## Overview Millis AI supports WebRTC integration, allowing users to connect their audio sources directly to Millis agents via WebRTC. This integration is ideal for a variety of applications, including: * **Phone systems** with WebRTC capabilities * **Voice agents in video conferencing platforms** (e.g., Zoom, Google Meet) to interact with participants via voice * **VoIP systems** that leverage WebRTC for real-time communication Millis AI enables users to build intelligent voice agents that can join these platforms, engage in conversations, and assist participants via voice interactions. ## Requirements To connect your phone system, video conferencing platform, or VoIP solution to Millis AI via WebRTC, ensure that: 1. Your system supports **WebRTC** for audio transmission. 2. You have a valid **agent\_id** to route calls or streams to the correct Millis agent. 3. You have a **private key** for authenticating requests to Millis. ## API Endpoint To initiate a WebRTC session, your system sends a WebRTC **offer** to Millis through the following API endpoint: ### **POST /webrtc/offer** This API is used to send the WebRTC offer to Millis, where it will be processed, and a WebRTC **answer** will be returned to complete the connection. ### Request Headers: * **Authorization**: Bearer token containing the **private key** to authenticate the request. * **Content-Type**: `application/json` ### Request Body: The request body contains the following fields: | Field | Type | Description | | ---------- | ------ | -------------------------------------------------------- | | `agent_id` | String | The ID of the Millis agent to handle the call or stream. | | `offer` | Object | The WebRTC offer details, containing `sdp` and `type`. | #### `offer` Object: | Field | Type | Description | | ------ | ------ | ------------------------------------------------------ | | `sdp` | String | The WebRTC offer's session description protocol (SDP). | | `type` | String | Type of the WebRTC request (typically "offer"). | #### Example Request: ```json theme={null} { "agent_id": "123456", "offer": { "sdp": "v=0\no=- 2566784789543948235 2 IN IP4 127.0.0.1...", "type": "offer" } } ``` ### Example cURL Command: ```bash theme={null} curl -X POST https://api-west.millis.ai/webrtc/offer \ -H "Authorization: " \ -H "Content-Type: application/json" \ -d '{ "agent_id": "123456", "offer": { "sdp": "v=0\no=- 2566784789543948235 2 IN IP4 127.0.0.1...", "type": "offer" } }' ``` ### Response: If the offer is valid, the API will respond with a WebRTC **answer** that can be used to complete the connection between your system and Millis. #### Example Response: ```json theme={null} { "answer": { "sdp": "v=0\no=- 9876543210987654321 2 IN IP4 127.0.0.1...", "type": "answer" } } ``` ### Establish connection and start streaming: 1. **Set the Remote Description**: Use the `sdp` from the answer to set the remote description on your WebRTC client. * Example (JavaScript/WebRTC): ```javascript theme={null} peerConnection.setRemoteDescription(new RTCSessionDescription({ type: 'answer', sdp: 'v=0\no=- 9876543210987654321 2 IN IP4 127.0.0.1...' })); ``` 2. **Complete ICE Candidate Exchange**: Ensure that ICE candidates are exchanged between your client and Millis to establish the media path. 3. **Start Media Transmission**: After completing SDP and ICE negotiations, audio will start flowing between your system and the Millis agent. *** ## Use Cases ### 1. **Phone Systems with SIP and WebRTC** * Route incoming calls from DID numbers through Millis AI for real-time voice interaction, using WebRTC for media transmission. ### 2. **Voice Agents in Video Conferencing Tools** * Connect voice agents to video conferencing platforms such as Zoom, Google Meet, and others. The voice agent can join calls and engage in real-time audio conversations with participants, providing support, answering questions, or automating workflows. ### 3. **Connecting Agents to Virtual Rooms** * Millis agents can be connected to virtual communication rooms via WebRTC, interacting with users in the room to provide assistance, answer questions, or drive conversations through audio. # Introduction Source: https://docs.millis.ai/introduction Millis AI is an advanced voice AI platform that helps builders/developers quickly build low latency, natural-sounding voice agents at low cost. Hero Light Hero Dark Check out our API Reference here: [https://api-west.millis.ai/docs](https://api-west.millis.ai/docs) ## Key Features and Capabilities * **Low Latency Interaction**: Experience smooth, natural dialogues with groundbreaking 600ms latency, nearly matching the gold standard for conversational response times. * **Natural Conversation Flow**: Our voice agents are built to handle complex conversational dynamics, including interruptions and human intent recognition, ensuring realistic and fluid interactions. * **Easy Integration**: Integrate voice agents effortlessly into your projects with minimal coding required. Choose from our proprietary models or connect your custom LLM-based chatbot for rapid deployment. * **Scalable Infrastructure**: Benefit from the expertise of our DevOps engineers who have scaled systems to support hundreds of millions of video call minutes daily, guaranteeing a robust, enterprise-grade infrastructure. ## Language support English, Bulgarian, Catalan, Czech, Danish, Dutch, Estonian, Finnish, French, German, Greek, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Latvian, Lithuanian, Malay, Norwegian, Polish, Portuguese, Romanian, Russian, Slovak, Spanish, Swedish, Thai, Turkish, Ukrainian, Vietnamese. If you need help or have any questions, please [reach out to us](mailto:thach@millis.ai). # Import phone number Source: https://docs.millis.ai/phone/import-phone-number ## Basic Learn how to import your phone number from Twilio/Vonage/Plivo and connect it to Millis voice agent. #### Prerequisites * Navigate to [Millis AI Playground](https://app.millis.ai/phones). * Click "Import" button on the top right corner. * Select Region/Country and enter phone number in the format +12124567890. * Provide your account credentials. Check the tutorials below for more details. ## Providers ### Twilio