# 使用 WebSocket 快速入门 Gemini Live API

Gemini Live API 允许与 Gemini 模型进行实时双向交互，支持音频、视频和文本输入以及本机音频输出。本指南介绍了如何使用原始 WebSocket 直接与 API 集成。

{% button
   href="https://aistudio.google.com/live"
   icon="mic"
   variant="primary" %}
尝试 Google AI Studio 中的 Live API
{% /button %}

{% button
   href="https://github.com/google-gemini/gemini-live-api-examples/tree/main/gemini-live-ephemeral-tokens-websocket"
   icon="code"
   variant="primary" %}
从 GitHub 克隆示例应用程序
{% /button %}

{% button href="/docs/coding-agents" icon="terminal" variant="primary" %}
使用编码智能体技巧
{% /button %}

## 概览  {% #overview %}

Gemini Live API 使用 WebSocket 进行实时通信。与使用 SDK 不同，此方法涉及直接管理 WebSocket 连接并以 API 定义的特定 JSON 格式发送/接收消息。

关键概念：

* **WebSocket 端点**：要连接的特定 URL。
* **消息格式**：所有通信均通过符合以下标准的 JSON 消息完成 [`BidiGenerateContentClientMessage`](/api-reference/live/bidi-generate-content#client-messages) 和 [`BidiGenerateContentServerMessage`](/api-reference/live#BidiGenerateContentServerMessage) 结构。
* **会话管理**：您负责维护 WebSocket 连接。

## 认证  {% #authentication %}

通过将您的 API 密钥作为查询参数包含在 WebSocket URL 中来处理身份验证。

端点格式为：

```none
wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=YOUR_API_KEY
```

更换 `YOUR_API_KEY` 与您实际的 API 密钥一起。

## 使用临时 token 进行身份验证  {% #authentication-with-ephemeral-tokens %}

如果您正在使用 [短暂的 token](/docs/live-api/ephemeral-tokens)，您需要连接到 `v1beta` 端点。临时 token 需要作为 `access_token` 查询参数。

临时密钥的端点格式为：

```none
wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContentConstrained?access_token={short-lived-token}
```

更换 `{short-lived-token}` 与实际的临时 token。

## 连接到 Live API  {% #connecting-to-the-live-api %}

要启动实时会话，请与经过身份验证的端点建立 WebSocket 连接。通过 WebSocket 发送的第一条消息必须是 [`BidiGenerateContentSetup`](/api-reference/live#BidiGenerateContentSetup) 含有 `config`。有关完整的配置选项，请参阅 [Live API - WebSocket API 参考](/api-reference/live).

{% tabs %}
{% tab label="Python" %}
```python
import asyncio
import websockets
import json

API_KEY = "YOUR_API_KEY"
MODEL_NAME = "gemini-3.8-live"
WS_URL = f"wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key={API_KEY}"

async def connect_and_configure():
    async with websockets.connect(WS_URL) as websocket:
        print("WebSocket Connected")

        # 1. Send the initial configuration
        setup_message = {
            "setup": {
                "model": f"models/{MODEL_NAME}",
                "responseModalities": ["AUDIO"],
                "systemInstruction": {
                    "parts": [{"text": "You are a helpful assistant."}]
                }
            }
        }
        await websocket.send(json.dumps(setup_message))
        print("Configuration sent")

        # Keep the session alive for further interactions
        await asyncio.sleep(3600) # Example: keep open for an hour

async def main():
    await connect_and_configure()

if __name__ == "__main__":
    asyncio.run(main())
```
{% /tab %}

{% tab label="JavaScript" %}
```js
const API_KEY = "YOUR_API_KEY";
const MODEL_NAME = "gemini-3.8-live";
const WS_URL = `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=${API_KEY}`;

const websocket = new WebSocket(WS_URL);

websocket.onopen = () => {
  console.log('WebSocket Connected');

  // 1. Send the initial configuration
  const setupMessage = {
    setup: {
      model: `models/${MODEL_NAME}`,
      responseModalities: ['AUDIO'],
      systemInstruction: {
        parts: [{ text: 'You are a helpful assistant.' }]
      }
    }
  };
  websocket.send(JSON.stringify(setupMessage));
  console.log('Configuration sent');
};

websocket.onmessage = (event) => {
  const response = JSON.parse(event.data);
  console.log('Received:', response);
  // Handle different types of responses here
};

websocket.onerror = (error) => {
  console.error('WebSocket Error:', error);
};

websocket.onclose = () => {
  console.log('WebSocket Closed');
};
```
{% /tab %}
{% /tabs %}

## 发送短信  {% #sending-text %}

要发送文本输入，请构造一个 [`BidiGenerateContentRealtimeInput`](/api-reference/live#BidiGenerateContentRealtimeInput) 消息与 `text` 场。

{% tabs %}
{% tab label="Python" %}
```python
# Inside the websocket context
async def send_text(websocket, text):
    text_message = {
        "realtimeInput": {
            "text": text
        }
    }
    await websocket.send(json.dumps(text_message))
    print(f"Sent text: {text}")

# Example usage: await send_text(websocket, "Hello, how are you?")
```
{% /tab %}

{% tab label="JavaScript" %}
```js
function sendTextMessage(text) {
  if (websocket.readyState === WebSocket.OPEN) {
    const textMessage = {
      realtimeInput: {
        text: text
      }
    };
    websocket.send(JSON.stringify(textMessage));
    console.log('Text message sent:', text);
  } else {
    console.warn('WebSocket not open.');
  }
}

// Example usage:
sendTextMessage("Hello, how are you?");
```
{% /tab %}
{% /tabs %}

## 发送音频  {% #sending-audio %}

音频需要作为原始 PCM 数据发送（原始 16 位 PCM 音频，16kHz，小端）。构造一个 [`BidiGenerateContentRealtimeInput`](/api-reference/live#BidiGenerateContentRealtimeInput) 带有音频数据的消息。这 `mimeType` 至关重要。

{% tabs %}
{% tab label="Python" %}
```python
# Inside the websocket context
async def send_audio_chunk(websocket, chunk_bytes):
    import base64
    encoded_data = base64.b64encode(chunk_bytes).decode('utf-8')
    audio_message = {
        "realtimeInput": {
            "audio": {
                "data": encoded_data,
                "mimeType": "audio/pcm;rate=16000"
            }
        }
    }
    await websocket.send(json.dumps(audio_message))
    # print("Sent audio chunk") # Avoid excessive logging

# Assuming 'chunk' is your raw PCM audio bytes
# await send_audio_chunk(websocket, chunk)
```
{% /tab %}

{% tab label="JavaScript" %}
```js
// Assuming 'chunk' is a Buffer of raw PCM audio
function sendAudioChunk(chunk) {
  if (websocket.readyState === WebSocket.OPEN) {
    const audioMessage = {
      realtimeInput: {
        audio: {
          data: chunk.toString('base64'),
          mimeType: 'audio/pcm;rate=16000'
        }
      }
    };
    websocket.send(JSON.stringify(audioMessage));
    // console.log('Sent audio chunk');
  }
}
// Example usage: sendAudioChunk(audioBuffer);
```
{% /tab %}
{% /tabs %}

有关如何从客户端设备（例如浏览器）获取音频的示例，请参阅上的端到端示例 [GitHub](https://github.com/google-gemini/gemini-live-api-examples/blob/main/gemini-live-ephemeral-tokens-websocket/frontend/mediaUtils.js#L38-L74).

## 发送视频  {% #sending-video %}

视频帧作为单独的图像（例如 JPEG 或 PNG）发送。与音频类似，使用 `realtimeInput` 与一个 `Blob`，指定正确的 `mimeType`.

{% tabs %}
{% tab label="Python" %}
```python
# Inside the websocket context
async def send_video_frame(websocket, frame_bytes, mime_type="image/jpeg"):
    import base64
    encoded_data = base64.b64encode(frame_bytes).decode('utf-8')
    video_message = {
        "realtimeInput": {
            "video": {
                "data": encoded_data,
                "mimeType": mime_type
            }
        }
    }
    await websocket.send(json.dumps(video_message))
    # print("Sent video frame")

# Assuming 'frame' is your JPEG-encoded image bytes
# await send_video_frame(websocket, frame)
```
{% /tab %}

{% tab label="JavaScript" %}
```js
// Assuming 'frame' is a Buffer of JPEG-encoded image data
function sendVideoFrame(frame, mimeType = 'image/jpeg') {
  if (websocket.readyState === WebSocket.OPEN) {
    const videoMessage = {
      realtimeInput: {
        video: {
          data: frame.toString('base64'),
          mimeType: mimeType
        }
      }
    };
    websocket.send(JSON.stringify(videoMessage));
    // console.log('Sent video frame');
  }
}
// Example usage: sendVideoFrame(jpegBuffer);
```
{% /tab %}
{% /tabs %}

有关如何从客户端设备（例如浏览器）获取视频的示例，请参阅上的端到端示例 [GitHub](https://github.com/google-gemini/gemini-live-api-examples/blob/main/gemini-live-ephemeral-tokens-websocket/frontend/mediaUtils.js#L185-L222).

## 接收回复  {% #receiving-responses %}

WebSocket 将发回 [`BidiGenerateContentServerMessage`](/api-reference/live#BidiGenerateContentServerMessage) 消息。您需要解析这些 JSON 消息并处理不同类型的内容。

{% tabs %}
{% tab label="Python" %}
```python
# Inside the websocket context, in a receive loop
async def receive_loop(websocket):
    async for message in websocket:
        response = json.loads(message)
        print("Received:", response)

        if "serverContent" in response:
            server_content = response["serverContent"]
            # Receiving Audio
            if "modelTurn" in server_content and "parts" in server_content["modelTurn"]:
                for part in server_content["modelTurn"]["parts"]:
                    if "inlineData" in part:
                        audio_data_b64 = part["inlineData"]["data"]
                        # Process or play the base64 encoded audio data
                        # audio_data = base64.b64decode(audio_data_b64)
                        print(f"Received audio data (base64 len: {len(audio_data_b64)})")

            # Receiving Text Transcriptions
            if "inputTranscription" in server_content:
                print(f"User: {server_content['inputTranscription']['text']}")
            if "outputTranscription" in server_content:
                print(f"Gemini: {server_content['outputTranscription']['text']}")

        # Handling Tool Calls
        if "toolCall" in response:
            await handle_tool_call(websocket, response["toolCall"])

# Example usage: await receive_loop(websocket)
```
{% /tab %}

{% tab label="JavaScript" %}
```js
websocket.onmessage = (event) => {
  const response = JSON.parse(event.data);
  console.log('Received:', response);

  if (response.serverContent) {
    const serverContent = response.serverContent;
    // Receiving Audio
    if (serverContent.modelTurn?.parts) {
      for (const part of serverContent.modelTurn.parts) {
        if (part.inlineData) {
          const audioData = part.inlineData.data; // Base64 encoded string
          // Process or play audioData
          console.log(`Received audio data (base64 len: ${audioData.length})`);
        }
      }
    }

    // Receiving Text Transcriptions
    if (serverContent.inputTranscription) {
      console.log('User:', serverContent.inputTranscription.text);
    }
    if (serverContent.outputTranscription) {
      console.log('Gemini:', serverContent.outputTranscription.text);
    }
  }

  // Handling Tool Calls
  if (response.toolCall) {
    handleToolCall(response.toolCall);
  }
};
```
{% /tab %}
{% /tabs %}

有关如何处理响应的示例，请参阅上的端到端示例 [GitHub](https://github.com/google-gemini/gemini-live-api-examples/blob/main/gemini-live-ephemeral-tokens-websocket/frontend/geminilive.js#L22-L75).

## 处理工具调用  {% #handling-tool-calls %}

当模型请求工具调用时， [`BidiGenerateContentServerMessage`](/api-reference/live#BidiGenerateContentServerMessage)
将包含一个 `toolCall` 场地。您必须在本地执行该函数并将结果发送回WebSocket使用 [`BidiGenerateContentToolResponse`](/api-reference/live#BidiGenerateContentToolResponse)
信息。

{% tabs %}
{% tab label="Python" %}
```python
# Placeholder for your tool function
def my_tool_function(args):
    print(f"Executing tool with args: {args}")
    # Implement your tool logic here
    return {"status": "success", "data": "some result"}

async def handle_tool_call(websocket, tool_call):
    function_responses = []
    for fc in tool_call["functionCalls"]:
        # 1. Execute the function locally
        try:
            result = my_tool_function(fc.get("args", {}))
            response_data = {"result": result}
        except Exception as e:
            print(f"Error executing tool {fc['name']}: {e}")
            response_data = {"error": str(e)}

        # 2. Prepare the response
        function_responses.append({
            "name": fc["name"],
            "id": fc["id"],
            "response": response_data
        })

    # 3. Send the tool response back to the session
    tool_response_message = {
        "toolResponse": {
            "functionResponses": function_responses
        }
    }
    await websocket.send(json.dumps(tool_response_message))
    print("Sent tool response")

# This function is called within the receive_loop when a toolCall is detected.
```
{% /tab %}

{% tab label="JavaScript" %}
```js
// Placeholder for your tool function
function myToolFunction(args) {
  console.log(`Executing tool with args:`, args);
  // Implement your tool logic here
  return { status: 'success', data: 'some result' };
}

function handleToolCall(toolCall) {
  const functionResponses = [];
  for (const fc of toolCall.functionCalls) {
    // 1. Execute the function locally
    let result;
    try {
      result = myToolFunction(fc.args || {});
    } catch (e) {
      console.error(`Error executing tool ${fc.name}:`, e);
      result = { error: e.message };
    }

    // 2. Prepare the response
    functionResponses.push({
      name: fc.name,
      id: fc.id,
      response: { result }
    });
  }

  // 3. Send the tool response back to the session
  if (websocket.readyState === WebSocket.OPEN) {
    const toolResponseMessage = {
      toolResponse: {
        functionResponses: functionResponses
      }
    };
    websocket.send(JSON.stringify(toolResponseMessage));
    console.log('Sent tool response');
  } else {
    console.warn('WebSocket not open to send tool response.');
  }
}
// This function is called within websocket.onmessage when a toolCall is detected.
```
{% /tab %}
{% /tabs %}

## 后续步骤  {% #whats-next %}

* 阅读完整的 Live API [功能](/docs/live-api/capabilities) 关键功能和配置指南；包括语音活动检测和本机音频功能。
* 阅读 [工具使用](/docs/live-api/tools) 指南了解如何将 Live API 与工具和函数调用集成。
* 阅读 [会话管理](/docs/live-api/session-management) 管理长时间运行的对话的指南。
* 阅读 [临时 token](/docs/live-api/ephemeral-tokens) 安全认证指南 [客户端到服务器](/docs/live-api#implementation-approach) 应用程序。
* 有关底层 WebSocket API 的更多信息，请参阅 [WebSocket API 参考](/api-reference/live).
