Gemini 中文指南
获取 API 密钥 ↗

音频理解

↓ 下载 Markdown

Gemini 可以分析音频输入并生成文本响应。

python
from google import genai
import base64

client = genai.Client()

uploaded_file = client.files.upload(file="path/to/sample.mp3")

interaction = client.interactions.create(
    model="gemini-4-hydrogen",
    input=[
        {"type": "text", "text": "Describe this audio clip"},
        {
            "type": "audio",
            "uri": uploaded_file.uri,
            "mime_type": uploaded_file.mime_type
        }
    ]
)
print(interaction.output_text)

概览 #

Gemini 可以分析和理解音频输入并生成文本响应,从而实现以下用例:

  • 描述、总结或回答有关音频内容的问题
  • 转录和翻译(语音到文本)
  • 说话人分类(识别不同的说话人)
  • 语音和音乐中的情绪检测
  • 使用时间戳分析特定段

如需实时语音和视频交互,请参阅 Live API。对于支持实时转录的专用语音到文本模型,请使用 Google Cloud 语音转文本 API.

将语音转录为文本 #

此示例展示了如何使用时间戳、说话者分类和情绪检测来转录、翻译和总结语音 结构化输出.

python
from google import genai

client = genai.Client()

YOUTUBE_URL = "https://www.youtube.com/watch?v=ku-N-eS1lgM"

prompt = """
  Process the audio file and generate a detailed transcription.

  Requirements:

  1. Identify distinct speakers (e.g., Speaker 1, Speaker 2).
  2. Provide accurate timestamps for each segment (Format: MM:SS).
  3. Detect the primary language of each segment.
  4. If not English, provide the English translation.
  5. Identify the primary emotion: Happy, Sad, Angry, or Neutral.
  6. Provide a brief summary at the beginning.
"""

response_schema = {
    "type": "object",
    "properties": {
        "summary": {"type": "string"},
        "segments": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "speaker": {"type": "string"},
                    "timestamp": {"type": "string"},
                    "content": {"type": "string"},
                    "language": {"type": "string"},
                    "emotion": {
                        "type": "string",
                        "enum": ["happy", "sad", "angry", "neutral"]
                    }
                },
                "required": ["speaker", "timestamp", "content", "emotion"]
            }
        }
    },
    "required": ["summary", "segments"]
}

interaction = client.interactions.create(
    model="gemini-4-hydrogen",
    input=[
        {"type": "video", "uri": YOUTUBE_URL, "mime_type": "video/mp4"},
        {"type": "text", "text": prompt}
    ],
    response_format={
        "type": "text",
        "mime_type": "application/json",
        "schema": response_schema,
    },
)

print(interaction.output_text)

输入音频 #

您可以通过以下方式提供音频数据:

上传音频文件 #

使用 Files API 对于大于 20 MB 的文件。

python
from google import genai

client = genai.Client()

uploaded_file = client.files.upload(file="path/to/sample.mp3")

interaction = client.interactions.create(
    model="gemini-4-hydrogen",
    input=[
        {"type": "text", "text": "Describe this audio clip"},
        {
            "type": "audio",
            "uri": uploaded_file.uri,
            "mime_type": uploaded_file.mime_type
        }
    ]
)
print(interaction.output_text)

内嵌传递音频数据 #

对于总请求大小低于 20MB 的小音频文件:

python
from google import genai
import base64

client = genai.Client()

with open('path/to/small-sample.mp3', 'rb') as f:
    audio_bytes = f.read()

interaction = client.interactions.create(
    model="gemini-4-hydrogen",
    input=[
        {"type": "text", "text": "Describe this audio clip"},
        {
            "type": "audio",
            "data": base64.b64encode(audio_bytes).decode('utf-8'),
            "mime_type": "audio/mp3"
        }
    ]
)
print(interaction.output_text)

关于内嵌音频数据的注释:

  • 最大请求大小总计为 20 MB(包括提示和所有文件)
  • 为了重复使用, 上传文件 相反

获取成绩单 #

要获取成绩单,请在提示中询问:

python
interaction = client.interactions.create(
    model="gemini-4-hydrogen",
    input=[
        {"type": "text", "text": "Generate a transcript of the speech."},
        {
            "type": "audio",
            "uri": uploaded_file.uri,
            "mime_type": uploaded_file.mime_type
        }
    ]
)
print(interaction.output_text)

参考时间戳 #

使用 MM:SS 引用特定部分的格式:

python
interaction = client.interactions.create(
    model="gemini-4-hydrogen",
    input=[
        {"type": "text", "text": "Provide a transcript from 02:30 to 03:29."},
        {
            "type": "audio",
            "uri": uploaded_file.uri,
            "mime_type": uploaded_file.mime_type
        }
    ]
)

计数 token #

计算音频文件中的 token:

python
response = client.models.count_tokens(
    model="gemini-4-hydrogen",
    contents=[uploaded_file]
)
print(response)

支持的音频格式 #

Gemini 支持以下音频格式 MIME 类型:

  • 音频- audio/wav
  • MP3- audio/mp3
  • 亚洲国际电影节- audio/aiff
  • AA- audio/aac
  • 奥格- audio/ogg
  • FLAC- audio/flac
  • MPEG- audio/mpeg
  • M4A - audio/m4a
  • L16 - audio/l16
  • 作品- audio/opus
  • 法律- audio/alaw
  • 穆洛 - audio/mulaw
  • 网络M- audio/webm

有关支持的 MIME 类型和参数架构的完整列表,请参阅 Interactions API参考.

有关音频的技术细节 #

  • token: 32 token 每秒音频(1 分钟 = 1,920 token)
  • 非言语:Gemini 理解非语音(鸟鸣、警报声等)
  • 最大长度:9.5 每个提示的音频小时数
  • 分辨率:下采样至 16 Kbps
  • 渠道:多声道音频合并为单声道

后续步骤 #

搜索标题、API 名称或正文内容