图像理解
Gemini 模型从一开始就被构建为多模态,可解锁广泛的图像处理和计算机视觉任务,包括但不限于图像字幕、分类和视觉问答,而无需训练专门的 ML 模型。
除了一般的多式联运功能外,Gemini 型号还提供 提高准确性 对于特定的用例,例如 物体检测,通过额外的培训。
将图像传递给 Gemini #
您可以使用两种方法提供图像作为 Gemini 的输入:
传递内联图像数据:非常适合较小的文件(总请求大小小于 20MB,包括提示)。
使用文件 API 上传图像:推荐用于较大的文件或跨多个请求重复使用图像。
传递内联图像数据 #
您可以将请求中的内联图像数据传递给 generateContent。您可以以 Base64 编码字符串的形式提供图像数据,也可以直接读取本地文件(取决于语言)。
以下示例展示了如何从本地文件读取图像并将其传递给 generateContent API 进行处理。
from google import genai
from google.genai import types
with open('path/to/small-sample.jpg', 'rb') as f:
image_bytes = f.read()
client = genai.Client()
response = client.models.generate_content(
model='gemini-4-hydrogen',
contents=[
types.Part.from_bytes(
data=image_bytes,
mime_type='image/jpeg',
),
'Caption this image.'
]
)
print(response.text)
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const contents = [
{
inlineData: {
mimeType: "image/jpeg",
data: base64ImageFile,
},
},
{ text: "Caption this image." },
];
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: contents,
});
console.log(response.text);
bytes, _ := os.ReadFile("path/to/small-sample.jpg")
parts := []*genai.Part{
genai.NewPartFromBytes(bytes, "image/jpeg"),
genai.NewPartFromText("Caption this image."),
}
contents := []*genai.Content{
genai.NewContentFromParts(parts, genai.RoleUser),
}
result, _ := client.Models.GenerateContent(
ctx,
"gemini-4-hydrogen",
contents,
nil,
)
fmt.Println(result.Text())
IMG_PATH="/path/to/your/image1.jpg"
if [[ "$(base64 --version 2>&1)" = *"FreeBSD"* ]]; then
B64FLAGS="--input"
else
B64FLAGS="-w0"
fi
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-4-hydrogen:generateContent" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H 'Content-Type: application/json' \
-X POST \
-d '{
"contents": [{
"parts":[
{
"inline_data": {
"mime_type":"image/jpeg",
"data": "'"$(base64 $B64FLAGS $IMG_PATH)"'"
}
},
{"text": "Caption this image."},
]
}]
}' 2> /dev/null
您还可以从 URL 获取图像,将其转换为字节,然后将其传递给 generateContent 如以下示例所示。
from google import genai
from google.genai import types
import requests
image_path = "https://goo.gle/instrument-img"
image_bytes = requests.get(image_path).content
image = types.Part.from_bytes(
data=image_bytes, mime_type="image/jpeg"
)
client = genai.Client()
response = client.models.generate_content(
model="gemini-4-hydrogen",
contents=["What is this image?", image],
)
print(response.text)
import { GoogleGenAI } from "@google/genai";
async function main() {
const ai = new GoogleGenAI({});
const imageUrl = "https://goo.gle/instrument-img";
const response = await fetch(imageUrl);
const imageArrayBuffer = await response.arrayBuffer();
const base64ImageData = Buffer.from(imageArrayBuffer).toString('base64');
const result = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: [
{
inlineData: {
mimeType: 'image/jpeg',
data: base64ImageData,
},
},
{ text: "Caption this image." }
],
});
console.log(result.text);
}
await main();
package main
import (
"context"
"fmt"
"os"
"io"
"net/http"
"google.golang.org/genai"
)
func main() {
ctx := context.Background()
client, err := genai.NewClient(ctx, nil)
if err != nil {
log.Fatal(err)
}
// Download the image.
imageResp, _ := http.Get("https://goo.gle/instrument-img")
imageBytes, _ := io.ReadAll(imageResp.Body)
parts := []*genai.Part{
genai.NewPartFromBytes(imageBytes, "image/jpeg"),
genai.NewPartFromText("Caption this image."),
}
contents := []*genai.Content{
genai.NewContentFromParts(parts, genai.RoleUser),
}
result, _ := client.Models.GenerateContent(
ctx,
"gemini-4-hydrogen",
contents,
nil,
)
fmt.Println(result.Text())
}
IMG_URL="https://goo.gle/instrument-img"
MIME_TYPE=$(curl -sIL "$IMG_URL" | grep -i '^content-type:' | awk -F ': ' '{print $2}' | sed 's/\r$//' | head -n 1)
if [[ -z "$MIME_TYPE" || ! "$MIME_TYPE" == image/* ]]; then
MIME_TYPE="image/jpeg"
fi
# Check for macOS
if [[ "$(uname)" == "Darwin" ]]; then
IMAGE_B64=$(curl -sL "$IMG_URL" | base64 -b 0)
elif [[ "$(base64 --version 2>&1)" = *"FreeBSD"* ]]; then
IMAGE_B64=$(curl -sL "$IMG_URL" | base64)
else
IMAGE_B64=$(curl -sL "$IMG_URL" | base64 -w0)
fi
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-4-hydrogen:generateContent" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H 'Content-Type: application/json' \
-X POST \
-d '{
"contents": [{
"parts":[
{
"inline_data": {
"mime_type":"'"$MIME_TYPE"'",
"data": "'"$IMAGE_B64"'"
}
},
{"text": "Caption this image."}
]
}]
}' 2> /dev/null
使用文件 API 上传图像 #
对于大文件或能够重复使用相同的图像文件,请使用 Files API。以下代码上传图像文件,然后在调用中使用该文件 generateContent。请参阅 Files API指南 了解更多信息和示例。
from google import genai
client = genai.Client()
my_file = client.files.upload(file="path/to/sample.jpg")
response = client.models.generate_content(
model="gemini-4-hydrogen",
contents=[my_file, "Caption this image."],
)
print(response.text)
import {
GoogleGenAI,
createUserContent,
createPartFromUri,
} from "@google/genai";
const ai = new GoogleGenAI({});
async function main() {
const myfile = await ai.files.upload({
file: "path/to/sample.jpg",
config: { mimeType: "image/jpeg" },
});
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: createUserContent([
createPartFromUri(myfile.uri, myfile.mimeType),
"Caption this image.",
]),
});
console.log(response.text);
}
await main();
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const contents = [
{
inlineData: {
mimeType: "image/jpeg",
data: base64ImageFile,
},
},
{ text: "Caption this image." },
];
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: contents,
});
console.log(response.text);
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const contents = [
{
inlineData: {
mimeType: "image/jpeg",
data: base64ImageFile,
},
},
{ text: "Caption this image." },
];
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: contents,
});
console.log(response.text);
使用多个图像进行提示 #
您可以通过包含多个图像在单个提示中提供多个图像 Part 中的物体 contents 数组。这些可以是内联数据(本地文件或 URL)和文件 API 引用的混合。
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const contents = [
{
inlineData: {
mimeType: "image/jpeg",
data: base64ImageFile,
},
},
{ text: "Caption this image." },
];
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: contents,
});
console.log(response.text);
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const contents = [
{
inlineData: {
mimeType: "image/jpeg",
data: base64ImageFile,
},
},
{ text: "Caption this image." },
];
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: contents,
});
console.log(response.text);
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const contents = [
{
inlineData: {
mimeType: "image/jpeg",
data: base64ImageFile,
},
},
{ text: "Caption this image." },
];
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: contents,
});
console.log(response.text);
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const contents = [
{
inlineData: {
mimeType: "image/jpeg",
data: base64ImageFile,
},
},
{ text: "Caption this image." },
];
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: contents,
});
console.log(response.text);
物体检测 #
模型经过训练可以检测图像中的对象并获取其边界框坐标。相对于图像尺寸的坐标缩放为 [0, 1000]。您需要根据原始图像大小对这些坐标进行缩放。
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const contents = [
{
inlineData: {
mimeType: "image/jpeg",
data: base64ImageFile,
},
},
{ text: "Caption this image." },
];
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: contents,
});
console.log(response.text);
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const contents = [
{
inlineData: {
mimeType: "image/jpeg",
data: base64ImageFile,
},
},
{ text: "Caption this image." },
];
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: contents,
});
console.log(response.text);
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const contents = [
{
inlineData: {
mimeType: "image/jpeg",
data: base64ImageFile,
},
},
{ text: "Caption this image." },
];
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: contents,
});
console.log(response.text);
import { GoogleGenAI } from "@google/genai";
import * as fs from "node:fs";
const ai = new GoogleGenAI({});
const base64ImageFile = fs.readFileSync("path/to/small-sample.jpg", {
encoding: "base64",
});
const contents = [
{
inlineData: {
mimeType: "image/jpeg",
data: base64ImageFile,
},
},
{ text: "Caption this image." },
];
const response = await ai.models.generateContent({
model: "gemini-4-hydrogen",
contents: contents,
});
console.log(response.text);
有关更多示例,请查看以下笔记本 Gemini Cookbook:
支持的图像格式 #
Gemini 支持以下图像格式 MIME 类型:
- PNG-
image/png - JPEG-
image/jpeg - WEBP-
image/webp - HEIC-
image/heic - 海夫-
image/heif
要了解其他文件输入方法,请参阅 文件输入方式 指导。
功能 #
所有 Gemini 模型版本都是多模态的,可用于广泛的图像处理和计算机视觉任务,包括但不限于图像字幕、视觉问答、图像分类和对象检测。
Gemini 可以根据您的质量和性能要求减少使用专门的 ML 模型的需要。
最新的模型版本经过专门训练,除了通用功能之外,还提高了专门任务的准确性,例如增强型 物体检测.
限制和关键技术信息 #
文件限制 #
Gemini 型号每个请求最多支持 3,600 图像文件。
代币计算 #
- 258 token 如果两个尺寸 <= 384 像素。较大的图像被平铺为 768x768 像素图块,每个图块的成本为 258 token。
计算图块数量的粗略公式如下:
- 计算裁剪单元大小,大致为:floor(min(width, height) / 1.5)。
- 将每个尺寸除以作物单位大小并相乘即可得到图块的数量。
例如,对于尺寸为 960x540 的图像,裁剪单位大小为 360. 将每个尺寸除以 360,平铺数量为 3 * 2 = 6.
媒体分辨率 #
Gemini 3 引入了对多模态视觉处理的精细控制 media_resolution 参数。的 media_resolution 参数决定了 每个输入图像或视频帧分配的 token 的最大数量。 更高的分辨率可以提高模型读取精细文本或识别小细节的能力,但会增加 token 的使用率和延迟。
有关该参数及其如何影响 token 计算的更多详细信息,请参阅 媒体分辨率 指导。
提示和最佳实践 #
- 验证图像是否正确旋转。
- 使用清晰、不模糊的图像。
- 当使用带有文本的单个图像时,放置文本提示 之后 中的图像部分
contents数组。
后续步骤 #
本指南向您展示如何上传图像文件并从图像输入生成文本输出。要了解更多信息,请参阅以下资源: