进阶 ai.google.dev 2026-10-15 02:43:21 · 36 阅读

第14章 Agentic vision:Gemini 模型视觉智能体应用指南

Gemini 4 Argon 现已推出,立即试用。

首页

Gemini API

文档

Agentic vision(智能体视觉)

Gemini Robotics ER 模型可以编写并执行 Python 代码,在回答前对图像进行处理和逻辑运算。本页介绍代码执行的几个示例:缩放裁剪式物体检测、仪表读数、容器液位测量、电路板标识读取以及图像标注。

要将这些示例适配到自己的场景,只需替换提示词和上传的图像文件即可。你还可以调整提示词中要求的 JSON schema 以匹配应用所需的输出结构,或添加 system_instruction 来强制输出格式和精度。

完整可运行的代码请参阅 Robotics cookbook。

思考等级(Thinking level)

你可以控制模型的思考等级,在延迟和准确性之间权衡。物体检测这类空间任务用低思考等级就能表现良好;而计数、重量估算这类复杂任务则受益于更高的思考等级。

下面的示例为一个复杂的计数任务将思考等级设为高:

Pythonfrom google import genai

client = genai.Client()

uploaded_file = client.files.upload(file="scene.jpeg")

interaction = client.interactions.create( model="gemini-robotics-er-2-preview", input=[ { "type": "image", "uri": uploaded_file.uri, "mime_type": uploaded_file.mime_type }, {"type": "text", "text": "Identify and count all objects on the table."} ], generation_config={ "thinking_level": "high" # Use "minimal" or "low" for faster spatial tasks } )

print(interaction.output_text)

详情参见 Thinking。

物体检测(缩放与裁剪)

下面的示例使用代码执行对图像进行缩放和裁剪,以便在检测物体并返回边界框时获得更清晰的视图。

Pythonfrom google import genai

client = genai.Client()

uploaded_file = client.files.upload(file="sorting.jpeg")

prompt = """ Return JSON in the format {label: val, y: val, x: val, y2: val, x2: val} for the compostable objects in this scene. Please Zoom and crop the image for a clearer view. Return an annotated image of the final result with the bounding boxes drawn on it to the API caller as a part of your process. """

interaction = client.interactions.create( model="gemini-robotics-er-2-preview", input=[ { "type": "image", "uri": uploaded_file.uri, "mime_type": uploaded_file.mime_type }, {"type": "text", "text": prompt} ], tools=[{"type": "code_execution"}] )

print(interaction.output_text)

模型输出类似如下 JSON 响应: [ {"label": "compostable", "y": 256, "x": 482, "y2": 295, "x2": 546}, {"label": "compostable", "y": 317, "x": 478, "y2": 350, "x2": 542}, {"label": "compostable", "y": 586, "x": 556, "y2": 668, "x2": 595}, {"label": "compostable", "y": 463, "x": 669, "y2": 511, "x2": 718}, {"label": "compostable", "y": 178, "x": 565, "y2": 250, "x2": 609} ]

下图展示了模型返回的边界框。

读取模拟仪表并进行逻辑运算

下面的示例演示如何让模型读取模拟仪表读数并进行时间计算。它使用系统指令强制输出 JSON。

Pythonfrom google import genai

client = genai.Client()

uploaded_file = client.files.upload(file="gauge.jpeg")

interaction = client.interactions.create( model="gemini-robotics-er-2-preview", system_instruction="Be precise. When JSON is requested, reply with ONLY that JSON (no preface, no code block).", input=[ { "type": "image", "uri": uploaded_file.uri, "mime_type": uploaded_file.mime_type }, {"type": "text", "text": """Read the current value from this gauge. Then, calculate how long it will take at the current rate for the value to reach maximum. Reply in JSON: {"current_value": val, "max_value": val, "time_to_max_minutes": val}"""} ], tools=[{"type": "code_execution"}] )

print(interaction.output_text)

测量容器内的液体

下面的示例演示如何使用代码执行来测量容器内的液位。

Pythonfrom google import genai

client = genai.Client()

uploaded_file = client.files.upload(file="fluid.jpeg")

interaction = client.interactions.create( model="gemini-robotics-er-2-preview", system_instruction="Be precise. When JSON is requested, reply with ONLY that JSON (no preface, no code block).", input=[ { "type": "image", "uri": uploaded_file.uri, "mime_type": uploaded_file.mime_type }, {"type": "text", "text": """Measure the amount of fluid in the container. Reply in JSON: {"fluid_level_ml": val, "container_capacity_ml": val, "percentage_full": val}"""} ], tools=[{"type": "code_execution"}] )

print(interaction.output_text)

读取电路板上的标识

下面的示例演示如何使用代码执行来读取电路板上的标识。

Pythonfrom google import genai

client = genai.Client()

uploaded_file = client.files.upload(file="circuit_board.jpeg")

interaction = client.interactions.create( model="gemini-robotics-er-2-preview", system_instruction="Be precise. When JSON is requested, reply with ONLY that JSON (no preface, no code block).", input=[ { "type": "image", "uri": uploaded_file.uri, "mime_type": uploaded_file.mime_type }, {"type": "text", "text": """Read all visible component labels and markings on this circuit board. Reply in JSON: {"components": [{"label": val, "location": [y, x]}]}"""} ], tools=[{"type": "code_execution"}] )

print(interaction.output_text)

图像标注

下面的示例演示如何使用代码执行来标注图像(例如为投放说明绘制箭头),并返回修改后的图像。

Pythonfrom google import genai

client = genai.Client()

# Load your image uploaded_file = client.files.upload(file="sorting.jpeg")

prompt = """ Look at this image and return it as an annotated version using arrows of different colors to represent which items should go in which bins for disposal. You must return the final image to the API caller. """

interaction = client.interactions.create( model="gemini-robotics-er-2-preview", input=[ { "type": "image", "uri": uploaded_file.uri, "mime_type": uploaded_file.mime_type }, {"type": "text", "text": prompt} ], tools=[{"type": "code_execution"}] )

print(interaction.output_text)

下面是一张示例输入图像。

模型输出类似如下: 标注后的图像展示了桌上物品的建议投放位置: - 绿色垃圾桶(堆肥/有机):青椒、红椒、葡萄和樱桃。 - 蓝色垃圾桶(回收):黄色压扁易拉罐和塑料容器。 - 黑色垃圾桶(垃圾):巧克力包装纸、Welch's 包装袋和白色纸巾。

后续内容

任务编排 —— 通过自定义机器人 API 完成长时程任务。 流式机器人控制 —— 实时双向流式传输(仅限 Gemini Robotics ER 2)。 视频理解 —— 时刻查找与进度分类(仅限 Gemini Robotics ER 2)。

评论 (0)