多模态输入与媒体表情回复

约 2457 字大约 8 分钟

2026-10-16

本页包含已接线核心流程及可选扩展；具体子能力必须按设备、平台与用户授权求交集。参见支持范围、场景索引和命令手册。图中S编号只用于本文定位，不放入实际JSON。

图中respMedia属于扩展输出示意，当前none会话不启用该分支；实际输出仅文字，表情事件不增加输出媒体模态。图片和文件输入仍须模型及引用授权。

前置条件

设备已认证；素材已经由合法文件上传服务取得授权引用。本页展示无实时媒体的AI会话，文字、图片、录音、视频片段、文件用同一contents结构。模型不支持的模态返回422，不因为设备能上传就认为模型能读。

设备附件授权（部分实现，端到端待验收）

上述“取得授权引用”使用新增设备附件专用入口，不等于旧通用上传接口。未声明对应能力的设备上传返回的文件地址不绑定原AI会话、连接代次、主人或模型读取授权；工作台附件绑定真实App、项目和权限指纹，与设备会话范围不同。两条原有路径都不能直接生成本页需要的contentRef，不能伪造AppID或以“同一主人”为由跨范围读取。

责任端	当前实现与剩余检查
Things／DMS	原主人、企业、启用／在线、连接及精确能力预检已实现；设备Basic鉴权HTTP Prepare／Complete已转接正式AISVR RPC，客户端不能自报主人或企业。该入口须显式开启并配置，旧上传接口不变
IM	活Owner及原SQL范围核验、最终附件模态交集和`inputSend`关联已实现；提交期间只读核验视图避免反向RPC互锁，不释放主控制门，不把焦点权限当文件权限
AISVR文件域	Prepare／Complete、私有签名PUT、MIME／长度／摘要、原会话／实际nonce绑定及上传后／消费前复核已实现；内部输入只传引用，实际读原件后生成模型内容。真实平台授权和实际模型消费仍待完整E2E
C++平台适配	可移植HttpPort／ContentUploader、桌面HTTPS适配及`device_text`图片／文件worker已有实现；MQTT仅传typed引用，不传原件URL／Base64或内部nonce。正式平台联调和浏览器附件入口仍待验收／接线

设备附件使用/api/v1/things/device/edge/content/prepare、/api/v1/things/device/edge/content/complete与限时签名PUT，不新增MQTT上传命令。HTTP设备密码鉴权加连接代次不等于证明原物理MQTT连接；如需后者，必须由原连接下行颁发短期上传票据，不能从历史SQL补造。重连、主人转移、会话结束、过期或撤权后，旧引用不能继续消费；已物化内容也必须在准入和实际模型调用前核权。重复请求须校验内容一致，MIME或摘要不符拒绝。私有OSS签名PUT只授予限时对象写入，不授予模型读取；撤权后Complete／Consume必须拒绝，但不承诺已签发URL即时失效，也不承诺抹去供应商此前已经接收的内容。

完整时序

报文索引

编号按图阅读；编号保留历史引用，不要求数值连续。字段定义以命令链接为准。

编号	发送→接收	Topic/接口	方法及契约	应答/等待	状态或作用
S21-M01	D→P	SIG↑	capabilityReport	见命令；10秒	按对应契约更新本调用状态
S21-M02	P→D	SIG↓	requestAck	见命令；事件无ACK	按对应契约更新本调用状态
S21-M03	D→P	SIG↑	sessionCreate	见命令；10秒	按对应契约更新本调用状态
S21-M04	P→D	SIG↓	sessionCreated	见命令；事件无ACK	按对应契约更新本调用状态
S21-M05	P→D	SIG↓	sessionEstablished	见命令；事件无ACK	按对应契约更新本调用状态
S21-M06	D→P	MSG↑	inputSend	requestAck；10秒	按对应契约更新本调用状态
S21-M07	P→D	MSG↓	requestAck	—；事件无ACK	按对应契约更新本调用状态
S21-M08	P→D	MSG↓	respCreated	见命令；事件无ACK	按对应契约更新本调用状态
S21-M09	P→D	MSG↓	respTextDone	见命令；事件无ACK	按对应契约更新本调用状态
S21-M10	P→D	MSG↓	respMedia	—；事件无ACK	按对应契约更新本调用状态
S21-M11	P→D	MSG↓	respEmotion	—；事件无ACK	按对应契约更新本调用状态
S21-M12	P→D	MSG↓	respDone	—；事件无ACK	按对应契约更新本调用状态
S21-M13	P→D	MSG↓	requestAck	—；事件无ACK	失败不产生新副作用
S21-M14	D→P	SIG↑	sessionEnd	见命令；10秒	按对应契约更新本调用状态
S21-M15	P→D	SIG↓	sessionEnded	见命令；事件无ACK	按对应契约更新本调用状态

逐条JSON

S21-M01 · capabilityReport

{
  "msgToken": "s21-1",
  "method": "capabilityReport",
  "params": {
    "revision": 1,
    "purposes": [
      "aiConversation"
    ],
    "transports": [
      "none"
    ],
    "features": [],
    "inputModalities": [
      "text",
      "image"
    ],
    "outputModalities": [
      "text"
    ]
  }
}

S21-M02 · requestAck

{
  "msgToken": "s21-1",
  "method": "requestAck",
  "code": 200,
  "data": {
    "requestMethod": "capabilityReport",
    "result": {
      "acceptedRevision": 1
    }
  }
}

S21-M03 · sessionCreate

{
  "msgToken": "s21-3",
  "method": "sessionCreate",
  "params": {
    "purpose": "aiConversation",
    "transport": "none",
    "tracks": [],
    "inputModalities": [
      "text",
      "image"
    ],
    "outputModalities": [
      "text"
    ]
  }
}

S21-M04 · sessionCreated

{
  "msgToken": "s21-3",
  "method": "sessionCreated",
  "sessionId": "s21-session",
  "code": 200,
  "data": {
    "state": "negotiating",
    "selected": {
      "transport": "none",
      "inputModalities": [
        "text",
        "image"
      ],
      "outputModalities": [
        "text"
      ]
    },
    "tracks": []
  }
}

S21-M05 · sessionEstablished

{
  "msgToken": "s21-5",
  "method": "sessionEstablished",
  "sessionId": "s21-session",
  "eventSeq": 1,
  "data": {
    "state": "established",
    "mediaRevision": 0
  }
}

S21-M06 · inputSend

{
  "msgToken": "s21-6",
  "method": "inputSend",
  "sessionId": "s21-session",
  "params": {
    "inputId": "input-1",
    "contents": [
      {
        "type": "text",
        "text": "描述这张图片"
      },
      {
        "type": "image",
        "contentRef": "<authorized-image-ref>",
        "mimeType": "image/jpeg"
      }
    ],
    "outputModalities": [
      "text"
    ]
  }
}

S21-M07 · requestAck

{
  "msgToken": "s21-6",
  "method": "requestAck",
  "sessionId": "s21-session",
  "code": 200,
  "data": {
    "requestMethod": "inputSend",
    "result": {
      "inputId": "input-1",
      "state": "accepted"
    }
  }
}

S21-M08 · respCreated

{
  "msgToken": "s21-8",
  "method": "respCreated",
  "sessionId": "s21-session",
  "messageSeq": 1,
  "data": {
    "respId": "reply-1",
    "inputId": "input-1"
  }
}

S21-M09 · respTextDone

{
  "msgToken": "s21-9",
  "method": "respTextDone",
  "sessionId": "s21-session",
  "messageSeq": 2,
  "data": {
    "respId": "reply-1",
    "contentIndex": 0,
    "text": "图片中有一只猫。"
  }
}

S21-M10 · respMedia

{
  "msgToken": "s21-10",
  "method": "respMedia",
  "sessionId": "s21-session",
  "messageSeq": 3,
  "data": {
    "respId": "reply-1",
    "contentIndex": 1,
    "mediaType": "image",
    "contentRef": "<authorized-result-ref>",
    "mimeType": "image/png"
  }
}

S21-M11 · respEmotion

{
  "msgToken": "s21-11",
  "method": "respEmotion",
  "sessionId": "s21-session",
  "messageSeq": 4,
  "data": {
    "respId": "reply-1",
    "emoji": "😊",
    "emotion": "happy"
  }
}

S21-M12 · respDone

{
  "msgToken": "s21-12",
  "method": "respDone",
  "sessionId": "s21-session",
  "messageSeq": 5,
  "data": {
    "respId": "reply-1",
    "inputId": "input-1",
    "status": "completed",
    "contentCount": 1,
    "audioState": "none"
  }
}

S21-M13 · requestAck

{
  "msgToken": "s21-6",
  "method": "requestAck",
  "sessionId": "s21-session",
  "code": 403,
  "data": {
    "requestMethod": "inputSend",
    "reason": "permissionDenied"
  }
}

S21-M14 · sessionEnd

{
  "msgToken": "s21-14",
  "method": "sessionEnd",
  "sessionId": "s21-session",
  "params": {
    "reason": "userStopped"
  }
}

S21-M15 · sessionEnded

{
  "msgToken": "s21-14",
  "method": "sessionEnded",
  "sessionId": "s21-session",
  "code": 200,
  "data": {
    "state": "ended",
    "reason": "userStopped"
  }
}

fe3a7-docs(AI与音视频): 按场景整理完整接入流程与命令契约于 2026/10/16

平台认知

基本概念

物模型

架构与开源

定价与更新

联犀协议

AI与音视频接入

协议参考

场景

第三方协议

网关接入

音视频

物联卡

组态大屏