联犀开发者文档

人与AI实时语音

约 4029 字大约 13 分钟

2026-10-16

本页包含已接线核心流程及可选扩展；具体子能力必须按设备、平台与用户授权求交集。参见支持范围、场景索引和命令手册。图中S编号只用于本文定位，不放入实际JSON。

前置条件

平台选择 realtime，设备已声明 device AEC；本页展示UDP路径，WebRTC全双工及音视频关联另见S20。KWS/按键/本地 VAD 是设备触发策略，不替代云端端点判断。

图中P代表经鉴权的平台业务控制器和媒体节点，不要求同一进程；A是用户应用，D是设备。控制图描述一种合法顺序，跨通道不承诺严格同序。

已选udpReplyStartReceipt=true时，每个回复的Start至UDP发送之间复用S02安装确认子流程及其局部到期分支；它不暂停持续上行，也不改变插话语义。本页示例未声明该可选能力，仍展示未选时的路径；安装回执字段只见命令契约。

完整时序

报文索引

编号按图阅读；编号保留历史引用，不要求数值连续。字段定义以命令链接为准。

编号发送→接收Topic/接口方法及契约应答/等待状态或作用
S03-M01D→PSIG↑capabilityReportrequestAck；10秒全量替换本设备当前能力；断线重连重新上报，不授予权限。
S03-M02P→DSIG↓requestAck—；不等待同 revision 同内容返回原结果；同 revision 不同内容返回 409。
S03-M03D→PSIG↑sessionCreatesessionCreated；10秒设备发起会话；平台验证设备能力、业务配置、资源与授权交集，不指定供应商。
S03-M04P→DSIG↓sessionCreated—；不等待仅代表创建并分配媒体资源成功，不代表媒体连通；成功必须带顶层 sessionId。
S03-M05D→PSIG↑mediaStaterequestAck；10秒报告设备实际传输状态。WebRTC 需 ICE/DTLS 成功；UDP 需有效会话媒体探测且平台也收到有效包。
S03-M06P→DSIG↓requestAck—；不等待仅确认状态上报；平台须独立验证自身连通状态。
S03-M07P→DSIG↓sessionEstablished—；不等待平台确认所需参与端媒体已连通后通知；不等于已收到首包或扬声器起播。
S03-M08D→PSIG↑audioStartrequestAck；10秒串联每轮新 inputId；实时一次启动持续多轮。恢复暂停用新 inputId，不新建会话。
S03-M09P→DSIG↓requestAck—；不等待输入通道已可用，设备才上行；并不表示用户已经讲话。
S03-M10P→DMSG↓respCreated—；不等待为内容与音频事件建立关联，不能据此推断UDP已到达。
S03-M11P→DMSG↓respSttDone—；不等待最终识别替换此前增量；不等于回复文本。
S03-M12P→DMSG↓respTextDone—；不等待最终完整文本取代增量；不是播放完成。
S03-M13P→DSIG↓respAudioStart—；不等待平台开始发送该回复音频
S03-M15P→DSIG↓audioSpeechInterrupted—；不等待目标回复被打断；设备按 respId 清旧播放队列，持续上行不停止
S03-M31P→DMSG↓respDone应答或事件，无ACK核对补齐，不以受理替代完成
S03-M16P→DMSG↓respCreated—；不等待为内容与音频事件建立关联，不能据此推断UDP已到达。
S03-M17P→DMSG↓respSttDone—；不等待最终识别替换此前增量；不等于回复文本。
S03-M18P→DMSG↓respTextDone—；不等待最终完整文本取代增量；不是播放完成。
S03-M19P→DSIG↓respAudioStart—；不等待平台开始发送该回复音频
S03-M33P→DMSG↓respDone应答或事件，无ACK核对补齐，不以受理替代完成
S03-M21P→DSIG↓sessionError—；不等待带 code 的异步错误；不可恢复时随后 sessionEnded，可恢复错误不自动结束主会话。
S03-M22D→PSIG↑respCancelrequestAck；10秒本协议必须明确目标；只取消指定回复和清对应播放队列，不撤销已下发设备动作，也不停止持续收音。
S03-M23P→DSIG↓requestAck—；不等待取消已受理/目标已终结，不等于硬件停止发声；未知 respId 返回404。
S03-M24P→DSIG↓audioSpeechInterrupted—；不等待目标回复被打断；设备按 respId 清旧播放队列，持续上行不停止
S03-M32P→DMSG↓respDone应答或事件，无ACK核对补齐，不以受理替代完成
S03-M35P→DSIG↓respAudioDone—；不等待记录回复末包边界，等待尾包及播放设备排空；不是扬声器完成。
S03-M34P→DMSG↓respDone应答或事件，无ACK核对补齐，不以受理替代完成
S03-M25D→PSIG↑audioStoprequestAck；10秒串联提交该轮输入；实时只暂停，不把结束判断强加给供应商。重复请求不重复提交。
S03-M26P→DSIG↓requestAck—；不等待停止输入已接受；回复可以继续，串联下一轮需等待自身策略或取消旧回复。
S03-M27D→PSIG↑audioStartrequestAck；10秒串联每轮新 inputId；实时一次启动持续多轮。恢复暂停用新 inputId，不新建会话。
S03-M28P→DSIG↓requestAck—；不等待输入通道已可用，设备才上行；并不表示用户已经讲话。
S03-M29D→PSIG↑sessionEndsessionEnded；10秒设备主动结束整个会话，立即本地停采集/播放；平台撤销轨、处理器、密钥/令牌及资源占用。
S03-M30P→DSIG↓sessionEnded—；不等待主动结束用原 token 和 code；其他结束用独立事件 token 与 eventSeq。ended 是唯一会话终态。

逐条JSON

S03-M01 · capabilityReport

{
  "msgToken": "cap-S03",
  "method": "capabilityReport",
  "params": {
    "revision": 1,
    "purposes": [
      "aiConversation"
    ],
    "transports": [
      "udp"
    ],
    "supportedInteractionModes": [
      "fullDuplex"
    ],
    "aecMode": "device",
    "audio": {
      "format": "opus",
      "sampleRate": 16000,
      "channels": 1,
      "frameDuration": 20
    },
    "cameras": [],
    "features": [
      "audio"
    ]
  }
}

S03-M02 · requestAck

{
  "msgToken": "cap-S03",
  "method": "requestAck",
  "code": 200,
  "data": {
    "requestMethod": "capabilityReport",
    "result": {
      "acceptedRevision": 1
    }
  }
}

S03-M03 · sessionCreate

{
  "msgToken": "s03-36",
  "method": "sessionCreate",
  "params": {
    "purpose": "aiConversation",
    "transport": "udp",
    "tracks": [
      {
        "trackId": "mic-1",
        "kind": "audio",
        "sourceId": "mic",
        "direction": "sendonly",
        "codec": "opus",
        "state": "requested"
      },
      {
        "trackId": "speaker-1",
        "kind": "audio",
        "sourceId": "speaker",
        "direction": "recvonly",
        "codec": "opus",
        "state": "requested"
      }
    ],
    "supportedInteractionModes": [
      "fullDuplex"
    ],
    "audioParams": {
      "format": "opus",
      "sampleRate": 16000,
      "channels": 1,
      "frameDuration": 20
    }
  }
}

S03-M04 · sessionCreated

{
  "msgToken": "s03-36",
  "method": "sessionCreated",
  "sessionId": "s03-session",
  "code": 200,
  "data": {
    "state": "negotiating",
    "selected": {
      "transport": "udp",
      "interactionMode": "fullDuplex",
      "audioParams": {
        "format": "opus",
        "sampleRate": 16000,
        "channels": 1,
        "frameDuration": 20
      }
    },
    "tracks": [
      {
        "trackId": "mic-1",
        "kind": "audio",
        "sourceId": "mic",
        "direction": "sendonly",
        "codec": "opus",
        "state": "requested"
      },
      {
        "trackId": "speaker-1",
        "kind": "audio",
        "sourceId": "speaker",
        "direction": "recvonly",
        "codec": "opus",
        "state": "requested"
      }
    ],
    "media": {
      "udp": {
        "server": "<media-host>",
        "port": 0,
        "key": "<redacted-key>",
        "nonce": "<redacted-nonce>"
      }
    }
  }
}

S03-M05 · mediaState

{
  "msgToken": "s03-37",
  "method": "mediaState",
  "sessionId": "s03-session",
  "params": {
    "mediaRevision": 1,
    "state": "connected"
  }
}

S03-M06 · requestAck

{
  "msgToken": "s03-37",
  "method": "requestAck",
  "sessionId": "s03-session",
  "code": 200,
  "data": {
    "requestMethod": "mediaState",
    "result": {
      "mediaRevision": 1,
      "state": "connected"
    }
  }
}

S03-M07 · sessionEstablished

{
  "msgToken": "s03-38",
  "method": "sessionEstablished",
  "sessionId": "s03-session",
  "eventSeq": 1,
  "data": {
    "state": "established",
    "mediaRevision": 1
  }
}

S03-M08 · audioStart

{
  "msgToken": "s03-39",
  "method": "audioStart",
  "sessionId": "s03-session",
  "params": {
    "inputId": "continuous-1",
    "firstPacketSeq": 256
  }
}

S03-M09 · requestAck

{
  "msgToken": "s03-39",
  "method": "requestAck",
  "sessionId": "s03-session",
  "code": 200,
  "data": {
    "requestMethod": "audioStart",
    "result": {
      "inputId": "continuous-1"
    }
  }
}

S03-M10 · respCreated

{
  "msgToken": "s03-40",
  "method": "respCreated",
  "sessionId": "s03-session",
  "messageSeq": 1,
  "data": {
    "respId": "reply-old",
    "utteranceId": "utterance-old"
  }
}

S03-M11 · respSttDone

{
  "msgToken": "s03-41",
  "method": "respSttDone",
  "sessionId": "s03-session",
  "messageSeq": 2,
  "data": {
    "respId": "reply-old",
    "utteranceId": "utterance-old",
    "text": "你好"
  }
}

S03-M12 · respTextDone

{
  "msgToken": "s03-42",
  "method": "respTextDone",
  "sessionId": "s03-session",
  "messageSeq": 3,
  "data": {
    "respId": "reply-old",
    "contentIndex": 0,
    "text": "你好，有什么可以帮你？"
  }
}

S03-M13 · respAudioStart

{
  "msgToken": "s03-43",
  "method": "respAudioStart",
  "sessionId": "s03-session",
  "eventSeq": 2,
  "data": {
    "respId": "reply-old",
    "trackId": "speaker-1",
    "mediaRevision": 1,
    "udpNonce": "0000000000000001",
    "firstPacketSeq": 128
  }
}

S03-M15 · audioSpeechInterrupted

{
  "msgToken": "s03-45",
  "method": "audioSpeechInterrupted",
  "sessionId": "s03-session",
  "eventSeq": 4,
  "data": {
    "respId": "reply-old",
    "trackId": "speaker-1",
    "mediaRevision": 1
  }
}

S03-M31 · respDone

{
  "method": "respDone",
  "msgToken": "s03-audit-3",
  "sessionId": "s03-session",
  "messageSeq": 4,
  "data": {
    "respId": "reply-old",
    "utteranceId": "utterance-old",
    "status": "cancelled",
    "contentCount": 1,
    "audioState": "cancelled",
    "reason": "bargeIn"
  }
}

S03-M16 · respCreated

{
  "msgToken": "s03-46",
  "method": "respCreated",
  "sessionId": "s03-session",
  "messageSeq": 5,
  "data": {
    "respId": "reply-new",
    "utteranceId": "utterance-new"
  }
}

S03-M17 · respSttDone

{
  "msgToken": "s03-47",
  "method": "respSttDone",
  "sessionId": "s03-session",
  "messageSeq": 6,
  "data": {
    "respId": "reply-new",
    "utteranceId": "utterance-new",
    "text": "你好"
  }
}

S03-M18 · respTextDone

{
  "msgToken": "s03-48",
  "method": "respTextDone",
  "sessionId": "s03-session",
  "messageSeq": 7,
  "data": {
    "respId": "reply-new",
    "contentIndex": 0,
    "text": "你好，有什么可以帮你？"
  }
}

S03-M19 · respAudioStart

{
  "msgToken": "s03-49",
  "method": "respAudioStart",
  "sessionId": "s03-session",
  "eventSeq": 5,
  "data": {
    "respId": "reply-new",
    "trackId": "speaker-1",
    "mediaRevision": 1,
    "udpNonce": "0000000000000002",
    "firstPacketSeq": 12928
  }
}

S03-M33 · respDone

{
  "msgToken": "s03-final-59",
  "method": "respDone",
  "sessionId": "s03-session",
  "messageSeq": 8,
  "data": {
    "respId": "reply-new",
    "utteranceId": "utterance-new",
    "status": "failed",
    "reason": "upstreamUnavailable",
    "contentCount": 0,
    "audioState": "failed"
  }
}

S03-M21 · sessionError

{
  "msgToken": "s03-51",
  "method": "sessionError",
  "sessionId": "s03-session",
  "code": 502,
  "eventSeq": 7,
  "data": {
    "reason": "modelTimeout",
    "retryable": true,
    "scope": "response",
    "respId": "reply-new"
  }
}

S03-M22 · respCancel

{
  "msgToken": "s03-52",
  "method": "respCancel",
  "sessionId": "s03-session",
  "params": {
    "respId": "reply-new"
  }
}

S03-M23 · requestAck

{
  "msgToken": "s03-52",
  "method": "requestAck",
  "sessionId": "s03-session",
  "code": 200,
  "data": {
    "requestMethod": "respCancel",
    "result": {
      "respId": "reply-new",
      "state": "cancelled"
    }
  }
}

S03-M24 · audioSpeechInterrupted

{
  "msgToken": "s03-53",
  "method": "audioSpeechInterrupted",
  "sessionId": "s03-session",
  "eventSeq": 8,
  "data": {
    "respId": "reply-new",
    "trackId": "speaker-1",
    "mediaRevision": 1
  }
}

S03-M32 · respDone

{
  "method": "respDone",
  "msgToken": "s03-audit-4",
  "sessionId": "s03-session",
  "messageSeq": 9,
  "data": {
    "respId": "reply-new",
    "utteranceId": "utterance-new",
    "status": "cancelled",
    "contentCount": 1,
    "audioState": "cancelled",
    "reason": "userCancelled"
  }
}

S03-M35 · respAudioDone

{
  "msgToken": "s03-audio-finished",
  "method": "respAudioDone",
  "sessionId": "s03-session",
  "eventSeq": 7,
  "data": {
    "respId": "reply-new",
    "trackId": "speaker-1",
    "mediaRevision": 1,
    "lastPacketSeq": 25600
  }
}

S03-M34 · respDone

{
  "msgToken": "s03-final-61",
  "method": "respDone",
  "sessionId": "s03-session",
  "messageSeq": 10,
  "data": {
    "respId": "reply-new",
    "utteranceId": "utterance-new",
    "status": "completed",
    "contentCount": 1,
    "audioState": "sent"
  }
}

S03-M25 · audioStop

{
  "msgToken": "s03-54",
  "method": "audioStop",
  "sessionId": "s03-session",
  "params": {
    "inputId": "continuous-1",
    "lastPacketSeq": 12800
  }
}

S03-M26 · requestAck

{
  "msgToken": "s03-54",
  "method": "requestAck",
  "sessionId": "s03-session",
  "code": 200,
  "data": {
    "requestMethod": "audioStop",
    "result": {
      "inputId": "continuous-1"
    }
  }
}

S03-M27 · audioStart

{
  "msgToken": "s03-55",
  "method": "audioStart",
  "sessionId": "s03-session",
  "params": {
    "inputId": "continuous-2",
    "firstPacketSeq": 12928
  }
}

S03-M28 · requestAck

{
  "msgToken": "s03-55",
  "method": "requestAck",
  "sessionId": "s03-session",
  "code": 200,
  "data": {
    "requestMethod": "audioStart",
    "result": {
      "inputId": "continuous-2"
    }
  }
}

S03-M29 · sessionEnd

{
  "msgToken": "s03-56",
  "method": "sessionEnd",
  "sessionId": "s03-session",
  "params": {
    "reason": "userHangup"
  }
}

S03-M30 · sessionEnded

{
  "msgToken": "s03-56",
  "method": "sessionEnded",
  "sessionId": "s03-session",
  "code": 200,
  "data": {
    "state": "ended",
    "reason": "userHangup"
  }
}

媒体与本地动作

编号路径格式与约束
S03-R01D→PUDP：16字节头 + AES-CTR 加密的 Opus 静音探测帧；复用已分配通道；探测不送模型、不作为用户有效语音。端口和密钥为不可连接占位符。
S03-R02P→DUDP 原字节回显静音探测包；平台和设备双向验证网络通道，不播放也不送入模型；不是实际人声解码或 AEC 验收，不改变包头。
S03-R03D→PUDP Opus 持续上行；同一 inputId 可包含多轮语音；静音费用按实际 Provider 计费，不能承诺免费。
S03-R04P→DUDP：16字节头 + AES-CTR Opus；nonce 对应 S03-M13 的 udpNonce，使用会话密钥；包头没有 respId，关联由统一信令授权的每回复 nonce 建立。
S03-R05D→PUDP Opus 插话音频；新语音不等待下行播放结束。
S03-R06P→DUDP：16字节头 + AES-CTR Opus；nonce 对应 S03-M19 的 udpNonce，必须不同于旧回复；包序号按新通道128步长递增。

异常、状态与资源释放

以下JSON仅为报文正文；编号、方向、说明不发送。loop下一轮需分配新token/inputId；同一次请求重试必须沿用原token。字段以命令手册为准。

路由缩写均展开于公共约定。ACT/PROP超时以产品配置为准；内部工具以工具超时为准，不套用10秒设备统一信令超时。

旧回复终态即使晚于新回复开始，也只更新旧 respId，不清新回复。统一信令 UDP 使用每回复 nonce 与输入序号边界，在软件层拒绝跨回复迟到包；该保证不代表扬声器回灌、硬件 AEC 或现场听感已验收。暂停不关闭会话，结束才释放 Provider。取消不撤销已发送的设备 action。

能力不支持、无权限、设备拒绝、协商失败、断网、重复请求、过期会话及迟到媒体按S11共通失败流程和公共错误表处理。业务专属分支以本页为准，不因错误静默打开额外能力。

验证入口

检查本场景对应的支持范围，以及公共流程。示例可解析不等于网络、权限或硬件运行已验证。

更新日志

2026/10/16 16:04
查看所有更新日志
  • fe3a7-docs(AI与音视频): 按场景整理完整接入流程与命令契约于