今日速览
| 序号 | 标题 | 来源 | 日期 | 主题 | 推荐等级 |
|---|---|---|---|---|---|
| 1 | OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport | arXiv | 2026-10-08 | AI-Agent | 高 |
| 2 | Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict | arXiv | 2026-10-08 | AI-Agent | 高 |
重点论文与技术动态
1. OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport
- 来源：arXiv
- 日期：2026-10-08
- 作者/机构：Babak Barazandeh, Connor Swanson, Chinmay Kulkarni, Nikhil Mungel
- 主题标签：
AI-Agent,arXiv - 推荐等级：高
- 分类：cs.AI, cs.CL, cs.CY, cs.LG
一句话结论
提出流式监控机制 OnTrack，通过比对智能体步骤与依赖关系，以每步约1毫秒的低延迟预警或阻断失败轨迹，兼顾安全与低成本。
核心内容
- 克服护栏智能体高延迟及事后评估滞后的缺陷，实现毫秒级实时干预。
- 适配三种数据受限场景（完整参考、仅工具模式、仅生成日志），相应检测计划违规、循环、停滞或重复调用。
- 仅凭前8步识别失败轨迹优于内容相似度法（AUROC+0.057），结合中止策略可省约18%算力，拦截准确率达83%。
方法与数据
- 运用流式结构感知最优传输比对成功运行记录；使用 SWE-bench 轨迹数据评估。
价值判断
- 值得关注：毫秒级实时阻断，避免不可逆损害并实质性节省无效算力。
- 可复用点：按先验数据可得性分级设计监控能力的降级思路。
- 局限/待核查：核心算法细节摘要未明确；中止策略存在一定误判（6次拦截中1次误判）。
摘要
Agents are deployed in applications from trip planners and stock trading to IT incident triage. In most cases, LLM agents work autonomously with minimal rule-based safeguarding, leading to cost and safety issues from irreversible actions. Recent works resolve this either by using a safeguard agent to monitor behavior or evaluating logs post-hoc. The first adds cost and latency to every step; the second delivers its verdict after the run, when tokens are burned and damage is done. To overcome this, we propose OnTrack, a streaming monitoring mechanism that compares an agent's steps and dependencies against recorded successful runs to alert users or block the agent in about a millisecond per step. We study this problem in three regimes of decreasing access: full reference access (historical runs and tool schemas), intermediate access (only tool schemas), and no prior knowledge (only step logs as generated). Expectation of OnTrack's monitoring capabilities reduces as data access drops, ranging from plan violation detection to identifying loops, stalls, and repeated tool calls. Finally, we evaluate OnTrack using SWE-bench trajectories. Based on the first 8 steps, our method ranks failing trajectories below succeeding ones better than content similarity approaches (+0.057 AUROC). With an abort policy, we save about 18% of compute that would be burned on failing runs, where 83% of interrupted runs were actually heading to failure (5 out of 6 aborts were correct).2. Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict
- 来源：arXiv
- 日期：2026-10-08
- 作者/机构：Kaiser Sun, Bernal Jimenez Gutierrez, Hongjun Liu, Jingyu Zhang, Jie Gao, Mark Dredze
- 主题标签：
AI-Agent,arXiv - 推荐等级：高
- 分类：cs.AI, cs.CL
一句话结论
本文评估了LLM智能体在知识冲突下的“认知谦逊”，发现高任务准确率不等于高谦逊度，且提升后者常以牺牲准确率为代价。
核心内容
- 提出识别、解决、上报（ISE）三个轨迹维度，评估智能体识别、应对和传达不确定性的意愿。
- 智能体常在早期发现冲突但后期未能维持或解决，部分高准确率配置也未在错误终答中传达不确定性。
- 模型级干预虽可提升认知谦逊，但往往会降低任务准确率，表明认知谦逊源于基座模型、智能体框架与环境的交互。
方法与数据
- 针对参数知识与外部证据、或不同上下文源间的冲突，设受控与多步自然冲突两种场景（配无冲突对照）评估四个智能体。
- 具体数据集、被测智能体架构及模型干预细节摘要未明确。
价值判断
- 值得关注：超越单一成功率指标，揭示了多步执行中冲突处理的轨迹缺陷及准确率与谦逊度的权衡。
- 可复用点：ISE三维轨迹评估框架与双场景知识冲突对照设计。
- 局限/待核查：四个被测智能体的具体配置与模型级干预方法待核查。