Most engineering teams have gone through the same curve:
The root issue is rarely model quality. The issue is that engineering workflows were not redesigned for agent collaboration. Agentic Engineering is not about letting agents write more code. It is about converting the full delivery lifecycle—requirements, design, implementation, validation, release, and postmortems—into a repeatable pipeline where humans and agents can collaborate safely.
Part 1 focuses on one practical goal: turning “requirements to release” from personal craftsmanship into team-level operating procedure.
A lot of failures start with vague requests:
“Can we AI-enable this workflow quickly?”
For frontline engineers, the first move is not implementation. The first move is turning a request into verifiable constraints. A minimal requirement package should include:
Think of this as shared context between humans and agents. If your requirement is ambiguous, your agent will amplify ambiguity at speed.
A simple text template is enough:
1 | Goal: |
Traditional design docs often optimize for “this can work.” In Agentic Engineering, design docs should optimize for decision traceability and verification.
A practical design document has four layers:
Even a plain text flow is enough to lock decisions:
1 | Requirement clarification -> Design review -> Tests first (expected fail) -> Minimal implementation -> Verification -> Canary release -> Observability & postmortem |
The core idea: every phase must have an explicit entry criterion for the next phase.
Many teams worry that agent-generated changes drift in style and scope. Style guidelines alone won’t solve that. Gates will.
Use a fixed four-step implementation discipline:
Execution grammar can be expressed as:
1 | Baseline Gate -> TDD First Fail -> Minimal Change -> Full Verification x2 |
A useful principle for frontline teams:
Let agents accelerate execution. Keep requirement boundaries and release decisions human-owned.
Agentic Engineering discourages all-at-once releases.
A safer release model is experiment-driven:
A practical release checklist:
Fix: let agents generate and execute; keep decisions and acceptance criteria human-owned.
Fix: prioritize boundary tests—timeouts, retries, duplicate submissions, auth failures.
Fix: normalize vocabulary in requirement/design docs (task, run, replay, gate).
Fix: run at least one pre-release rollback drill.
Fix: always identify which guardrail failed and how to upgrade it.
The first principle of Agentic Engineering is simple:
stable delivery matters more than local speed gains.
Once your requirement-to-release pipeline is stable, you can safely pursue deeper automation.
In Part 2, we will focus on the hard production reality:
turning evals, regression discipline, cost controls, and safety checks into explicit release gates.
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>过去一年，很多团队都经历了同一个阶段：
问题不在于模型不够强，而在于工程流程没有跟上。Agentic Engineering 的核心并不是“让 Agent 自动写更多代码”，而是把需求、设计、实现、验证、发布、复盘这些老问题，用面向 Agent 协作的方式重新组织成可重复流水线。
这篇是连载第一篇，我们只解决一个问题：如何把“需求到发布”的过程从个人经验，变成团队可复制的工程通路。
很多失败都从一句模糊需求开始：
“把这个页面 AI 化一下。”
对于一线工程师来说，第一步不是开写，而是把需求翻译为可执行约束。最小集合建议包含 6 项：
你可以把它理解为“让 Agent 与人共享同一份任务上下文”。如果需求本身没有判定标准，Agent 只会把模糊性放大。
一个简单的文本模板就够用：
1 | 需求目标： |
在传统开发里，设计文档常常写“方案”；在 Agentic Engineering 里，设计文档更重要的是写“决策与证据”。
建议把设计拆成四层：
用一张纯文本流程图就能把关键决策固定下来：
1 | 需求澄清 -> 设计评审 -> 先写测试并确认失败 -> 最小实现 -> 自测与回归 -> 灰度发布 -> 观测复盘 |
关键点在于：每一步都必须有“下一步的入场条件”。例如，测试没先失败，不允许进入实现；覆盖率不达标，不允许进入发布。
很多人担心 Agent 参与后代码风格会漂。我的经验是：风格不是靠“提醒”，而是靠门禁。
开发阶段推荐固定四个动作：
对应的执行口径可以是：
1 | Baseline Gate -> TDD First Fail -> Minimal Change -> Full Verification x2 |
对一线工程师最实用的原则是：不要让 Agent 直接决定“做什么”，只让它加速“怎么做”。需求边界、风险判断、发布时机依旧由你主导。
Agentic Engineering 不鼓励“全量一次性上线”。
更稳妥的方法是把发布定义为实验流程：
一个可执行的发布清单示例：
正确做法：让 Agent 承担生成与执行，把决策和验收留在人类工程师手中。
正确做法：边界条件优先写测试，尤其是超时、重试、重复提交、权限失败。
正确做法：在需求与设计文档里固定术语，例如“任务”“运行”“回放”“门禁”。
正确做法：发布前先做一次“假故障回滚”演练。
正确做法：复盘必须回答“哪条门禁失效了，如何补门禁”。
Agentic Engineering 的第一性原理是：稳定交付优先于局部效率。先把需求到发布这条流水线打通，团队才有资格讨论更高阶的自动化。
下一篇（Part 2）我们会进入最关键的工程现实：如何把评估、回归、成本与安全变成上线闸门，而不是上线后的补救动作。
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>In agent-based systems, the most dangerous illusion is:
“The demo works, so we can ship.”
Demos prove possibility. Production requires repeatability under noisy conditions.
This article focuses on one practical objective:
combine evals, regression discipline, and risk controls into enforceable release gates.
Without a shared scorecard, each team declares success differently. The result is predictable: every team is “green,” while production quality remains unstable.
A practical scorecard for frontline teams has four dimensions:
Define hard red lines before optimization goals.
Example thresholds (illustrative):
1 | Correctness >= 90% |
Teams that rely on a single test layer usually miss important failures. A safer pattern is to run three layers together:
Validate local logic: prompt templates, retrieval quality, parameter mapping.
Validate multi-step collaboration: planning -> tool use -> synthesis -> final response.
Replay realistic requests against full dependencies to validate production behavior.
Practical role split:
The key difference in Agentic Engineering is not “more tests.” It is tests-first sequencing.
Recommended gate order:
1 | (1) Baseline green |
Why this works:
Many teams delay budget and safety checks to “later governance.” That is costly.
At minimum:
max_tokens, latency cap).At minimum:
For frontline engineers, the practical takeaway is simple:
convert non-negotiable risks into executable assertions.
1 | Release is allowed only if: |
Suggested role ownership:
Correction: include repeatability and variance metrics.
Correction: add cost delta table to release reports.
Correction: encode risky scenarios as mandatory regression tests.
Correction: visualize prevented incidents and rework to show value.
The quality core of Agentic Engineering is replacing pre-release luck with pre-release evidence.
In Part 3, we complete the loop:
how observability, rollback discipline, and postmortems determine long-term system stability.
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>在 Agent 项目里，最容易出现的错觉是：
本地演示通过了，说明可以上线。
现实往往相反：演示关注的是“能不能工作”，上线关注的是“在波动条件下能否持续工作”。
这篇我们解决的是第二个问题：如何把评估（Eval）、回归（Regression）、风险控制（Safety/Cost）整合成可执行上线闸门。
如果每个团队各自定义成功标准，最终会出现“都说自己通过了，但生产表现仍然不稳定”。
对一线工程师最实用的是四维评分板：
建议先定“红线阈值”，再谈优化目标。
示例（仅示意）：
1 | 正确性 >= 90% |
很多团队只做一种测试，导致覆盖盲区。更稳妥的方式是三层并行：
验证 Prompt 模板、检索召回、工具参数映射等局部逻辑。
验证多步骤协作：计划 -> 调工具 -> 汇总 -> 输出。
模拟真实请求，验证完整链路在真实依赖下是否稳定。
可执行原则：
Agentic Engineering 的关键不是“测试多”，而是“测试先”。
在上线相关开发中，建议固定门禁顺序：
1 | (1) Baseline 绿 |
这套顺序的意义在于：
很多团队把成本和安全放到“后续治理”。这是高风险做法。
至少要做三件事：
至少覆盖四类检查：
对一线工程师最重要的是：把“不可接受风险”写成可执行断言，而不是写成口号。
1 | 发布前必须满足： |
配套角色建议：
纠偏：引入“重复运行一致性”指标。
纠偏：把成本变化纳入发布对比表。
纠偏：把高风险场景做成固定回归用例。
纠偏：把门禁结果可视化，证明其在减少事故和返工。
Agentic Engineering 的质量核心，是把“上线前的侥幸”替换成“上线前的证据”。
下一篇（Part 3）我们进入最后一块：上线后的可观测、回滚与复盘机制，如何决定这套流程能否长期稳定运行。
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>Teams often over-focus on the release moment and under-invest in post-release operations. In production:
To close the loop in Agentic Engineering, three capabilities are non-negotiable:
observability, rollback readiness, and postmortem discipline.
A practical observability model has four layers:
If you only watch model metrics, you will misdiagnose many incidents. The model can return successfully while the tool path fails, producing business failure.
A minimal telemetry schema should include:
1 | trace_id / request_id / tenant / use_case / model / tool / latency / tokens / cost / error_class |
Alerting is not about noticing anomalies. It is about triggering predefined actions.
A practical 3-tier policy:
Predefine the action map:
1 | P1 -> rollback immediately + notify on-call + freeze new releases |
Many systems are “rollback-capable in theory” but fail in real incidents. Common reasons:
Run one small rollback drill each iteration:
Suggested SLO-style targets:
A high-quality postmortem answers five questions:
Use a mechanism-first structure:
1 | Timeline -> Decision points -> Guardrail failures -> Reinforcements -> Owner & due date |
A postmortem is only complete when conclusions are encoded back into gates, tests, and alert rules.
Agentic Engineering maturity is not measured by one expert’s skill. It is measured by whether a new engineer can deliver safely in two weeks.
Institutionalize three assets:
These assets reduce quality variance caused by staffing changes.
Across this three-part series:
That is the practical value of Agentic Engineering for frontline engineers:
not replacing human decisions, but turning those decisions into executable, testable, reusable system behavior.
If you are adopting this in your team, start with a minimal loop:
Once these are stable, expand to more agent-powered workflows with far lower risk.
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>很多团队把大量精力放在“上线那一刻”，却忽略了上线后的运行现实：
Agentic Engineering 的闭环必须覆盖三件事：可观测、可回滚、可复盘。没有这三项，前面的需求、设计、闸门都会慢慢失效。
建议把观测对象分成四层：
如果只看模型层指标，你会误判很多问题。例如模型成功返回了，但工具执行失败，业务结果仍然失败。
建议最小观测字段：
1 | trace_id / request_id / tenant / use_case / model / tool / latency / tokens / cost / error_class |
告警不是为了“发现有问题”，而是为了“触发动作”。
一线团队可先定义三档阈值：
对应动作必须提前写好：
1 | P1 -> 立即切回旧版本 + 通知值班 + 冻结新发布 |
很多系统“理论可回滚”，但故障时回不去。原因通常有三类：
建议每个迭代至少做一次小型演练：
并把目标时间写死，例如：
高质量复盘必须回答 5 个问题：
建议用“机制导向”结构写复盘：
1 | 事实时间线 -> 决策点 -> 失效门禁 -> 补强动作 -> 负责人与截止时间 |
只有当复盘结果沉淀进门禁、测试、告警规则，复盘才算完成。
Agentic Engineering 成熟度的关键标志，不是某个工程师多强，而是新同事加入后也能在两周内按流程稳定交付。
建议在团队内固定三类资产：
这三类资产会显著降低“人员变动导致质量波动”的风险。
回看三篇连载：
这就是 Agentic Engineering 对一线工程师真正有价值的地方：
不是替你做决策，而是把你的决策转化成可执行、可验证、可复用的工程系统。
如果你准备在团队里落地，建议从最小闭环开始：
当这四件事稳定后，再扩到更多 Agent 场景，成功率会高很多。
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>Most teams ship their first LLM feature the same way: a product service calls a provider SDK directly. It is the fastest path to a demo—and often the fastest path to production. The problem is what happens next, when you go from “one team experimenting” to “many teams, many products, many models, many tenants”.
The failure modes are surprisingly consistent:
At that stage, you do not need “yet another wrapper”. You need to treat model calls as a platform capability—governed the way mature teams govern APIs. That platform entry point is an AI Gateway.
Here is a minimal definition that stays practical:
Conceptually, it looks like this:
1 | Client / Service / Agent |
Without a contract, governance fragments. Each team invents its own fields, error handling, and retry logic. Then “platform policies” cannot be applied uniformly because there is no single place to apply them.
A pragmatic contract is less about “more fields” and more about clear semantics. At minimum, you want:
tenant_id / user_id / channel / use_case / data_classification trace_id / request_id so you can join product traces with gateway spans.[3] idempotency_key to prevent duplicate charges or duplicate downstream actions during retries. max_tokens / max_cost / max_latency_ms as first‑class constraints. model_family / quality_tier / region_preference so you can swap implementations without rewriting apps. redaction_profile / retention_policy / pii_mode Two properties matter most:
Below is a capability breakdown organised by governance goals. Use it as a design review checklist.
Classic API auth (keys, OAuth, mTLS) is still relevant—but LLM usage needs additional policy dimensions:
In practice, most teams converge to a policy mapping like: (tenant, app, use_case) -> policy.
LLM cost is not a fixed unit cost. It is a compound function of prompt length, output length, retries, retrieval context injection, and tool execution loops.
An AI Gateway should offer budget controls at three layers:
max_tokens / max_cost / max_latency_ms so a single call cannot blow up spend.The key difference from spreadsheets: budgets become enforced in seconds, not reviewed at month end.
Once you have multiple models (or the same model in multiple regions), routing is no longer optional:
quality_tier=high/standard/cheap; the gateway chooses the concrete model and parameters.This is also where you attach canaries and A/B tests: feed 1% traffic to a new model, watch cost/latency/error signals, then ramp.
Caching is one of the most effective levers to reduce cost, but it is also one of the easiest ways to create privacy and compliance problems.
Two common patterns:
Doing caching at the gateway has two advantages:
LLM failure modes go beyond basic HTTP errors: queueing, long‑tail latency, streaming interruptions, and content policy blocks are common.
A gateway should productise reliability patterns for LLM semantics:
Many of these primitives are mature in Envoy‑style proxy stacks; the gateway’s job is to apply them with LLM‑aware policies.[2]
In production, security is not just “does the output contain banned content”. It is a broader set of risks:
Most mature designs apply security in three layers:
When the system gets complex, observability becomes a survival requirement. One of the gateway’s biggest wins is that it turns LLM calls into first‑class telemetry:
model/provider/tenant/use_case, linked to the product trace.[3] The goal is not “more logs”. The goal is to answer questions quickly:
Aligning with OpenTelemetry helps avoid building a proprietary telemetry island.[3]
There is no single form factor. Two patterns are common:
Pros: consistent policy enforcement, unified routing/caching, single audit trail and cost dashboard.
Cons: it becomes a critical path component; you need high availability, capacity planning, and often regional front doors.
Pros: locality (lower cross‑network latency), natural fit with service meshes, smaller blast radius.
Cons: policy distribution/versioning becomes harder; unified caching and audit collection require extra work.
If you run on Kubernetes, the Gateway API is a useful abstraction for standardising gateway configuration across implementations, while Envoy‑class proxies provide the mature data plane.[1][2]
If you already have many services calling providers directly, a hard cutover is risky. A staged rollout is safer:
The principle is simple: build the evidence trail before you draw the red lines.
A common “scale pain” story looks like this:
With an AI Gateway, two small but hard moves usually unlock the rest:
tenant/use_case/trace_id, recorded in a single audit trail. Once those are in place, caching, routing, and fine‑grained permissions become incremental improvements rather than a full rewrite.
An AI Gateway does not make your model smarter. It makes your system more controllable, reliable, and scalable—because you introduce contracts, policies, telemetry, auditability, canaries, and rollbacks where they belong: at a single, governed entry point.
If you see any of these signals—multi‑team integration sprawl, unexplained cost drift, stricter compliance demands, hard‑to‑reproduce incidents—you are likely at the point where an AI Gateway is no longer “nice to have”.
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>很多团队做生成式能力落地，第一步几乎都是“业务服务直接调模型 SDK”。这条路径能快速上线，但一旦从「单个团队试点」走到「多团队、多产品、多模型、多租户」，你会发现问题开始以一种非常工程化的方式堆积：
这时你真正需要的，不是再写一套 SDK 封装，而是把 LLM 调用从“散落的代码片段”升级为平台级能力：像治理 API 一样治理模型调用。这个平台入口，就是本文要讲的 AI Gateway。
我对 AI Gateway 的“最小定义”是三句话：
你可以把它类比为“API Gateway + Service Mesh + FinOps + 审计”的组合，但对象从 HTTP API 变成了 LLM（以及 Agent 的工具调用链）。
一个典型调用链可以画成这样：
1 | Client / Service / Agent |
如果你没有统一契约，治理会很快碎片化：每个团队会在自己的 wrapper 里定义不同的字段、不同的错误处理、不同的重试策略。久而久之，“平台治理”无法落地，因为你连“在一个地方做策略”都做不到。
一个实用的契约至少要包含这些要素（不等于具体实现，只是对外的“口径”）：
tenant_id / user_id / channel / use_case / data_classification trace_id / span_id / request_id（便于端到端追踪）[3] idempotency_key（避免重试导致重复扣费/重复执行工具） max_tokens / max_cost / max_latency_ms（把预算变成一等公民） model_family / quality_tier / region_preference（而不是写死某个具体模型名） redaction_profile / retention_policy / pii_mode 这件事的关键不是“字段越多越好”，而是你要能做到两点：
下面我按“治理目标”把能力拆成 6 组，你可以用它做需求清单，也可以当作架构评审 checklist。
传统 API 里，鉴权常见是 API Key / OAuth / mTLS。到了 LLM 场景，你还需要更细的策略维度：
落地时，常见做法是在网关统一做身份解析与策略匹配：(tenant, app, use_case) -> policy。
LLM 的“成本”不像传统 API 那样稳定：prompt 长度、输出长度、重试次数、工具调用、检索结果注入都会影响 token 与费用。
AI Gateway 应该提供三种层级的预算控制：
max_tokens / max_cost / max_latency_ms，超出直接中断或降级。这样做的好处是：成本治理不再是“月底对账”，而是能在分钟级甚至秒级生效。
当你接入多个模型（或同一模型的多区域/多供应商），路由与降级会变成日常：
quality_tier=high/standard/cheap，网关再映射到具体模型与参数。更进一步，网关还可以承载灰度与 A/B：让新模型只吃 1% 流量，观察成本/延迟/正确性指标，再逐步扩大。
缓存是把成本打下来的“硬手段”，但也是最容易踩安全坑的地方。两类缓存常见：
网关层做缓存有两个优势：
务必注意：缓存与日志一样，是合规与隐私的高风险点。你需要明确保留策略与脱敏规则。[5]
LLM 调用的失败模式通常比普通 HTTP 更复杂：上游限流、排队、长尾延迟、流式中断、响应被过滤等。网关需要把这些能力“产品化”：
timeout；“交互式”优先低延迟，“离线总结”可以容忍更久。这类机制并不新，Envoy 等代理体系已把它们做成成熟能力；AI Gateway 要做的是把这些能力“贴合到 LLM 语义”上。[2]
当 LLM 走进生产，安全不再只是“输出有没有违规”，而是更广义的风险面：
AI Gateway 常见的做法是把安全能力分成三段：
如果你已经写过分布式系统，你会知道：当系统复杂度上来，“可观测性不是锦上添花，而是生存条件”。AI Gateway 的价值之一，是把 LLM 调用天然变成一个可观测节点：
model/provider/tenant/use_case 等维度，并能关联到上游业务 trace。[3] 你最终要达到的效果不是“有一堆日志”，而是能回答这些问题：
这类观测体系最好直接对齐 OpenTelemetry 语义与采集链路，避免自建孤岛。[3]
AI Gateway 并不只有一种形态，常见两类：
优点：策略集中、统一审计、统一缓存与路由、成本看板口径一致。
缺点：成为关键路径，需要高可用与容量规划；跨区域场景要考虑就近接入与数据驻留。
优点：就近处理、降低跨网延迟、与服务网格天然融合；局部故障影响面小。
缺点：策略与版本治理更复杂；统一缓存与统一审计更难做。
如果你在 Kubernetes 上，Gateway API 是一个值得关注的抽象：它让“网关能力”从具体实现（Ingress Controller）中抽离出来，用统一 CRD 描述路由与策略，再由实现（例如 Envoy）承载数据面。[1][2]
如果你已经有一堆业务在直连模型，直接“切网关”往往风险很高。我更推荐 4 个阶段的渐进路线：
这条路线的核心是：先建立证据链，再谈红线。没有可观测与审计，任何治理都会变成拍脑袋。
一个很典型的场景（已脱敏）：
引入 AI Gateway 后，团队先做了两件“小而硬”的事：
tenant/use_case/trace_id，并写入统一审计。 max_tokens 与租户级预算阈值，超限自动降级到更便宜的模型或返回短回复模板。有了这两点，后续再逐步叠加缓存、路由与更细的权限策略，就进入了“可运营”的轨道：成本、延迟与风险不再是黑箱，而是可以被度量与治理的对象。
AI Gateway 的本质不是“再加一层代理”，而是把模型调用这件事，按平台工程的方法论重新做一遍：契约、策略、观测、审计、灰度、回滚。
如果你的团队正在经历以下任一信号：
那么你大概率已经到了“需要 AI Gateway”的阶段。它不会让模型更聪明，但会让你的系统更可控、更可靠，也更能承受规模化。
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>Most teams start RAG (Retrieval‑Augmented Generation) with the shortest path to a demo:
This works—until it becomes a real product. In production, failures repeat in predictable ways, and they rarely come from the LLM being “not smart enough”. They come from treating retrieval as a single step instead of a governed system.
The most common failure modes look like this:
If you zoom out, the key lesson is simple: RAG is not “feeding docs to a model”. RAG is building a retrieval product—where data quality, security, and operational discipline matter as much as prompts.
When I say “RAG 2.0”, I mean one thing: treat retrieval as a pipeline you can measure, evolve, audit, and roll back.
The goal is not occasional brilliance. The goal is consistent behaviour under real constraints:
A practical pipeline mental model is:
1 | User question |
This structure stays stable even as you swap tools, models, or indexing strategies.
RAG 2.0 is less about “which vector store” and more about having a pipeline where each stage can be owned and improved independently.
Candidate retrieval works best when you treat it as multiple parallel channels:
Filtering is not a cosmetic step. It must guarantee two invariants:
In practice, “Filter” tends to include:
Even a good candidate set is noisy. Rerank and packaging should explicitly handle:
A minimal grounded system should:
query, index_version, doc_ids, scores, filter reasons),Hybrid retrieval is not a buzzword—it is acknowledging that “knowledge” has different shapes. A stable division of labour looks like this:
| Query pattern | Primary channel | Secondary channel | Why it works |
|---|---|---|---|
| IDs, error codes, proper nouns | BM25 | vectors | Precision first, semantics as a fallback.[1][2] |
| Paraphrases, “how do I…” | vectors | BM25 | Semantic recall first, lexical constraints second. |
| Hard facts (status/threshold/owner/time) | SQL | text | Don’t force the LLM to infer structured fields. |
| Relationship chains (deps/ownership/impact) | graph/CMDB | text | Relationships are more reliable than prose.[6] |
A useful rule: if the answer is a structured field, fetch it as a structured field. That is cheaper, more stable, and more auditable than prompting the model to “figure it out from text”.
Once users depend on RAG, freshness becomes your hardest SLA. The fix is to treat indexing as a versioned artifact.
At minimum, log:
index_version (build or incremental version),watermark (latest synced position from the source),source_updated_at (document last updated time).When an answer is wrong, you can now ask the right questions:
For system‑of‑record data in databases, CDC (Change Data Capture) is often the cleanest path to reliable incremental indexing. Debezium is a widely used implementation for capturing changes and driving downstream pipelines.[5]
You do not need CDC everywhere on day one, but you should design for:
Permissions are where RAG stops being “search” and becomes “production software”.
Three common approaches (from strongest isolation to most shared) are:
tenant_id/ACL tags and apply mandatory filters in the retrieval query (not after results return). Even when results are filtered, systems can leak side‑channels (counts, score distributions, error behaviour). Security guidance for LLM applications repeatedly emphasises designing with auditability and policy enforcement as first‑class concerns, not as patchwork.[7][8]
Knowledge graphs are not a silver bullet. They shine when the problem is relationships, not paragraphs:
Graphs are a poor fit when:
If you do add a graph, treat it as another evidence channel inside the same pipeline and citation framework. Neo4j docs are a solid starting point for graph modelling and querying.[6]
The output of RAG is not “a paragraph”. It is a user experience where people decide whether to rely on the system.
A good citation includes:
You do not need perfect probability calibration to improve safety. Simple heuristics work early:
If you want RAG 2.0 to become a reusable capability, ship it in stages:
doc_ids, scores, index_version). Here is a minimal acceptance checklist you can drop into a design review:
1 | ✅ RAG 2.0 Minimum Acceptance Checklist |
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>很多团队第一次做 RAG（Retrieval‑Augmented Generation）时，会走一条最短路径：
这条路能快速出 demo，但上生产后会反复踩同一类坑——不是模型不聪明，而是“检索与数据治理”没被当作系统来设计。最常见的失效点包括：
把这些问题合起来，你会发现：RAG 的核心挑战不是“把文档喂给模型”，而是“让知识在工程上可检索、可治理、可追责”。
我把“RAG 2.0”总结成一句话：检索不只是一个向量搜索，而是一条可观测、可演进、可审计的检索管道。它的目标不是“偶尔答得惊艳”，而是长期稳定地满足这四个指标：
一个实用的心智模型是把 RAG 拆成四步（每一步都可以被度量、被治理）：
1 | 用户问题 |
RAG 2.0 的关键不是“选哪个向量库”，而是把管道结构固定下来，让能力可以逐层演进。
建议把“候选召回”视作多路并行，典型组合是：
过滤不只是“把结果列表删一删”，而是要保证两件事：
实践上，你可以把“Filter”拆成三类规则：
候选集里“相关”不等于“可用”。重排阶段建议显式做三件事：
最低标准建议做到：
query、index_version、doc_ids、scores、过滤原因、最终上下文摘要；把“检索”拆成不同类型的证据通道，往往比在一个向量库里硬凹更稳。
下面是一个可落地的分工表（不是标准答案，但很少出错）：
| 问题类型 | 主通道 | 辅通道 | 备注 |
|---|---|---|---|
| 专有名词/编号/报错信息（如 E1234、INC‑20391） | BM25 | 向量 | 先命中精确 token，再用向量补同义。BM25 的工程实现成熟。[1][2] |
| 口语化/同义改写（“怎么把告警静音？”） | 向量 | BM25 | 向量负责语义，BM25 用于补关键字约束。 |
| 强结构化事实（状态、时间、阈值、负责人） | SQL | BM25/向量 | 结构化字段尽量别让模型“猜”。 |
| 关系链（“谁依赖谁/影响面”） | 图谱/CMDB | 文本检索 | 关系优先，文本补解释。 |
一个经验法则：能用结构化字段回答的问题，不要用大段文本去逼模型推断。 这不是“模型能力不行”，而是工程上更便宜、更稳定、更可审计。
RAG 走向生产后，新鲜度往往成为最硬的 SLA：知识更新慢一小时，客服/运维就会用脚投票。
建议为每次索引构建/增量更新生成一个可追踪的版本标识，例如：
index_version（时间戳或递增版本）watermark（数据源最新同步位点）source_updated_at（文档最后更新时间）当用户看到错误答案时，你至少能回答两个问题：
当知识来自数据库/业务系统时，CDC（Change Data Capture）通常是更稳的增量路径。Debezium 等工具提供了以日志方式捕获变更并驱动下游索引的工程方案。[5]
你不一定要“一开始就全量 CDC”，但至少要规划：
RAG 的权限问题经常被低估：很多团队把权限当作“检索后过滤”，但这在工程上并不等价于“安全”。
常见的可落地做法有三类（按隔离强度从强到弱）：
tenant_id/acl_tags，并在检索查询时做强制过滤（而不是拿到结果再过滤）。 即使最终结果被过滤，某些系统仍可能泄露侧信道信息（例如命中数、相似度分布、错误提示差异）。这也是为什么许多安全基线强调：把“策略与证据链”做成可审计、可复盘的系统工程，而不是零散的补丁。[7][8]
“上知识图谱”经常被当作 RAG 的银弹，但图谱真正擅长的是关系，而不是大段事实文本。它适合的典型场景：
图谱不适合的场景也很明确：
如果要引入图谱，建议把它当作“证据通道之一”，和 BM25/向量/SQL 一样走同一套管道与引用输出。Neo4j 文档可作为图数据建模与查询的权威入口。[6]
RAG 的终点不是“回答一段话”，而是“让用户敢用、敢依赖”。
建议引用至少包含：
你不需要一开始就做复杂的概率校准，也可以用工程启发式做第一版：
如果你要把 RAG 2.0 真正落到团队可复用的能力里，一个更现实的推进方式是“先稳再全”：
最后给一个面向测试/验收的最小 Checklist（可直接放进评审）：
1 | ✅ RAG 2.0 最小验收清单（建议） |
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>When teams move from a monolith to distributed systems, the biggest shift is not code size—it is loss of visibility. A single request fans out across services, queues, caches, and databases. Debugging by intuition stops working, so we lean on logs, metrics, and traces to answer three simple questions:
Agent systems amplify the “black box” problem. A user-visible “one response” may involve planning, retrieval, multiple tool calls, retries, safety filters, fallbacks, and even human handoffs. Without observability, you will repeatedly hit the same failure modes:
This is the first principle of AgentOps: treat an agent run like a production pipeline—each prompt render, retrieval step, tool call, and model inference should be traceable, aggregatable, alertable, and reviewable.
This series is planned as three parts:
Classic services observe HTTP latency, DB queries, cache hit rates, and queue depth. Agents need explicit telemetry for “reasoning-time” operations; otherwise you are stuck guessing the process from the final output.
Break an agent run into six observable domains:
Task / Session entry
Context assembly
Retrieval / RAG
Tools / Actions
Model inference
Policies / Guardrails
With this inventory, you can finally attribute “latency/cost/quality/risk” to concrete stages.
The most practical approach is to model an agent run as a distributed trace: one root span + many child spans. You do not need perfect standardisation on day one, but you must ensure:
trace_id, session_id) If you need a mental model, imagine a support agent that “looks fine” at the UI level but keeps producing expensive runs. Without traces, your team will argue about whether the model is “worse lately”. With traces, you can see the actual culprit: retrieval is returning low-confidence results, so the planner loops; a tool call is timing out and retrying; or a policy keeps downgrading to a smaller model and forcing extra turns.
One practical naming tip: treat a single end-to-end user interaction as a session, and a single attempt to complete the task as a run. Use stable identifiers:
session_id: groups multiple runs and user turns into a conversation-level unit.run_id: unique per attempt; useful when the same session retries or forks.trace_id: your distributed tracing glue; it should cover every step inside a run (including asynchronous tool executions).Example trace shape (plain text):
1 | agent.run (root) |
Two extra techniques make traces more useful in agent systems:
route.selected, guardrail.triggered) instead of hiding reasoning in free-form logs.To answer “what is slow/expensive/flaky”, each span type needs a minimal set of groupable attributes. Example (illustrative only):
1 | { |
The key is to treat cost and quality as first-class signals, not afterthoughts. Agents are “trade-off engines”, not pure compute.
Design attributes with care to avoid self-inflicted telemetry outages:
tenant_id, task_type, tool.name, model.name.prompt_version, policy_version, tool_schema_version. When an incident happens, “what changed?” is the fastest path to a fix.Observability is not “store everything forever”. Use tiering:
In practice, teams often use a three-tier storage model:
Align these rules with governance frameworks (for example, NIST AI RMF and the OWASP LLM Top 10).
Traces answer “why this run failed”. Metrics answer “what is trending and when to page someone”. For agent systems, start with four metric groups:
Latency and reliability
Cost and resources
Quality (start with weak supervision)
Risk and guardrails
One operational nuance: “success” for agents is multi-layered. A run can be technically successful (no exceptions) but still unacceptable (wrong answer, wrong action, unsafe output). Many teams separate SLIs:
Once you define SLIs, you can adopt classic SRE practices like error budgets and burn rates: treat “bad answers” and “budget violations” as first-class budget consumers, not merely “ML quirks”.
Once metrics and traces share the same dimensions, you can answer: “Is cost rising because tools are retrying, or because retrieval quality dropped and the agent is looping?”
Treat agent logs as two layers:
Decision logs should be structured and bound to trace_id. A practical minimal schema:
This turns postmortems from “opinions” into “facts”.
Two guardrails that matter in real organisations:
If you are starting from zero, ship an MVP before building an “agent platform”. A one-week MVP can achieve:
Acceptance checklist:
agent.run records tenant_id, task_type, final status (success/failed/downgraded/handoff)tool.call has tool.name, latency, status_code, retry.count (payloads redacted)model.infer has model.name, tokens in/out, latency, cost (if available)retrieval.search includes index_version, top_k, hit count, filtered count (auth)Common pitfalls to avoid in the first week:
trace_id but lose it at the first async boundary (queue, background worker, tool runner).This post introduced a practical inventory plus a unified Trace/Metrics/Logs model, with one goal: turn agent runs into explainable pipelines.
In Part 2, I will make this concrete: how to instrument an agent workflow with an OpenTelemetry mindset, build dashboards and alerts without creating a “platform tax”, and implement safe replay patterns.
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>当你把一个应用从“单体服务”演进到“分布式系统”，最大的变化往往不是代码量，而是系统变成了黑箱：一次请求跨越多个服务、多个队列、多个数据库，靠直觉排查很快失效，于是你需要日志、指标、追踪，来回答三个最朴素的问题：
Agent（编排 + 工具调用 + 记忆 + 检索 + 模型推理）把黑箱进一步放大：同样是“生成一段回复”，背后可能发生了多轮规划、检索、调用、重试、回退、降级。没有可观测性，你会在以下场景反复踩坑：
因此，AgentOps 的第一性原理是：把一次 Agent 任务还原成“可追踪的流水线”，让每一次 Prompt、每一次检索、每一次工具调用、每一次模型推理，都能像分布式链路一样被观测、聚合、告警、复盘。
本系列计划三篇：
传统服务最常见的观测对象是：HTTP 请求、数据库查询、缓存命中、队列积压。Agent 系统需要把“推理相关”的对象显式建模，否则你会永远停留在“看结果猜过程”的状态。
建议把 Agent 任务拆成 6 类观测对象（从用户视角到平台视角逐层展开）：
入口请求（Task / Session）
上下文构建（Context Assembly）
检索与知识（Retrieval / RAG）
工具调用（Tools / Actions）
模型推理（Model Inference）
策略与护栏（Policy / Guardrails）
把这 6 类对象完整串起来，你才能真正回答：“一次任务的成本、延迟、质量、风险，分别由哪个环节贡献”。
最实用的做法是把 Agent 任务视作一次分布式调用链：一个 root span + 多个子 span。你不需要追求一开始就“完全标准化”，但要保证三件事：
下面是一个典型的 Trace 结构（纯文本示意）：
1 | agent.run (root) |
为了后续能统计“谁最慢、谁最贵、谁最不稳定”，每类 span 至少要有一组可聚合字段。示例（仅示意，字段名可按你团队规范调整）：
1 | { |
关键点：把“成本与质量”也当作一等公民。Agent 不是纯计算服务，它是“质量/成本/风险”的权衡机器；只看延迟会误导你。
可观测性不是“把一切都记录下来”。建议给出明确的分级策略：
这类策略建议与安全团队对齐，并结合风险框架执行（可参考 NIST AI RMF 与 OWASP LLM Top 10 的治理思路）。
追踪解决“单次请求的因果链”，指标解决“整体趋势与告警”。对 Agent 系统来说，建议至少落地四组核心指标：
延迟与可靠性
成本与资源
质量（可以从“弱监督”开始）
风险与护栏
把这些指标和 trace 的维度字段打通，你就能回答：“某租户的成本上升，是因为工具失败重试，还是因为 RAG 命中变差导致多轮推理？”
Agent 系统的日志最好分成两层：
强烈建议给“决策日志”一个结构化格式，并与 trace_id 绑定。一个可落地的最小字段集合：
有了这些字段，你的复盘会从“讨论观点”变成“对齐事实”。
如果你要从 0 到 1 落地，建议先做 MVP，而不是一口气建“全功能 Agent 平台”。一周内可以完成的 MVP 目标是：
Checklist（像测试一样可验收）：
本文把 Agent 可观测性拆成了“对象清单 + Trace/Metric/Log 的统一模型”，核心目标只有一个：让 Agent 任务从黑箱变成可解释的流水线。
下一篇（落地篇）我会把这套模型进一步具体化：如何在不引入过多平台复杂度的前提下，用 OpenTelemetry 的思路把埋点、看板、告警、回放跑起来，并给出一套“从 0 到可用”的工程步骤与常见坑清单。
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>AI assistants are no longer “just writing code”. The useful ones can fetch metrics, pull logs, open tickets, draft pull requests, and even propose rollout plans. Once an agent can act, the ceiling is no longer your model’s intelligence—it is whether the agent can reliably and safely connect to your tools and data.
In practice, that “tool layer” is where teams get stuck:
Model Context Protocol (MCP) reframes this layer as a protocol and a contract. MCP lets an AI client (agent/IDE/CLI) discover and call capabilities in a consistent way, while the MCP server—running inside your boundary—owns authentication, policy, and audit trails [1]. If you have seen Context7 used to bring documentation context into workflows, that is an example of an MCP server in the wild [2].
This post focuses on the engineering questions that matter:
MCP is a standard protocol between AI clients and external tools/data, designed to decouple “agent decisions” from “tool execution” so multiple clients can reuse the same tool surface while governance stays centralised on the server side [1].
A useful analogy is “USB‑C for tool access”:
Tool calling exists everywhere today—function calling, “tools”, plugins—but each ecosystem has its own framing. If your team supports multiple surfaces (IDE + CLI + web assistant + batch agents), you either accept duplicated glue code or you standardise.
MCP standardises the tool interface: write one MCP server, reuse it across MCP‑capable clients.
Once tool calls are sprinkled across clients, it becomes painful to answer basic questions:
With MCP, these guardrails fit naturally on the server side, where execution happens.
Tools evolve: schemas change, upstream APIs deprecate, permissions tighten. When clients integrate directly, every change ripples outward. MCP acts like a contract layer: versioning and compatibility strategies can live in the MCP server instead of forcing every client to keep up.
You do not need to memorise specs—this table is enough for day‑to‑day design [1]:
| Concept | Practical meaning | Examples |
|---|---|---|
| MCP Client | the caller side | IDE extension, CLI agent, desktop assistant |
| MCP Server | the governed execution side | docs search, ticketing, monitoring, repo tools |
| Tools | executable actions with arguments | search_docs(query), create_ticket(title, body) |
| Resources | readable objects (like files/docs) | runbook://incident/123, repo://.../README |
| Prompts | reusable task templates | incident triage template, change review checklist |
An underrated detail is that not everything must be a “function”. For knowledge integration, Resources + Prompts often produce a cleaner contract than a single overloaded tool.
Here is a simplified sequence:
sequenceDiagram autonumber participant U as User participant C as MCP Client (Agent) participant S as MCP Server (Tool layer) participant B as Business system / API U->>C: Diagnose this alert and propose next steps C->>S: list_tools / list_resources S-->>C: Available tools/resources (with schemas) C->>S: call_tool(get_alert_detail, {id}) S->>B: Read alert details using scoped credentials B-->>S: Structured data / error codes S-->>C: Redacted + bounded result C-->>U: Answer with evidence + suggested actionsThe important takeaway: the MCP server is where “execution” and “governance” converge. Clients decide what to do; servers decide what is allowed and how it is done safely.
If you treat MCP as “just another RPC”, you will recreate the same safety problems. Treat it as a tool gateway and the design becomes clearer.
A pragmatic rule: many small tools beat one universal tool.
Good patterns:
rollout_restart(namespace, deployment)), instead of passing through raw kubectl;plan_change() → human approval → apply_change();Risky patterns:
Do not place long‑lived privileged credentials into model context. Safer defaults:
MCP does not magically solve auth, but it gives you a single layer to implement it consistently.
Rollouts succeed when three controls ship together:
At minimum you should be able to trace:
Treat tool calls like external dependencies and align logs/metrics/traces with your observability stack. OpenTelemetry provides a useful baseline for unified instrumentation [4].
Here is a lightweight go‑live checklist:
1 | ✅ MCP Tool Go‑Live Checklist (minimum) |
A safe adoption path is incremental:
This approach forces you to build the governance foundation early, before an agent gains the ability to press production buttons.
Two additional practices make rollouts smoother:
MCP is not free. You might skip it when:
MCP matters because it pulls tool access out of proprietary model features and into a governed engineering layer. As soon as agents can touch systems, data, and actions, the most valuable investment is usually not more prompt tricks—it is the contract: controlled, auditable, and evolvable.
Start with one read‑only scenario. Wrap a high‑leverage internal capability (runbooks or metrics queries) into an MCP server, ship audit logs and bounds, and expand only after the team agrees on safety boundaries.
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>过去一年，AI 助手的能力从“写段代码/改个函数”，走向“能调用工具把事情做完”：查指标、拉日志、改配置、生成 PR、写发布说明……真正决定体验上限的，不再是模型会不会写，而是它能不能稳定、可控地接入你的工具与数据。
但现实里，“工具接入”往往是最脏、最容易失控的那层：
Model Context Protocol（MCP）尝试把这层从“胶水代码”升级为“协议 + 契约”：让 AI 客户端（Agent/IDE/CLI）用一致方式发现与调用工具，让工具提供方（MCP Server）在你的边界内承接鉴权、治理与审计。你可能已经在一些工具里见过 MCP 的影子，例如把文档上下文接入到工作流的 Context7（一个 MCP Server）[2]。
本文不打算背定义，而是回答三个更实用的问题：
一句话：MCP 是 AI 客户端与外部工具/数据之间的标准通信协议，核心目的是把“模型能力”与“工具能力”解耦，让多种客户端用一致方式接入同一批工具，同时把安全治理集中在工具侧实现[1]。
你可以把它类比成“工具层的 USB-C”：
不同平台的工具调用（function calling / tools / plugins）都有自己的细节：参数 schema、消息格式、流式返回、错误结构……当你需要同时服务“IDE + CLI + Web 助手 + 批处理代理”时，集成成本会按客户端数线性增长。
MCP 的价值是把“工具协议”抽出来：你写一次 MCP Server，就能被多个 MCP Client 复用，避免每个平台各写一套。
如果工具调用逻辑散落在不同客户端里，你很难统一回答这些问题：
把工具封装到 MCP Server 后，这些能力更适合在“工具侧”落地：集中治理、集中审计。
工具的 schema、返回结构、权限策略一旦变化，如果客户端直连，很容易出现“客户端升级跟不上、线上能力碎片化”。MCP 更像一个契约层：工具侧可以用版本化、兼容策略、灰度发布去承接演进压力。
从落地角度，不必纠结术语，抓住这张“心智模型表”就够了[1]：
| 概念 | 你可以把它理解为 | 典型例子 |
|---|---|---|
| MCP Client | 发起“发现/调用”的一端 | IDE 插件、CLI Agent、桌面助手 |
| MCP Server | 把能力封装成协议的一端 | 文档检索、工单系统、监控平台、代码仓库工具 |
| Tools | 可执行动作（有输入参数） | search_docs(query)、create_ticket(title, body) |
| Resources | 可读取资源（像文件/文档/对象） | runbook://incident/123、repo://.../README |
| Prompts | 可复用的提示模板/任务配方 | “告警排障模板”“变更评审清单” |
你会发现：MCP 不强迫你把一切都做成“函数”。对于很多知识类接入，Resources + Prompts 往往更自然：把“可读材料”和“如何用材料”都标准化。
下面是一条最典型的链路（为便于理解做了简化）：
sequenceDiagram autonumber participant U as 用户 participant C as MCP Client（Agent） participant S as MCP Server（工具侧） participant B as 业务系统/API U->>C: 给我定位这次告警的根因，并给出处理建议 C->>S: list_tools / list_resources S-->>C: 返回可用工具/资源（含参数 schema） C->>S: call_tool(get_alert_detail, {id}) S->>B: 以受限凭证读取告警详情 B-->>S: 返回结构化数据/错误码 S-->>C: 返回结果（可脱敏/裁剪） C-->>U: 总结结论 + 引用证据 + 下一步建议这张图里最关键的边界是：MCP Server 是“治理与执行”的落点。客户端负责“决策”（该调用哪个工具、参数是什么、如何组织答案），而服务端负责“约束”（能不能调、调到哪里、返回多少、怎么记录）。
如果你只把 MCP 当成“又一层 RPC”，最后大概率会踩回老坑。把它当成“工具能力的 API Gateway”，才更符合它在组织里的位置。
一个经验法则：宁可多做几个小工具，也不要暴露一个“万能工具”。
推荐：
rollout_restart(namespace, deployment)，不要给 kubectl 原样透传；plan_change() 生成变更计划，再由人确认后 apply_change()；不推荐：
无论你用哪种客户端，不要把长期有效的高权限凭证交给模型上下文。更稳妥的做法是：
这并不依赖 MCP 独有能力，而是“把凭证与策略留在服务端”的工程共识。MCP 的优势在于：你可以把这套策略集中放在 Server 层，而不是散落在每个客户端里。
落地时建议把以下三件事当成一个整体：
可以把它当成“把 LLM 变成调用方之后，你必须补齐的客户端治理能力”。
上线后你至少需要能回答：
实践上，建议把工具调用日志结构化，并与链路追踪体系对齐（例如基于 OpenTelemetry 的指标/追踪规范）[4]，为后续复盘与风控留证据。
下面是一份可以直接贴进上线评审的最小清单：
1 | ✅ MCP 工具上线前 Checklist（最小集） |
如果你准备在团队里引入 MCP，一个更稳的路径是“先读后写、先低风险后高风险”：
这样做的好处是：你先把“协议、审计、脱敏、限流”这些地基打牢，再逐步扩大工具半径，而不是一开始就让代理握着生产变更按钮。
MCP 并不是对所有团队都划算，你可以按下面的判断：
适合 MCP 的情况：
不一定要 MCP 的情况：
MCP 的意义不在于“又多了一个协议”，而在于它把工具接入从“某个模型的私有能力”拉回到“工程团队可治理的公共基础设施”。当你的 AI 代理开始真正触达系统、数据与动作时，最值得投入的往往不是提示词花活，而是这层契约：可控、可审计、可演进。
建议从一个只读场景开始：把你最常用的 Runbook/指标查询封装成 MCP Server，先把治理闭环做出来；当团队对“工具调用的边界”达成共识后，再逐步把能力扩展到写入与自动化执行。
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>Generative AI workloads are pushing inference from a single API call into a full-fledged microservice stack that must balance latency, throughput, and budget. Whether you run on a self-managed GPU fleet or a managed platform, success now depends on a mature architecture, a disciplined metrics program, and relentless cost hygiene. This article distills the patterns we see across recent projects and public case studies to help teams design, observe, and optimize LLM inference services.
LLM inference is no longer a black-box API—it is a production system whose stability and unit economics determine how far AI capabilities can reach the business. By carefully selecting the right deployment model, operational metrics, and cost levers, teams can iteratively harden their inference microservices and create headroom for future models or multimodal workloads.
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>生成式 AI 的业务压力，正在把“推理服务”从单一 API 演变为具备自治扩缩容、可观测与成本治理能力的微服务体系。无论是云端的大模型平台，还是自建 GPU 集群，团队都必须在高吞吐、低延迟与预算约束之间取得平衡。本文结合近期项目经验与业界公开资料，梳理 LLM 推理微服务的典型架构模式、关键指标与成本治理手段，并通过真实案例总结排障思路。
LLM 推理微服务的成熟度，决定了大模型能力能否稳定地触达业务场景。从架构范式选择、指标体系设计到成本治理，都需要贯穿在工程团队的日常运维与复盘流程中。通过动态批处理、弹性扩缩容与完善的可观测性，将帮助团队在保证体验的同时控制预算，并为未来的模型升级与多模态拓展夯实基础。
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>Production teams no longer optimise only for page load times or requests per second. Every AI-powered feature introduces a chain of inference calls, vector lookups, feature pipelines, and GPU scheduling decisions. Guidance from Cloudflare Workers AI highlights why each model deserves its own guardrails—token ceilings, concurrency caps, and fallbacks—so a single overloaded edge node will not cascade into downtime. The latest DORA research from Google Cloud echoes the organisational lesson: elite teams rely on metric-driven automation to stay fast and resilient.
Modern budgets should spell out “inference envelopes”. A retrieval-augmented generation (RAG) flow might target an end-to-end P95 latency of 1.5 seconds, a per-request ceiling of ¥0.02, and a cache hit rate above 70%. Such targets combine provider pricing tables with historical telemetry, translating abstract GPU consumption into knobs that product owners understand. Once a service exhausts its envelope, the platform can throttle traffic, downgrade to a smaller model, or require users to opt into a paid high-precision mode.
Composability matters as soon as multiple models or tenants join the mix. A conversational assistant may orchestrate intent detection, knowledge retrieval, and long-form generation—each stage carries its own mini-budget and rolls up into a global guardrail. Solo builders can run the same playbook: estimate incremental token burn before exposing a feature, and surface a “performance mode” toggle when the burn threatens the baseline experience.
Budgets that cannot be observed will be ignored. Anchoring instrumentation on the OpenTelemetry specification keeps metrics, traces, and logs consistent across services. Real-time streams catch guardrail breaches—P95 latency spikes, GPU utilisation nearing saturation, or cache misses exploding. Daily snapshots and usage histograms reveal slow drifts, while trace sampling stitches parameters and payloads together so engineers can replay the exact context of an expensive request.
One SaaS vendor wired its LLM gateway to company SLOs: whenever the primary model exceeded the latency guardrail, traffic automatically shifted to a distilled sibling model and a counter tracked downgrade duration. The team funnelled routing events and inference stats through OpenTelemetry Collector dashboards, exposing the “spike → downgrade → recovery” loop. Lower-level signals from eBPF probes or cloud GPU telemetry helped them confirm whether the bottleneck lived in the model, storage layer, or network.
A budget earns its keep once it triggers action. Borrowing from the FinOps Framework, each guardrail should ship with policy-as-code: when costs approach the ceiling, enable aggressive response caching or fall back to quantised models; if latency climbs, spin up extra inference replicas or reroute requests to a nearer region. Multi-tenant products can mix these actions with anomaly detection to flag abusive traffic and to keep premium customers within higher bounds.
Automation still needs a parachute. Store thresholds and playbooks in Git, deliver them through GitOps, and archive every successful rollout so that a single command restores the last-known-good configuration. Feature-flag platforms add traceability by logging activation timestamps and correlating them with business metrics, proving whether an adaptive tweak generated measurable value.
Performance budgeting in the age of AI is a social contract across product, platform, and operations. By combining composable metrics, unified telemetry, and automated guardrails, teams can delight users without losing control of cost or reliability. The next milestone is to tie inference budgets directly to business KPIs, closing the loop from infrastructure tuning to customer impact so every optimisation tells a value story.
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>生成式 AI 正在席卷用户界面、内部工具与业务中台，推理链路成为新的性能热点。单靠传统的页面加载时间 (PLT) 或请求吞吐量不足以描述真实体验，模型上下文长度、向量检索命中率与 GPU 利用率都必须纳入预算边界。Cloudflare Workers AI 文档 就建议为每个模型设定最大 tokens、并发阈值与回退策略，避免边缘部署因资源争抢而级联失败。同样地，Google Cloud 2023 年 DORA 报告 强调“以指标驱动自动化决策”的团队在恢复能力和交付速度上表现更佳，为性能预算走向自适应提供了组织层面的背景。
性能预算的第一步是扩展维度，将推理成本与传统指标绑定：例如为检索增强生成 (RAG) 设定端到端 P95 延迟 ≤ 1.5 秒、单次推理成本 ≤ ¥0.02、缓存命中率 ≥ 70%。这些数字可以通过模型提供商的费用结构与历史数据推导出来，然后嵌入产品路线图。为了避免预算流于形式，需要将算力抽象成“推理配额”：上下文长度、批量大小、量化策略都映射为可调整的拨杆，一旦配额耗尽，系统要么降级到轻量模型，要么开启速率限制。
在多模型协同或多租户场景，还需要组合预算。例如智能客服可能串联意图识别、知识检索与长文本生成，可分别定义预算并最后聚合成全链路门槛。对于独立开发者，预算可以与功能可见性挂钩：上线前先预估新增模块的推理消耗，再决定是否在 UI 中加入“高精度模式”切换，以便在预算被击穿时向用户解释并给出替代方案。
没有可观测性，预算就是纸上谈兵。建议以 OpenTelemetry 官方规范 为中心，将推理延迟、token 消耗、模型错误率与缓存命中率统一埋点，再配合追踪把链路串起来。实时指标负责触发告警，例如 P95 延迟或 GPU 利用率接近上限；离线画像则帮助评估预算趋势，例如日均 token 消耗与峰值波动；采样追踪记录异常请求的上下文参数，为后续回放提供素材。
在生产案例中，一家 B2B SaaS 团队把 LLM Gateway 与业务 SLO 深度绑定：当延迟超阈值时自动降级至蒸馏模型，并记录降级次数与恢复时间。团队借助 OpenTelemetry Collector 将这些指标回传到统一的可视化平台，运维可以快速看到“高延迟→降级→恢复”的闭环。另外，eBPF 探针或云厂商原生监控可以补充底层网络与 GPU 状态，帮助识别瓶颈是否来自模型、存储或者网络。
预算的价值在于触发动作。可以借鉴 FinOps 框架 的治理思路，为每条预算配置“应对策略 + 审批流程”：例如成本逼近阈值时自动启用回答草稿缓存或切换到低精度模型；当延迟上升时临时增加推理副本或把请求迁移到邻近区域。多租户平台可以按客群设定分级预算，并结合异常检测识别恶意流量或 API 滥用。
策略自动化需要安全网。建议把预算阈值与策略写进 Git 仓库，通过 GitOps 在变更时触发审计，并必须保留最近一次成功发布的配置快照以便一键回滚。对于实验性调整，可以配合特性开关平台记录“启用/停用”时间戳，再将相关指标与业务效果关联，确保每次自适应都产生可量化的收益。
AI 时代的性能预算是一场跨部门协作：产品要定义体验底线，平台团队要提供预算拨杆，运维则负责可观测性与自动化执行。通过组合指标、统一数据闭环与策略化调优，我们可以在追求新体验的同时，控制成本与风险。下一步值得探索的是将推理预算与业务 KPI 直接对齐，形成“预算→策略→价值”的闭环，使每次模型调优都能量化其对业务的贡献。
本作品系原创，采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，转载请注明出处。
]]>