INCIDENT · HOW IT STARTED
We accepted the message. We lost the reply.
STACHKA · Anton Surikov · Sber
You may acknowledge receipt only after the work is persisted
The offset is saved before the reply. If the reply never went out and the process restarted, Telegram will not send that update again. "Accepted" and "done" are not the same thing here.
ACK only after the result is saved
STACHKA · Anton Surikov · Sber
Redelivery is now possible — so idempotency is required
async def on_update(update: Update) -> None:
task_id = await store.register(update.update_id) # запись в БД до работы
try:
result = await handle(task_id, update)
await store.save_result(task_id, result)
await bot.reply(update, result)
except Exception:
await store.rollback_offset(update.update_id) # update придёт снова
raise
if await store.task_id_for(update.update_id) == task_id:
await store.ack(update.update_id)
task_id is created upfront
failure → update comes again
our task_id, not someone else's
OUTCOME · WHAT THIS TALK IS ABOUT
An LLM agent is an unreliable component
STACHKA · Anton Surikov · Sber
This is a talk about lessons and failures, not a finished product
Reliability comes from ordinary Python engineering, not from prompts.
Slide labels: PROTOTYPE — works · SPEC — target requirement · INCIDENT — what actually broke.
• task contract · state in the DB · idempotency
• rights outside the text · end-to-end tests
Why several agents are needed here at all
STACHKA · Anton Surikov · Sber
The reason for a multi-agent network is not the number of systems but different owners and rights
different teams own data and decisions
HR, Finance, Docs, mail, calendar
one IT account is no longer enough
a task outlives the session
Five agents: one lead and four blocks
STACHKA · Anton Surikov · Sber
The lead issues the assignment, blocks 1–4 hold their own data. The A2A gateway and the interaction store handle delivery, states, duplicate protection and audit.
• ✓ Every message goes through the gateway
• ✓ Each agent has its own rights
• ✓ The gateway never reads the assignment text
What one failed cycle cost
STACHKA · Anton Surikov · Sber
A task needs limits: calls, tokens, time, retries, delegation depth
The 65 536 limit was not invented: the server runs with a 131 072 context and --parallel 2, i.e. 65 536 per slot.
| Metric | Value |
| Review request | 106 081 tokens |
| Limit | 65 536 |
| Does not fit | +40 545 |
| Input for one failed cycle | 46 583 |
| Local model reply | 3–5 minutes |
A tool call reached the user as plain text
STACHKA · Anton Surikov · Sber
LEAK = re.compile(r'<tool_call>|<function|"arguments"\s*:')
def call_llm(prompt: str) -> str:
for _ in range(3):
out = llm(prompt)
if not LEAK.search(out):
return out
raise ToolCallLeak(prompt)
tool_call in the text, sometimes 400
chat template did not match the model
detector and contract test
A network failure was reported as a policy violation
STACHKA · Anton Surikov · Sber
class Kind(Enum):
RETRYABLE = "retry" # 429, 5xx, timeout
INFRA = "infra" # DNS, 413, quota
SAFETY = "safety" # only a filter decision
INVALID = "invalid" # contract violated
match classify(exc):
case Kind.RETRYABLE: retry()
case _: raise exc
"safety violation" and zero retries
413 and DNS in a bare except Exception
When may you acknowledge processing
STACHKA · Anton Surikov · Sber
At-least-once: duplicates will happen and order is not guaranteed. ACK is about the message; task completion is a separate status
• OUTBOX — state and event in one transaction
• INBOX — UNIQUE(idempotency_key), a duplicate returns the earlier result
• RETRIES — 0/5/30/120 s + jitter → DLQ
The shared quota fell in a single wave
STACHKA · Anton Surikov · Sber
A backup that shares quota with the primary path is not a backup
A fallback path without an isolated quota is not a fallback.
Primary and backup models became unavailable at the same time — they share one limit.
6 of 9 tests were failing and the agent kept committing
STACHKA · Anton Surikov · Sber
Forbidding a commit to a red branch is a CI rule, not a request in a prompt
"Yes, I am able to rewrite my own code" — while the agent had not checked the files or the tooling.
• The model's self-report is not evidence
• Red tests were not a reason for it to stop
• The user never found out
Three layers and who is responsible for what
STACHKA · Anton Surikov · Sber
• A2A — messages, identity, statuses
• CONTROL PLANE — rights, states, deadlines, retries, audit
• MCP — business requests and actions
Three lessons for a Python developer
STACHKA · Anton Surikov · Sber
An LLM agent is an unreliable component. Reliability comes from the engineering around it.
1. Task state lives in the DB, not in asyncio.Task: a restart must not lose tasks.
2. Idempotency is proven by a test that crashes at every point: effect → commit → ACK.
3. Rights, deadlines and limits are checked by code, not by a prompt.