Stop an agent run¶
An agent run can take minutes and spend money at each step. The place of the run
decides how it stops. A run in its own web request stops when the client goes
away. A run in a background job stops when the job tells it to stop. The
StopWhen capability does that for both: before each step of the run, it asks
the application a question. When the answer is yes, it cancels the run.
The shop application in this guide has two agents. The assistant answers in
a web request. The researcher writes long reports in a worker, behind the
message bus.
Before you start¶
- Create the application with
uvx xtr-cli new shop --webapp. The template addsxtr-ormandxtr-messenger. - Add the package with
uv run xtr-recipes recipes:add "xtr-ai[openai]". The recipe listsAiBundleand writessrc/shop/config/packages/ai.py. - Add the database transport with
uv run xtr-recipes recipes:add "xtr-messenger[orm]". - Put the values of this machine in
.env.local:
- Run
uv run shop debug:bundles. The output namesai,ormandmessengerasactive.
Warning
Keep the key out of .env, because git tracks that file. .gitignore
excludes .env.local.
Configure the agents¶
- Open
src/shop/config/packages/ai.py. - Declare the platform and the two agents:
"""The language-model platforms and the agents built on them."""
from __future__ import annotations
from xtr_ai.bundle import AgentConfig, AiConfig, PlatformConfig
from xtr_dependency_injection import configure, env
__all__ = ["ai"]
@configure
def ai() -> AiConfig:
"""One platform, and the two agents of the application."""
return AiConfig(
platforms={
"openai": PlatformConfig(provider="openai", api_key=env("OPENAI_API_KEY")),
},
agents={
"assistant": AgentConfig(
platform="openai",
model="gpt-5-mini",
instructions="Answer briefly.",
),
"researcher": AgentConfig(
platform="openai",
model="gpt-5",
instructions="Search the library as often as you need, then write a report.",
),
},
)
- Run
uv run shop debug:ai. The output lists theopenaiplatform and the two agents.
Add the capability¶
StopWhen is a capability of the application. It holds no state, and it runs
no task of its own. pydantic-ai calls it before each model request and before
each tool call.
- Create
src/shop/ai/__init__.pywith a docstring. - Create
src/shop/ai/stop_when.py:
"""Stop an agent run at its next step, when the application says so."""
from __future__ import annotations
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from pydantic_ai import RunContext
from pydantic_ai.capabilities import AbstractCapability, ValidatedToolArgs
from pydantic_ai.messages import ToolCallPart
from pydantic_ai.models import ModelRequestContext
from pydantic_ai.tools import ToolDefinition
from typing import override
__all__ = ["StopWhen"]
@dataclass
class StopWhen(AbstractCapability[object]):
"""Cancel the run before its next model request or tool call, once ``should_stop`` is true.
A model response that streams now ends before the run stops.
"""
should_stop: Callable[[], Awaitable[bool]]
@override
async def before_model_request(
self, ctx: RunContext[object], request_context: ModelRequestContext
) -> ModelRequestContext:
"""Cancel the run before it asks the model again."""
if await self.should_stop():
ctx.cancel()
return request_context
@override
async def before_tool_execute(
self,
ctx: RunContext[object],
*,
call: ToolCallPart,
tool_def: ToolDefinition,
args: ValidatedToolArgs,
) -> ValidatedToolArgs:
"""Cancel the run before it calls a tool."""
del call, tool_def
if await self.should_stop():
ctx.cancel()
return args
@classmethod
@override
def get_serialization_name(cls) -> str | None:
"""Keep the capability out of the declarative agent definitions."""
return None
- Give the capability to a run, not to the agent:
agent.run(prompt, capabilities=[stop]). Each run then asks its own question.
A cancelled run raises RunCancelled. The messages that the run completed stay
in its history.
Stop a run in a web request¶
A run in a web request has two forms. A streamed run stops by itself. A run
that answers in one piece needs StopWhen.
- Open
src/shop/web/routes.py. - Add the imports and the type of the agent:
from collections.abc import AsyncIterator
from typing import Annotated
from fastapi import APIRouter, Request, Response
from fastapi.responses import StreamingResponse
from pydantic_ai.agent import AbstractAgent
from pydantic_ai.exceptions import RunCancelled
from xtr_dependency_injection import Target
from shop.ai.stop_when import StopWhen
Assistant = Annotated[AbstractAgent, Target("assistant")]
- Add a route that streams the answer:
@router.get("/ask")
async def ask(question: str, assistant: Assistant) -> StreamingResponse:
"""Stream the answer; the run stops when the client goes away."""
async def answer() -> AsyncIterator[str]:
async with assistant.run_stream(question) as result:
async for text in result.stream_text(delta=True):
yield text
return StreamingResponse(answer(), media_type="text/plain")
When the client goes away, the server cancels the stream. pydantic-ai then stops the run and closes the stream of the model. 4. Add a route that answers in one piece:
@router.get("/summary")
async def summary(question: str, request: Request, assistant: Assistant) -> Response:
"""Answer in one piece; stop the run at its next step if the client goes away."""
stop = StopWhen(request.is_disconnected)
try:
result = await assistant.run(question, capabilities=[stop])
except RunCancelled:
return Response(status_code=499)
return Response(str(result.output), media_type="text/plain")
The server does not cancel this route when the client goes away. StopWhen
asks request.is_disconnected before each step, and stops the run.
5. Serve the application with uv run shop-web.
6. Send curl --max-time 2 "localhost:8000/summary?question=Explain+anemia". Curl
stops after 2 seconds, and the run stops at its next step.
Stop a run in a background job¶
A background job runs in a worker, which can be on another server. The web
request that asks to stop the job cannot reach the run. The job must be a row in
the database: the request writes a flag on the row, and StopWhen reads it.
Store the job¶
- Create
src/shop/research/__init__.pywith a docstring. - Create
src/shop/research/research_job.py:
"""A research job, as one row of the ``research_job`` table."""
from __future__ import annotations
from enum import StrEnum
from typing import final
from advanced_alchemy.base import UUIDAuditBase
from sqlalchemy import String, Text
from sqlalchemy.orm import Mapped, mapped_column
__all__ = ["JobStatus", "ResearchJob"]
class JobStatus(StrEnum):
"""Where a research job stands."""
QUEUED = "queued"
RUNNING = "running"
DONE = "done"
CANCELLED = "cancelled"
FAILED = "failed"
@final
class ResearchJob(UUIDAuditBase):
"""One research job: what to research, where it stands, and what it found."""
__tablename__ = "research_job"
prompt: Mapped[str] = mapped_column(Text)
status: Mapped[str] = mapped_column(String(16), default=JobStatus.QUEUED)
cancel_requested: Mapped[bool] = mapped_column(default=False)
result: Mapped[str | None] = mapped_column(Text, default=None)
- Create
src/shop/research/job_store.py:
"""Research jobs, read and written in short transactions of their own."""
from __future__ import annotations
from typing import cast, final
from uuid import UUID
from sqlalchemy import CursorResult, select, update
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from xtr_dependency_injection import Injected, as_service
from shop.research.research_job import JobStatus, ResearchJob
__all__ = ["JobStore"]
_OPEN = (JobStatus.QUEUED, JobStatus.RUNNING)
@final
@as_service
class JobStore:
"""Each call opens its own session, so it sees what another server committed."""
def __init__(self, sessions: Injected[async_sessionmaker[AsyncSession]]) -> None:
"""Open sessions with ``sessions``."""
self._sessions = sessions
async def create(self, prompt: str) -> UUID:
"""Store a new job; answer its id."""
async with self._sessions.begin() as session:
job = ResearchJob(prompt=prompt)
session.add(job)
await session.flush()
return job.id
async def get(self, job_id: UUID) -> ResearchJob | None:
"""Answer the job, or ``None`` when no job has that id."""
async with self._sessions() as session:
return await session.get(ResearchJob, job_id)
async def start(self, job_id: UUID) -> bool:
"""Mark the job running; answer false when it already ended."""
return await self._update_open(job_id, status=JobStatus.RUNNING)
async def request_cancel(self, job_id: UUID) -> bool:
"""Set the flag; answer false when the job already ended."""
return await self._update_open(job_id, cancel_requested=True)
async def cancel_requested(self, job_id: UUID) -> bool:
"""Answer whether somebody asked the job to stop."""
async with self._sessions() as session:
flag = await session.scalar(
select(ResearchJob.cancel_requested).where(ResearchJob.id == job_id)
)
return bool(flag)
async def finish(self, job_id: UUID, status: JobStatus, result: str | None = None) -> None:
"""Record how the job ended."""
async with self._sessions.begin() as session:
_ = await session.execute(
update(ResearchJob)
.where(ResearchJob.id == job_id)
.values(status=status, result=result)
)
async def _update_open(self, job_id: UUID, **values: object) -> bool:
async with self._sessions.begin() as session:
result = cast(
"CursorResult[tuple[()]]",
await session.execute(
update(ResearchJob)
.where(ResearchJob.id == job_id, ResearchJob.status.in_(_OPEN))
.values(**values)
),
)
return result.rowcount == 1
- Write the migration with
uv run shop orm:migrations:diff --message "research jobs". The revision creates theresearch_jobtable and the table of the messages. - Apply it with
uv run shop orm:migrations:migrate.
Warning
Read the flag in a session of its own, as JobStore does. Do not read it on the
session of the handler. That session holds one transaction for the whole message.
On MySQL, that transaction never sees a flag that another server writes later.
Send the job to a worker¶
- Create
src/shop/research/message.py:
"""The message that starts a research job on a worker."""
from __future__ import annotations
from dataclasses import dataclass
from uuid import UUID
from xtr_messenger import as_message
__all__ = ["RunResearch"]
@as_message(name="shop.run_research.v1")
@dataclass(frozen=True, slots=True)
class RunResearch:
"""Run the research job ``job_id``."""
job_id: UUID
- Open
src/shop/config/packages/messenger.py. - Route the message to a
researchtransport:
@configure
def messenger() -> MessageBusConfig:
"""One transport, its broker read from ``MESSENGER_DSN``."""
return MessageBusConfig(
transports={"research": TransportConfig(env("MESSENGER_DSN"))},
routing={RunResearch: "research"},
)
- Add
from shop.research.message import RunResearchto the imports of the file.
Give the researcher a tool¶
- Create
src/shop/research/library.py:
"""The tools the researcher calls."""
from __future__ import annotations
from typing import final
from xtr_ai import as_tool
__all__ = ["Library"]
@final
class Library:
"""The clinic's library, searched by the researcher."""
@as_tool(agents=("researcher",))
async def search(self, query: str) -> str:
"""Search the clinic's library for a query."""
return f"three articles about {query}"
The researcher now calls search many times in one run. StopWhen asks its
question before each of those calls.
Run the job in the worker¶
- Create
src/shop/research/handler.py:
"""The worker's side of a research job."""
from __future__ import annotations
from typing import Annotated
from pydantic_ai.agent import AbstractAgent
from pydantic_ai.exceptions import RunCancelled
from xtr_dependency_injection import Injected, Target
from xtr_messenger import as_message_handler
from shop.ai.stop_when import StopWhen
from shop.research.job_store import JobStore
from shop.research.message import RunResearch
from shop.research.research_job import JobStatus
__all__ = ["run_research"]
@as_message_handler(RunResearch)
async def run_research(
message: RunResearch,
jobs: Injected[JobStore],
researcher: Annotated[AbstractAgent, Target("researcher")],
) -> None:
"""Run the researcher until it answers, or until the job is cancelled."""
job = await jobs.get(message.job_id)
if job is None or not await jobs.start(job.id):
return # a delivery of a job that already ended
stop = StopWhen(lambda: jobs.cancel_requested(job.id))
try:
result = await researcher.run(job.prompt, capabilities=[stop])
except RunCancelled:
await jobs.finish(job.id, JobStatus.CANCELLED)
return
except Exception:
await jobs.finish(job.id, JobStatus.FAILED)
raise
await jobs.finish(job.id, JobStatus.DONE, str(result.output))
- Keep the first check of the handler. The transport sends a message again
after a worker stops in the middle of a job. A job that ended stays ended.
A job that is still
runningstarts again from its prompt.
Start, follow and stop the job¶
- Add the imports to
src/shop/web/routes.py:
from uuid import UUID
from fastapi import HTTPException
from xtr_dependency_injection import Injected
from xtr_messenger import MessageBusInterface
from shop.research.job_store import JobStore
from shop.research.message import RunResearch
- Add the three routes of a job:
@router.post("/research", status_code=202)
async def start_research(
prompt: str, jobs: Injected[JobStore], bus: Injected[MessageBusInterface]
) -> dict[str, str]:
"""Create the job, and hand it to a worker."""
job_id = await jobs.create(prompt)
_ = await bus.dispatch(RunResearch(job_id=job_id))
return {"job_id": str(job_id)}
@router.post("/research/{job_id}/cancel", status_code=202)
async def cancel_research(job_id: UUID, jobs: Injected[JobStore]) -> dict[str, str]:
"""Ask the worker to stop; it stops at the run's next step."""
if not await jobs.request_cancel(job_id):
raise HTTPException(status_code=409, detail="The job already ended.")
return {"status": "cancelling"}
@router.get("/research/{job_id}")
async def research_status(job_id: UUID, jobs: Injected[JobStore]) -> dict[str, str | None]:
"""Answer where the job stands."""
job = await jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404)
return {"status": job.status, "result": job.result}
Try it¶
- Serve the application with
uv run shop-web. - In a second terminal, run the worker with
uv run shop messenger:consume research. - Start a job:
The answer gives the id of the job, for example
{"job_id":"3a2f2efb-6663-415a-92a5-42bec02a101f"}.
4. Ask for the state with curl -s localhost:8000/research/<job_id>. The answer
is {"status":"running","result":null}.
5. Stop the job with curl -s -X POST localhost:8000/research/<job_id>/cancel.
The answer is {"status":"cancelling"}.
6. Ask for the state again. The answer is {"status":"cancelled","result":null}.
7. Stop the job again. The answer has the status code 409, because the job ended.
When the run stops¶
| The run is | It stops |
|---|---|
| Streamed in a web request | When the client goes away, in the middle of the stream |
| In one piece in a web request | Before its next model request or tool call |
| In a background job | Before its next model request or tool call |
A model request that has started always ends before the run stops. If this
delay is too long for a job, send the signal to the worker instead. For example,
use LISTEN and NOTIFY of PostgreSQL. StopWhen then reads a value in memory.
Next step¶
- Send and handle messages describes the transports and the worker.
- Test an application runs a handler with no broker.
- xtr-ai describes the platforms, the agents and the conversation stores.