Skip to content

xtr_orm_messenger

A database transport for xtr-messenger, on the connections xtr-orm configures.

Messages are kept in a table on one of the application's connections, named by the DSN — orm://default. A worker claims the oldest one of its queue that is due, a delayed one falls due when its delay has passed, a message whose worker died is handed out again, and on PostgreSQL a waiting worker is woken the moment a message is sent.

DEFAULT_QUEUE module-attribute

DEFAULT_QUEUE: Final = 'default'

The queue a transport reads and writes when the configuration names none.

DEFAULT_TABLE_NAME module-attribute

DEFAULT_TABLE_NAME: Final = 'messenger_messages'

The table messages are kept in when the configuration names none.

ORM_OPTIONS module-attribute

ORM_OPTIONS: Final = ('auto_setup', 'base_delay_seconds', 'check_delayed_interval', 'failure_queue', 'get_notify_timeout', 'max_attempts', 'poll_interval', 'queue', 'redeliver_timeout', 'table_name', 'transactional', 'use_notify')

Every setting an orm:// transport accepts; any other is refused.

ORM_SCHEME module-attribute

ORM_SCHEME: Final = 'orm'

The DSN scheme the transport serves.

InvalidOrmDsnError

InvalidOrmDsnError(transport: str, connection: str, known: tuple[str, ...])

Bases: OrmMessengerError, LookupError

An orm:// DSN names a connection the application does not configure.

Also a :class:LookupError, as a missing key in any other mapping is.

Attributes:

  • transport (str) –

    The transport whose DSN names it.

  • connection (str) –

    The connection named.

  • known (tuple[str, ...]) –

    Every connection that is configured.

Record the transport, the connection it named and the ones that exist.

OrmMessengerError

Bases: MessageBusError

Base class for every error raised by this library.

Also a :class:~xtr_messenger.MessageBusError, so code catching what the bus can go wrong with catches this transport's failures too.

TransportError

TransportError(transport: str, operation: str, reason: str)

Bases: OrmMessengerError

The database refused what a transport asked of it.

Raised from the driver's own error, which stays the cause, so nothing it said is lost. The message names the transport and the operation, never the connection's URL, whose credentials a traceback would otherwise carry.

Attributes:

  • transport (str) –

    The transport the operation was for.

  • operation (str) –

    What was asked — send, get, ack, ...

  • reason (str) –

    What went wrong.

Record which transport failed, at what, and why.

Connection

Connection(engine_factory: Callable[[], Awaitable[AsyncEngine]], options: OrmOptions, *, session_factory: Callable[[], Awaitable[AsyncSession]] | None = None, clock: ClockInterface | None = None)

Sends, claims, settles and lists the messages of one queue of one table.

Built with what reaches the engine rather than the engine itself, so that building one does no I/O and reads no configuration: the engine is asked for the first time a statement runs.

Run statements on the engine engine_factory returns, as options say.

Parameters:

  • engine_factory (Callable[[], Awaitable[AsyncEngine]]) –

    Returns the engine of the connection the table is on; called each time a statement runs, so it must hand back the same engine every time.

  • options (OrmOptions) –

    The table, the queue and how to treat them.

  • session_factory (Callable[[], Awaitable[AsyncSession]] | None, default: None ) –

    Returns the session of the unit of work under way, raising :class:~xtr_orm.exception.SessionUnavailableError outside of one; consulted to send when options.transactional is on.

  • clock (ClockInterface | None, default: None ) –

    What reads the time; the system clock when left out.

options property

options: OrmOptions

The table, the queue and how this connection treats them.

table property

table: Table

The messages table, as this connection defines it.

send async

send(body: str, headers: Mapping[str, str], delay_ms: int = 0) -> str

Keep a message, due delay_ms milliseconds from now; return its id.

A positive delay is rounded up to a whole second: some databases keep a datetime to the second, and dropping the fraction would make the message due before the delay asked for.

With transactional on and a unit of work under way, the message is written in that unit's session and is sent only once the unit commits.

get async

get() -> StoredMessage | None

Claim the oldest message of the queue that is due, or return None.

The message is marked handed out in the same transaction, so no other worker claims it until :meth:ack, :meth:reject or redeliver_timeout releases it.

ack async

ack(message_id: str) -> None

Forget the message message_id: it was handled.

reject async

reject(message_id: str) -> None

Forget the message message_id: it failed for good, with nowhere left to go.

requeue async

requeue(message_id: str, *, queue: str, body: str, headers: Mapping[str, str], delay_ms: int) -> None

Put the message message_id back in line in queue, due delay_ms from now.

Its body and headers are replaced, so what the failed attempt learned — the attempt count, the reason — travels with it. The row keeps its id and the time it was first sent.

keepalive async

keepalive(message_id: str) -> None

Mark the message message_id handed out now, restarting its redelivery clock.

For a handler that runs longer than redeliver_timeout: called while it runs, the message is not handed to another worker under it.

find async

find(message_id: str) -> StoredMessage | None

Return the message message_id of the queue, handed out or not, if it is kept.

all async

all(limit: int | None = None) -> AsyncIterator[StoredMessage]

Yield the messages of the queue a worker could claim now, oldest first.

With no limit, every one is listed, read :data:LISTING_PAGE_SIZE at a time so a long queue is never held in memory whole.

count async

count() -> int

Return how many messages of the queue a worker could claim now.

setup async

setup() -> None

Create the messages table if it does not exist; an existing one is left as it is.

ping async

ping() -> bool

Tell whether the database answers, opening a connection if the pool holds none.

wait async

wait() -> None

Wait before a worker asks again, having found nothing due.

On PostgreSQL through asyncpg with use_notify, until a message is sent to the queue or check_delayed_interval passes, whichever is first; anywhere else, poll_interval.

The first time a PostgreSQL worker waits it only starts listening and returns: a message sent between the poll that found nothing and the moment the listening started announced itself to nobody, so the worker asks once more before it waits for real.

A listening connection the database dropped is replaced the same way on the next wait; while listening again fails, the worker waits its poll_interval and tries again the time after.

close async

close() -> None

Release the connection a PostgreSQL worker listens for notifications on.

MessagesTableDriver

MessagesTableDriver(table_name: str)

Bases: MappingDriverInterface

Adds the messages table to the definitions a connection's migrations compare with.

With it loaded, the table is written by the next revision like any of the application's own, instead of created by the transport the first time it runs.

Map the messages table named table_name.

load_metadata

load_metadata(metadata: MetaData) -> tuple[str, ...]

Add the table to metadata unless it is there already; return what was added.

OrmOptions dataclass

OrmOptions(table_name: str = DEFAULT_TABLE_NAME, queue: str = DEFAULT_QUEUE, redeliver_timeout: int = 3600, auto_setup: bool = True, poll_interval: float = 1.0, transactional: bool = False, use_notify: bool = True, check_delayed_interval: int = 60000, get_notify_timeout: int = 60000, max_attempts: int = 3, base_delay_seconds: float = 1.0, failure_queue: str = DEFAULT_FAILURE_QUEUE)

How an orm:// transport keeps, hands out and hands back its messages.

Attributes:

  • table_name (str) –

    The table messages are kept in.

  • queue (str) –

    The queue this transport reads and writes; transports sharing a table but naming different queues never see each other's messages.

  • redeliver_timeout (int) –

    Seconds after which a message handed to a worker and never settled is handed out again — the worker is taken to have died with it.

  • auto_setup (bool) –

    Create the table the first time an operation finds it missing.

  • poll_interval (float) –

    Seconds a worker waits before asking again when nothing was due.

  • transactional (bool) –

    Write a message in the session of the unit of work under way, so it commits or rolls back with the handler's own changes; outside a unit it is written in a transaction of its own.

  • use_notify (bool) –

    On PostgreSQL, wake a waiting worker the moment a message is sent instead of when its poll interval runs out.

  • check_delayed_interval (int) –

    Milliseconds a worker waiting for a notification waits at most before asking anyway — a delayed message falling due sends none.

  • get_notify_timeout (int) –

    Milliseconds one wait for a notification lasts at most.

  • max_attempts (int) –

    Attempts a message gets, the first included; a rejected message with attempts left is put back in its queue.

  • base_delay_seconds (float) –

    Seconds before the second attempt; each later one waits twice as long as the one before.

  • failure_queue (str) –

    The queue of the same table a message is moved to once its attempts run out, where it can be inspected, or consumed like any queue to try it again. A transport reading the failure queue itself deletes a message whose attempts there run out.

from_settings classmethod

from_settings(settings: Mapping[str, str]) -> OrmOptions

Read the options out of a transport's settings, the defaults filling the rest.

Raises:

  • UnknownTransportOptionError –

    If a setting is not one this transport accepts.

  • InvalidTransportOptionError –

    If a value is not what its setting takes.

OrmReceivedStamp dataclass

OrmReceivedStamp(id: str)

Bases: NonSendableStampInterface

The row an envelope came from, which settling it deletes.

Never sent on: it names a row of this table, which means nothing to a transport the envelope is dispatched to next.

OrmReceiver

OrmReceiver(connection: Connection, serializer: SerializerInterface, name: str, clock: ClockInterface | None = None)

Bases: ReceiverInterface

Claims the messages of one queue, deleting each once handled and retrying each rejected.

A rejected message with attempts left is put back in its queue, due after base_delay_seconds doubled for each attempt it already had, its attempt count kept in a :class:~xtr_messenger.RedeliveryStamp and the reason it failed in an :class:~xtr_messenger.ErrorDetailsStamp. Once its max_attempts are spent it is moved to the failure_queue with a fresh count, so whoever consumes that queue gets to try it again; a message failing for good in the failure queue itself is deleted. A message moved there carries a :class:~xtr_messenger.SentToFailureTransportStamp naming this transport and when it was moved, which the messenger:failed:* commands show and select it by.

Workers polling one table run into each other's transactions; a conflict says nothing about the message, so a poll sits through :data:MAX_RETRIES of them in a row before it gives up, and a settle tries again with a growing pause. A settle whose database went away waits up to :data:RECONNECT_TIMEOUT seconds for it and tries once more: losing an acknowledgement to a database restart would have the message handed out, and handled, again.

A message is kept alive while it is held: from the moment it is handed to the worker until it is settled, its claim is renewed every half of redeliver_timeout, so a handler running longer than that is not taken for dead and its message handed to another worker under it. A renewal the database refuses is logged and tried again at the next one.

Receive from connection as the transport name, decoding with serializer.

get async

get() -> AsyncIterator[Envelope]

Yield each message as it falls due, waiting between polls that find none.

Never runs dry: a worker stops it by cancelling. A message is yielded the moment it is claimed, with nothing awaited in between, so a cancellation never loses one; and one lost anyway — the process killed — is handed out again after redeliver_timeout.

A message that does not decode can never be handled, so it is rejected at once, its reason logged — never its body, which may carry anything a producer put in a message.

Raises:

  • TransportError –

    When the database refuses a poll for any reason but a conflict, or for a conflict :data:MAX_RETRIES times in a row.

ack async

ack(envelope: Envelope) -> None

Delete the message envelope was received as: it was handled.

Raises:

  • TransportError –

    When envelope was not received from this transport, or the database refuses the delete.

reject async

reject(envelope: Envelope) -> None

Retry the message envelope was received as, or move it to the failure queue.

Raises:

  • TransportError –

    When envelope was not received from this transport, or the database refuses the update.

will_retry

will_retry(envelope: Envelope) -> bool

Tell whether rejecting envelope puts it back in its queue for another attempt.

decode

decode(stored: StoredMessage) -> Envelope

Rebuild the envelope stored keeps, stamped with the row it came from.

A transport message id the envelope carried from an earlier transport is replaced by this row's, so the id it carries is the one that settles it here.

Raises:

OrmSender

OrmSender(connection: Connection, serializer: SerializerInterface, name: str)

Bases: SenderInterface

Writes each envelope it is handed into the table, due after its delay.

Send through connection as the transport name, encoding with serializer.

send async

send(envelope: Envelope) -> Envelope

Keep envelope, due after its :class:~xtr_messenger.DelayStamp, if any.

Returns it stamped with the id the row was given.

Raises:

  • MessageEncodingFailedError –

    When the envelope does not encode.

  • TransportError –

    When the database refuses the insert.

OrmTransport

OrmTransport(connection: Connection, serializer: SerializerInterface, *, name: str = 'orm', clock: ClockInterface | None = None)

Bases: TransportInterface, ListableReceiverInterface, MessageCountAwareInterface, SetupableTransportInterface

Sends to, receives from and lists one queue of a messages table.

Beyond sending and receiving, the queue can be looked into — counted, listed, one message found by id — and a message a long handler holds can be kept from being handed out again. That is what the console's messenger:setup-transports, messenger:stats, messenger:show and messenger:failed:* commands read it through.

Use connection as the transport name.

Both halves encode and decode with serializer, so a producer and a worker built from the same configuration always agree on the wire format.

name property

name: str

The transport's name, which every envelope it receives is stamped with.

connection property

connection: Connection

What runs this transport's statements.

send async

send(envelope: Envelope) -> Envelope

Keep envelope until a worker claims it; see :meth:OrmSender.send.

get

get() -> AsyncIterator[Envelope]

Yield each message as it falls due; see :meth:OrmReceiver.get.

ack async

ack(envelope: Envelope) -> None

Delete the message envelope was received as: it was handled.

reject async

reject(envelope: Envelope) -> None

Retry the message envelope was received as, or move it to the failure queue.

will_retry

will_retry(envelope: Envelope) -> bool

Tell whether rejecting envelope puts it back in its queue for another attempt.

keepalive async

keepalive(envelope: Envelope, seconds: int | None = None) -> None

Keep the message envelope was received as from being handed out again.

Restarts its redelivery clock: call it while a handler runs longer than redeliver_timeout. seconds is how long the caller means to wait before calling again, refused when longer than the timeout, which would hand the message out in between.

Raises:

  • TransportError –

    When seconds is longer than redeliver_timeout, envelope was not received from this transport, or the database refuses the update.

find async

find(message_id: str) -> Envelope | None

Return the envelope kept as message_id in this queue, or None.

Raises:

  • MessageDecodingFailedError –

    When the row does not decode.

  • TransportError –

    When the database refuses the query.

all async

all(limit: int | None = None) -> AsyncIterator[Envelope]

Yield the envelopes a worker could claim now, oldest first, limit at most.

Raises:

  • MessageDecodingFailedError –

    When a row does not decode.

  • TransportError –

    When the database refuses the query.

count async

count() -> int

Return how many messages a worker could claim now.

Raises:

setup async

setup() -> None

Create the messages table if it does not exist.

Raises:

close async

close() -> None

Release the connection a worker listens for notifications on, if any.

OrmTransportFactory

OrmTransportFactory(registry: ConnectionRegistry, serializer: SerializerInterface, clock: ClockInterface | None = None)

Bases: TransportFactoryInterface, PublisherClosingInterface

Builds orm:// transports, each on the connection its DSN names.

One transport per name, built the first time and kept: the bus sending to it and the worker draining it must hold the same one, as they would any transport holding a connection.

Not discovered by DSN scheme like a transport needing nothing: it needs the application's connections, which only the orm bundle holds. With a container, that bundle registers it when this package is installed; without one, pass it to the bus yourself::

MessageBusFactory(config, [OrmTransportFactory(registry, serializer)])

Build transports on registry's connections.

Parameters:

  • registry (ConnectionRegistry) –

    The application's connections, by name.

  • serializer (SerializerInterface) –

    What every transport encodes and decodes envelopes with.

  • clock (ClockInterface | None, default: None ) –

    What every transport reads the time from; the system clock when left out.

supports

supports(dsn: Dsn) -> bool

Recognise the orm scheme.

create

create(group: Mapping[str, TransportConfig]) -> Mapping[str, SenderInterface]

Return the transport for each name, building it the first time.

Building one opens nothing: the connection's engine is asked for the first time the transport runs a statement.

Raises:

  • InvalidOrmDsnError –

    When a DSN names a connection the registry does not hold.

  • UnknownTransportOptionError –

    When a transport carries a setting this one does not accept.

  • InvalidTransportOptionError –

    When a setting's value is not what it takes.

transport

transport(name: str, config: TransportConfig) -> OrmTransport

Return the transport name, building it from config the first time.

Raises:

  • InvalidOrmDsnError –

    When the DSN names a connection the registry does not hold.

  • UnknownTransportOptionError –

    When config carries a setting this transport does not accept.

  • InvalidTransportOptionError –

    When a setting's value is not what it takes.

close_publishers async

close_publishers() -> None

Release the connections the transports this factory built listen on.

Sending holds no connection of its own — the engine's pool does, and the connection's owner disposes it — so this leaves only the connection a PostgreSQL worker listens for notifications on.

StoredMessage dataclass

StoredMessage(id: str, body: str, headers: Mapping[str, str], queue: str, created_at: datetime, available_at: datetime, delivered_at: datetime | None)

One row of the messages table, its envelope still encoded.

Attributes:

  • id (str) –

    The row's identifier, as the transport hands it out.

  • body (str) –

    The encoded message.

  • headers (Mapping[str, str]) –

    The encoded envelope's headers: its type and its stamps.

  • queue (str) –

    The queue it was sent to.

  • created_at (datetime) –

    When it was sent, in UTC.

  • available_at (datetime) –

    When it falls due, in UTC — later than created_at for a delayed message.

  • delivered_at (datetime | None) –

    When a worker was last handed it, in UTC, or None while no worker has been.