xtr_orm_messenger ¶
A database transport for xtr-messenger, on the connections xtr-orm configures.
Messages are kept in a table on one of the application's connections, named by
the DSN — orm://default. A worker claims the oldest one of its queue that
is due, a delayed one falls due when its delay has passed, a message whose
worker died is handed out again, and on PostgreSQL a waiting worker is woken
the moment a message is sent.
DEFAULT_QUEUE
module-attribute
¶
The queue a transport reads and writes when the configuration names none.
DEFAULT_TABLE_NAME
module-attribute
¶
The table messages are kept in when the configuration names none.
ORM_OPTIONS
module-attribute
¶
ORM_OPTIONS: Final = ('auto_setup', 'base_delay_seconds', 'check_delayed_interval', 'failure_queue', 'get_notify_timeout', 'max_attempts', 'poll_interval', 'queue', 'redeliver_timeout', 'table_name', 'transactional', 'use_notify')
Every setting an orm:// transport accepts; any other is refused.
InvalidOrmDsnError ¶
Bases: OrmMessengerError, LookupError
An orm:// DSN names a connection the application does not configure.
Also a :class:LookupError, as a missing key in any other mapping is.
Attributes:
-
transport(str) –The transport whose DSN names it.
-
connection(str) –The connection named.
-
known(tuple[str, ...]) –Every connection that is configured.
Record the transport, the connection it named and the ones that exist.
OrmMessengerError ¶
Bases: MessageBusError
Base class for every error raised by this library.
Also a :class:~xtr_messenger.MessageBusError, so code catching what the
bus can go wrong with catches this transport's failures too.
TransportError ¶
Bases: OrmMessengerError
The database refused what a transport asked of it.
Raised from the driver's own error, which stays the cause, so nothing it said is lost. The message names the transport and the operation, never the connection's URL, whose credentials a traceback would otherwise carry.
Attributes:
-
transport(str) –The transport the operation was for.
-
operation(str) –What was asked —
send,get,ack, ... -
reason(str) –What went wrong.
Record which transport failed, at what, and why.
Connection ¶
Connection(engine_factory: Callable[[], Awaitable[AsyncEngine]], options: OrmOptions, *, session_factory: Callable[[], Awaitable[AsyncSession]] | None = None, clock: ClockInterface | None = None)
Sends, claims, settles and lists the messages of one queue of one table.
Built with what reaches the engine rather than the engine itself, so that building one does no I/O and reads no configuration: the engine is asked for the first time a statement runs.
Run statements on the engine engine_factory returns, as options say.
Parameters:
-
engine_factory(Callable[[], Awaitable[AsyncEngine]]) –Returns the engine of the connection the table is on; called each time a statement runs, so it must hand back the same engine every time.
-
options(OrmOptions) –The table, the queue and how to treat them.
-
session_factory(Callable[[], Awaitable[AsyncSession]] | None, default:None) –Returns the session of the unit of work under way, raising :class:
~xtr_orm.exception.SessionUnavailableErroroutside of one; consulted to send whenoptions.transactionalis on. -
clock(ClockInterface | None, default:None) –What reads the time; the system clock when left out.
send
async
¶
Keep a message, due delay_ms milliseconds from now; return its id.
A positive delay is rounded up to a whole second: some databases keep a datetime to the second, and dropping the fraction would make the message due before the delay asked for.
With transactional on and a unit of work under way, the message is
written in that unit's session and is sent only once the unit commits.
get
async
¶
Claim the oldest message of the queue that is due, or return None.
The message is marked handed out in the same transaction, so no other
worker claims it until :meth:ack, :meth:reject or
redeliver_timeout releases it.
reject
async
¶
Forget the message message_id: it failed for good, with nowhere left to go.
requeue
async
¶
requeue(message_id: str, *, queue: str, body: str, headers: Mapping[str, str], delay_ms: int) -> None
Put the message message_id back in line in queue, due delay_ms from now.
Its body and headers are replaced, so what the failed attempt learned — the attempt count, the reason — travels with it. The row keeps its id and the time it was first sent.
keepalive
async
¶
Mark the message message_id handed out now, restarting its redelivery clock.
For a handler that runs longer than redeliver_timeout: called
while it runs, the message is not handed to another worker under it.
find
async
¶
Return the message message_id of the queue, handed out or not, if it is kept.
all
async
¶
Yield the messages of the queue a worker could claim now, oldest first.
With no limit, every one is listed, read :data:LISTING_PAGE_SIZE
at a time so a long queue is never held in memory whole.
setup
async
¶
Create the messages table if it does not exist; an existing one is left as it is.
ping
async
¶
Tell whether the database answers, opening a connection if the pool holds none.
wait
async
¶
Wait before a worker asks again, having found nothing due.
On PostgreSQL through asyncpg with use_notify, until a message is
sent to the queue or check_delayed_interval passes, whichever is
first; anywhere else, poll_interval.
The first time a PostgreSQL worker waits it only starts listening and returns: a message sent between the poll that found nothing and the moment the listening started announced itself to nobody, so the worker asks once more before it waits for real.
A listening connection the database dropped is replaced the same way
on the next wait; while listening again fails, the worker waits its
poll_interval and tries again the time after.
close
async
¶
Release the connection a PostgreSQL worker listens for notifications on.
MessagesTableDriver ¶
Bases: MappingDriverInterface
Adds the messages table to the definitions a connection's migrations compare with.
With it loaded, the table is written by the next revision like any of the application's own, instead of created by the transport the first time it runs.
Map the messages table named table_name.
load_metadata ¶
Add the table to metadata unless it is there already; return what was added.
OrmOptions
dataclass
¶
OrmOptions(table_name: str = DEFAULT_TABLE_NAME, queue: str = DEFAULT_QUEUE, redeliver_timeout: int = 3600, auto_setup: bool = True, poll_interval: float = 1.0, transactional: bool = False, use_notify: bool = True, check_delayed_interval: int = 60000, get_notify_timeout: int = 60000, max_attempts: int = 3, base_delay_seconds: float = 1.0, failure_queue: str = DEFAULT_FAILURE_QUEUE)
How an orm:// transport keeps, hands out and hands back its messages.
Attributes:
-
table_name(str) –The table messages are kept in.
-
queue(str) –The queue this transport reads and writes; transports sharing a table but naming different queues never see each other's messages.
-
redeliver_timeout(int) –Seconds after which a message handed to a worker and never settled is handed out again — the worker is taken to have died with it.
-
auto_setup(bool) –Create the table the first time an operation finds it missing.
-
poll_interval(float) –Seconds a worker waits before asking again when nothing was due.
-
transactional(bool) –Write a message in the session of the unit of work under way, so it commits or rolls back with the handler's own changes; outside a unit it is written in a transaction of its own.
-
use_notify(bool) –On PostgreSQL, wake a waiting worker the moment a message is sent instead of when its poll interval runs out.
-
check_delayed_interval(int) –Milliseconds a worker waiting for a notification waits at most before asking anyway — a delayed message falling due sends none.
-
get_notify_timeout(int) –Milliseconds one wait for a notification lasts at most.
-
max_attempts(int) –Attempts a message gets, the first included; a rejected message with attempts left is put back in its queue.
-
base_delay_seconds(float) –Seconds before the second attempt; each later one waits twice as long as the one before.
-
failure_queue(str) –The queue of the same table a message is moved to once its attempts run out, where it can be inspected, or consumed like any queue to try it again. A transport reading the failure queue itself deletes a message whose attempts there run out.
from_settings
classmethod
¶
Read the options out of a transport's settings, the defaults filling the rest.
Raises:
-
UnknownTransportOptionError–If a setting is not one this transport accepts.
-
InvalidTransportOptionError–If a value is not what its setting takes.
OrmReceivedStamp
dataclass
¶
Bases: NonSendableStampInterface
The row an envelope came from, which settling it deletes.
Never sent on: it names a row of this table, which means nothing to a transport the envelope is dispatched to next.
OrmReceiver ¶
OrmReceiver(connection: Connection, serializer: SerializerInterface, name: str, clock: ClockInterface | None = None)
Bases: ReceiverInterface
Claims the messages of one queue, deleting each once handled and retrying each rejected.
A rejected message with attempts left is put back in its queue, due
after base_delay_seconds doubled for each attempt it already had, its
attempt count kept in a :class:~xtr_messenger.RedeliveryStamp and the
reason it failed in an :class:~xtr_messenger.ErrorDetailsStamp. Once
its max_attempts are spent it is moved to the failure_queue with
a fresh count, so whoever consumes that queue gets to try it again; a
message failing for good in the failure queue itself is deleted. A
message moved there carries a
:class:~xtr_messenger.SentToFailureTransportStamp naming this
transport and when it was moved, which the messenger:failed:*
commands show and select it by.
Workers polling one table run into each other's transactions; a conflict
says nothing about the message, so a poll sits through
:data:MAX_RETRIES of them in a row before it gives up, and a settle
tries again with a growing pause. A settle whose database went away waits
up to :data:RECONNECT_TIMEOUT seconds for it and tries once more: losing
an acknowledgement to a database restart would have the message handed
out, and handled, again.
A message is kept alive while it is held: from the moment it is handed
to the worker until it is settled, its claim is renewed every half of
redeliver_timeout, so a handler running longer than that is not
taken for dead and its message handed to another worker under it. A
renewal the database refuses is logged and tried again at the next one.
Receive from connection as the transport name, decoding with serializer.
get
async
¶
Yield each message as it falls due, waiting between polls that find none.
Never runs dry: a worker stops it by cancelling. A message is yielded
the moment it is claimed, with nothing awaited in between, so a
cancellation never loses one; and one lost anyway — the process
killed — is handed out again after redeliver_timeout.
A message that does not decode can never be handled, so it is rejected at once, its reason logged — never its body, which may carry anything a producer put in a message.
Raises:
-
TransportError–When the database refuses a poll for any reason but a conflict, or for a conflict :data:
MAX_RETRIEStimes in a row.
ack
async
¶
Delete the message envelope was received as: it was handled.
Raises:
-
TransportError–When
envelopewas not received from this transport, or the database refuses the delete.
reject
async
¶
Retry the message envelope was received as, or move it to the failure queue.
Raises:
-
TransportError–When
envelopewas not received from this transport, or the database refuses the update.
will_retry ¶
Tell whether rejecting envelope puts it back in its queue for another attempt.
decode ¶
Rebuild the envelope stored keeps, stamped with the row it came from.
A transport message id the envelope carried from an earlier transport is replaced by this row's, so the id it carries is the one that settles it here.
Raises:
-
MessageDecodingFailedError–When the row does not decode.
OrmSender ¶
Bases: SenderInterface
Writes each envelope it is handed into the table, due after its delay.
Send through connection as the transport name, encoding with serializer.
send
async
¶
Keep envelope, due after its :class:~xtr_messenger.DelayStamp, if any.
Returns it stamped with the id the row was given.
Raises:
-
MessageEncodingFailedError–When the envelope does not encode.
-
TransportError–When the database refuses the insert.
OrmTransport ¶
OrmTransport(connection: Connection, serializer: SerializerInterface, *, name: str = 'orm', clock: ClockInterface | None = None)
Bases: TransportInterface, ListableReceiverInterface, MessageCountAwareInterface, SetupableTransportInterface
Sends to, receives from and lists one queue of a messages table.
Beyond sending and receiving, the queue can be looked into — counted,
listed, one message found by id — and a message a long handler holds can
be kept from being handed out again. That is what the console's
messenger:setup-transports, messenger:stats, messenger:show
and messenger:failed:* commands read it through.
Use connection as the transport name.
Both halves encode and decode with serializer, so a producer and a
worker built from the same configuration always agree on the wire
format.
send
async
¶
Keep envelope until a worker claims it; see :meth:OrmSender.send.
get ¶
Yield each message as it falls due; see :meth:OrmReceiver.get.
ack
async
¶
Delete the message envelope was received as: it was handled.
reject
async
¶
Retry the message envelope was received as, or move it to the failure queue.
will_retry ¶
Tell whether rejecting envelope puts it back in its queue for another attempt.
keepalive
async
¶
Keep the message envelope was received as from being handed out again.
Restarts its redelivery clock: call it while a handler runs longer
than redeliver_timeout. seconds is how long the caller means to
wait before calling again, refused when longer than the timeout, which
would hand the message out in between.
Raises:
-
TransportError–When
secondsis longer thanredeliver_timeout,envelopewas not received from this transport, or the database refuses the update.
find
async
¶
Return the envelope kept as message_id in this queue, or None.
Raises:
-
MessageDecodingFailedError–When the row does not decode.
-
TransportError–When the database refuses the query.
all
async
¶
Yield the envelopes a worker could claim now, oldest first, limit at most.
Raises:
-
MessageDecodingFailedError–When a row does not decode.
-
TransportError–When the database refuses the query.
count
async
¶
Return how many messages a worker could claim now.
Raises:
-
TransportError–When the database refuses the query.
setup
async
¶
Create the messages table if it does not exist.
Raises:
-
TransportError–When the database refuses to create it.
OrmTransportFactory ¶
OrmTransportFactory(registry: ConnectionRegistry, serializer: SerializerInterface, clock: ClockInterface | None = None)
Bases: TransportFactoryInterface, PublisherClosingInterface
Builds orm:// transports, each on the connection its DSN names.
One transport per name, built the first time and kept: the bus sending to it and the worker draining it must hold the same one, as they would any transport holding a connection.
Not discovered by DSN scheme like a transport needing nothing: it needs the application's connections, which only the orm bundle holds. With a container, that bundle registers it when this package is installed; without one, pass it to the bus yourself::
MessageBusFactory(config, [OrmTransportFactory(registry, serializer)])
Build transports on registry's connections.
Parameters:
-
registry(ConnectionRegistry) –The application's connections, by name.
-
serializer(SerializerInterface) –What every transport encodes and decodes envelopes with.
-
clock(ClockInterface | None, default:None) –What every transport reads the time from; the system clock when left out.
create ¶
Return the transport for each name, building it the first time.
Building one opens nothing: the connection's engine is asked for the first time the transport runs a statement.
Raises:
-
InvalidOrmDsnError–When a DSN names a connection the registry does not hold.
-
UnknownTransportOptionError–When a transport carries a setting this one does not accept.
-
InvalidTransportOptionError–When a setting's value is not what it takes.
transport ¶
Return the transport name, building it from config the first time.
Raises:
-
InvalidOrmDsnError–When the DSN names a connection the registry does not hold.
-
UnknownTransportOptionError–When
configcarries a setting this transport does not accept. -
InvalidTransportOptionError–When a setting's value is not what it takes.
close_publishers
async
¶
Release the connections the transports this factory built listen on.
Sending holds no connection of its own — the engine's pool does, and the connection's owner disposes it — so this leaves only the connection a PostgreSQL worker listens for notifications on.
StoredMessage
dataclass
¶
StoredMessage(id: str, body: str, headers: Mapping[str, str], queue: str, created_at: datetime, available_at: datetime, delivered_at: datetime | None)
One row of the messages table, its envelope still encoded.
Attributes:
-
id(str) –The row's identifier, as the transport hands it out.
-
body(str) –The encoded message.
-
headers(Mapping[str, str]) –The encoded envelope's headers: its type and its stamps.
-
queue(str) –The queue it was sent to.
-
created_at(datetime) –When it was sent, in UTC.
-
available_at(datetime) –When it falls due, in UTC — later than
created_atfor a delayed message. -
delivered_at(datetime | None) –When a worker was last handed it, in UTC, or
Nonewhile no worker has been.