Skip to content

xtr_storage

Files on local disk, in memory or in object stores, behind one async interface.

A Storage wraps one adapter and turns paths into reads, writes, listings, copies and moves. The adapter decides where the bytes live — a directory on this machine, a dictionary in this process, a bucket on an object store — so code that stores files is written once and moved between them by its constructor.

Every call that touches a backend is awaited.

StorageAttributes

StorageAttributes = FileAttributes | DirectoryAttributes

One entry of a listing.

A closed union rather than a base class with optional fields: the two sides answer different questions — only a file has a size — and code that must handle both is made to say which it is holding, through is_file or a match.

FsspecAdapter

FsspecAdapter(root: str = '', *, mime_type_detector: MimeTypeDetectorInterface | None = None)

Bases: StorageAdapterInterface, ABC

Everything an fsspec backend does, bar the one line that builds it.

A subclass implements :meth:_create_filesystem and inherits a working adapter. Where its backend behaves differently — an object store that has no real directories, one that keeps a media type of its own — it overrides a hook rather than a whole method: :meth:_write_options to forward write settings, :meth:_is_hidden_entry to hide the markers it lists, :meth:_session_closer to shut a client down. Visibility is off by default, :meth:visibility and :meth:set_visibility raising until a subclass that has it overrides them.

Remember the root and the detector, and open nothing.

Parameters:

  • root (str, default: '' ) –

    What every path is stored under — a directory, a key prefix. Empty means the backend's own root, so the adapter is a pass-through.

  • mime_type_detector (MimeTypeDetectorInterface | None, default: None ) –

    How a media type is guessed from a name when the backend keeps none; a name-only detector by default.

read async

read(path: str) -> bytes

Return the whole file at path.

read_stream async

read_stream(path: str) -> AsyncIterator[bytes]

Yield the file at path a mebibyte at a time, reaching back for each.

The size is read once, up front, and the ranges are cut from it: a stream is a sequence of independent ranged reads, not a held handle. A file rewritten while it is being streamed therefore hands back a mixture — ranges from before the rewrite and ranges from after, or a short last range when it shrank — and one deleted mid-stream fails on the next range with :class:UnableToReadFileError. No backend this library speaks to offers a consistent snapshot across separate reads, so a caller who needs one writes a new object and swaps the name rather than rewriting a file something is reading.

write async

write(path: str, contents: bytes, config: Config) -> None

Write contents at path in one call.

write_stream async

write_stream(path: str, contents: AsyncIterable[bytes], config: Config) -> None

Write contents at path, in memory while small, through a temp above.

A stream that fits in :data:_SPOOL_MAX_SIZE is gathered in memory and written in one :meth:pipe_file; one that grows past it spills, from that point on, into a named temporary file the backend reads with :meth:put_file, so an upload of any size lands without ever holding the whole file in memory.

file_exists async

file_exists(path: str) -> bool

Return whether a file — not a directory — is at path.

directory_exists async

directory_exists(path: str) -> bool

Return whether a directory — not a file — is at path.

delete async

delete(path: str) -> None

Delete the file at path, treating a missing one as already gone.

delete_directory async

delete_directory(path: str) -> None

Delete the directory at path and its tree, missing being no failure.

create_directory async

create_directory(path: str, config: Config) -> None

Create the directory at path and its parents, once being enough.

list_contents

list_contents(path: str, deep: bool) -> AsyncIterator[StorageAttributes]

Return what is under path, one entry at a time.

move async

move(source: str, destination: str, config: Config) -> None

Move source onto destination, then settle its visibility.

copy async

copy(source: str, destination: str, config: Config) -> None

Copy source onto destination, then settle its visibility.

file_size async

file_size(path: str) -> FileAttributes

Return the size of the file at path.

last_modified async

last_modified(path: str) -> FileAttributes

Return when the file at path last changed, in whole epoch seconds.

mime_type async

mime_type(path: str) -> FileAttributes

Return what the file at path holds, from the backend or the name.

visibility async

visibility(path: str) -> FileAttributes

Raise: a plain filesystem adapter has no notion of visibility.

set_visibility async

set_visibility(path: str, visibility: Visibility) -> None

Raise: a plain filesystem adapter has no notion of visibility.

close async

close() -> None

Close the backend's session, if one was opened; safe to call again.

The bridge resets rather than being dropped, so the filesystem it wraps — and, for the in-memory backend, the files that live in it — survive the close while the networked session is released: a storage closed between requests is usable again, not stuck closed and not emptied.

GcsAdapter

GcsAdapter(bucket: str, prefix: str = '', *, project: str | None = None, token: str | None = None, endpoint_url: str | None = None, write_options: Mapping[str, object] | None = None, mime_type_detector: MimeTypeDetectorInterface | None = None)

Bases: FsspecAdapter

Files in a Google Cloud Storage bucket, behind the shared adapter contract.

The bucket, and an optional key prefix within it, are the root every path is placed under. Directories are the object-store fiction the shared helpers keep up; checksums, public URLs and signed URLs are real; visibility is refused, because the store offers no single answer to map onto.

Satisfies the checksum, public-url and temporary-url capability protocols structurally — by having their methods, not by inheriting them — so a storage discovers each at runtime.

Remember where and how to reach the bucket, and open nothing yet.

Parameters:

  • bucket (str) –

    The bucket every path lives in.

  • prefix (str, default: '' ) –

    A key prefix within the bucket that the caller never sees, so several storages may share one bucket without colliding.

  • project (str | None, default: None ) –

    The Google Cloud project the bucket belongs to, for the operations that need one; the backend's own default otherwise.

  • token (str | None, default: None ) –

    How the backend authenticates — a path to a credentials file, a token, or "anon" for an unauthenticated emulator.

  • endpoint_url (str | None, default: None ) –

    Where to reach the store, for pointing at an emulator; the real service otherwise.

  • write_options (Mapping[str, object] | None, default: None ) –

    Settings every write forwards unless a call overrides them — a content type, a cache-control header, custom metadata.

  • mime_type_detector (MimeTypeDetectorInterface | None, default: None ) –

    How a media type is guessed from a name when the object carries none; a name-only detector by default.

write async

write(path: str, contents: bytes, config: Config) -> None

Write contents at path, refusing a visibility this store cannot keep.

write_stream async

write_stream(path: str, contents: AsyncIterable[bytes], config: Config) -> None

Write path from chunks, refusing a visibility this store cannot keep.

create_directory async

create_directory(path: str, config: Config) -> None

Write the zero-byte marker that stands in for the directory at path.

The store has no directories, so an empty one is a key ending in a slash; the bucket root always exists, so a request for it does nothing. A visibility named for the directory is refused, as it is on a write.

directory_exists async

directory_exists(path: str) -> bool

Return whether a marker or any key stands under path.

checksum async

checksum(path: str, config: Config) -> str

Return the digest the store keeps beside the object at path.

The store records two: an MD5, handed back as the hex of the base64 the store keeps it in, and a CRC32C, handed back in the base64 the store keeps it in. Any other algorithm — and an object the store kept no digest of the asked-for kind for — is declined, which a storage takes as its cue to read the bytes and compute one itself.

Raises:

public_url async

public_url(path: str, config: Config) -> str

Return the lasting address of the object at path.

The address is the emulator endpoint, or the public service, followed by the bucket and the object's key, the key percent-escaped but for the slashes that structure it.

temporary_url async

temporary_url(path: str, expires_at: datetime, config: Config) -> str

Return a signed address for path, good until expires_at.

The deadline is turned into a whole number of seconds from now, never below one, and the store signs for that long. Anything the backend raises while signing becomes an :class:~xtr_storage.exception.UnableToGenerateTemporaryUrlError, the original kept as its cause.

Raises:

GenericFsspecAdapter

GenericFsspecAdapter(filesystem: AbstractFileSystem, root: str = '', *, closer: Callable[[AbstractFileSystem], Awaitable[None]] | None = None)

Bases: FsspecAdapter

A storage adapter over a filesystem somebody else built.

The instance is used exactly as given and never reconfigured: it is the caller's, built the way they wanted, and — unless a closer is handed over with it — it stays open when this adapter is closed so the rest of their code may go on using it. A caller that wants the filesystem shut with the adapter (the bundle, which builds the filesystem itself and owns its lifecycle) passes a closer, which the bridge runs once on close when a session was opened.

Reading, writing, listing, moving and metadata all work — they are the same calls whatever the backend. Visibility is not offered: see the module docstring for why.

Keep the filesystem and the root, and touch neither.

Parameters:

  • filesystem (AbstractFileSystem) –

    The filesystem to store through, already built.

  • root (str, default: '' ) –

    What every path is stored under — a directory, a key prefix. Empty stores at the filesystem's own root.

  • closer (Callable[[AbstractFileSystem], Awaitable[None]] | None, default: None ) –

    What closes the filesystem's session on close, for an owner that built it; None leaves it open, since the caller who built it owns it.

InMemoryAdapter

InMemoryAdapter(default_visibility: Visibility = Visibility.PUBLIC, *, mime_type_detector: MimeTypeDetectorInterface | None = None)

Bases: FsspecAdapter

Files in a dictionary, each with a visibility remembered beside it.

Every file this adapter is given is readable from it and from nowhere else: two adapters in one process share no store, so a test that writes has nothing to undo and cannot colour the next one.

Visibility is real here rather than refused. There is no permission to set on a dictionary entry, so the answer is simply remembered: a write records what its options asked for, or the adapter's default where they asked for nothing; a copy and a move carry it across; a delete forgets it, as does deleting a directory for everything beneath it.

Take the visibility a write falls back on, and hold no store yet.

Parameters:

  • default_visibility (Visibility, default: PUBLIC ) –

    What a file is worth when nothing said otherwise — readable by anyone, matching the openness of a store that only this process can reach anyway.

  • mime_type_detector (MimeTypeDetectorInterface | None, default: None ) –

    How a media type is guessed from a name; a name-only detector by default.

write async

write(path: str, contents: bytes, config: Config) -> None

Write contents at path and remember who may read it.

write_stream async

write_stream(path: str, contents: AsyncIterable[bytes], config: Config) -> None

Write path from chunks and remember who may read it.

move async

move(source: str, destination: str, config: Config) -> None

Move the file, and leave no visibility behind at the old path.

The base has already given the destination the visibility the rule calls for — the one the call named, or the source's — by the time this drops the source's, so what travelled is kept and what stayed is gone.

delete async

delete(path: str) -> None

Delete the file at path and forget who could read it.

delete_directory async

delete_directory(path: str) -> None

Delete the directory and forget the visibility of everything under it.

visibility async

visibility(path: str) -> FileAttributes

Return who may read the file at path.

Raises:

set_visibility async

set_visibility(path: str, visibility: Visibility) -> None

Record who may read the file at path.

Raises:

  • UnableToSetVisibilityError –

    When no file is stored there. A remembered answer about a file that does not exist would outlive nothing and mislead the next write.

LocalAdapter

LocalAdapter(directory: str | PathLike[str], *, visibility: VisibilityConverterInterface | None = None, link_handling: LinkHandling = LinkHandling.DISALLOW, mime_type_detector: MimeTypeDetectorInterface | None = None)

Bases: FsspecAdapter

A directory on this machine, everything under it and nothing above it.

The three things a real filesystem adds to the shared base are all here. Visibility becomes permission bits, through a converter the deployment may replace. Directories a write needs on the way are created with the mode that converter names rather than whatever the process umask leaves, since a storage told to keep its directories private must not depend on the shell that started the application. And symbolic links, which could point anywhere including outside the directory this storage is rooted in, are either left out of listings or refused outright.

Staying inside the root is enforced on every operation, not only on listings, and under either way of handling a link: each caller path is resolved through the filesystem before the backend is reached, and one that a link — its own last segment, or a directory on the way — carries outside the root is refused with :class:~xtr_storage.exception.SymbolicLinkEncounteredError, because such a path names a file the storage does not answer for. Skipping links decides what a listing shows; it does not make the tree above the root reachable through one.

Nothing is touched until the first operation: the root directory is created by the first write, so an application may declare a storage for a disk that is not mounted yet and pay nothing until it uses it.

Remember the directory and the rules, and touch the disk for nothing.

Parameters:

  • directory (str | PathLike[str]) –

    The directory every path is relative to. Made absolute here — reading the working directory, not the disk — because the paths a listing hands back are absolute and the root has to be strippable from them again. One that is itself a symbolic link is honoured — the link is the operator's own choice — and only links met below it are refused.

  • visibility (VisibilityConverterInterface | None, default: None ) –

    How a visibility becomes permission bits; the portable converter and its defaults when left out.

  • link_handling (LinkHandling, default: DISALLOW ) –

    What an operation does with a symbolic link it meets. Refusing by default: a link is rarely meant to be part of a storage, and silence about one is worse than a failure. Either way, no link is followed out of the root.

  • mime_type_detector (MimeTypeDetectorInterface | None, default: None ) –

    How a media type is guessed from a name, since a filesystem stores none.

write async

write(path: str, contents: bytes, config: Config) -> None

Write the bytes at path, directories and permissions as asked.

The bytes go to a hidden temporary file beside the target, which then replaces it in one step: a reader sees the old file or the new one, never part of either, and a failed write leaves the old file.

write_stream async

write_stream(path: str, contents: AsyncIterable[bytes], config: Config) -> None

Stream the bytes to path, replacing it whole once the stream ends.

create_directory async

create_directory(path: str, config: Config) -> None

Create the directory at path and its parents, with their own mode.

The mode comes from the call's directory visibility, or from the converter's default for directories when the call named none. Only the directories this call creates are given it: one that was already there keeps what it had, so creating the same directory twice is not a way to change its permissions.

visibility async

visibility(path: str) -> FileAttributes

Return who may read what is at path, from its permission bits.

set_visibility async

set_visibility(path: str, visibility: Visibility) -> None

Change the permission bits at path to the ones visibility means.

PathPrefixedAdapter

PathPrefixedAdapter(inner: StorageAdapterInterface, prefix: str)

Bases: StorageAdapterInterface, ChecksumProviderInterface, PublicUrlGeneratorInterface, TemporaryUrlGeneratorInterface

Another adapter, shown as if its files began one path further down.

Two applications sharing a bucket, one tenant per directory, a test suite that wants its own corner of a real backend: each is the same need, to hand out a storage whose root is somewhere inside another one. The caller works in plain paths and never learns the prefix, which is what makes the prefix changeable — it is configuration, not something spelled out at every call site and impossible to move later.

The translation goes both ways, and both halves matter. Paths go down with the prefix on. Listings come back with it off, because an entry a caller cannot pass back to the same storage is worse than no entry. And a failure the wrapped adapter raises is told again about the path the caller used, keeping the original as its __cause__ so the prefixed path is still there for whoever is debugging the backend.

Only the failures of an operation are retold that way. Anything else — a capability the backend does not have, an argument that makes no sense — passes through as it is: it is not about a path.

Wrap inner so everything it is asked for sits under prefix.

Parameters:

  • inner (StorageAdapterInterface) –

    The adapter that holds the files, under whatever root of its own it already has.

  • prefix (str) –

    The path this adapter's root is at, inside inner.

Raises:

  • InvalidArgumentError –

    When prefix names no path. A prefix that prefixes nothing would make this a pass-through, and a storage quietly writing to a shared root instead of its own corner is the kind of mistake that is found by finding the files.

  • PathTraversalDetectedError –

    When prefix carries a .. segment.

  • CorruptedPathDetectedError –

    When prefix carries a control character.

file_exists async

file_exists(path: str) -> bool

Return whether a file is at path under the prefix.

directory_exists async

directory_exists(path: str) -> bool

Return whether a directory is at path under the prefix.

write async

write(path: str, contents: bytes, config: Config) -> None

Write contents at path under the prefix.

write_stream async

write_stream(path: str, contents: AsyncIterable[bytes], config: Config) -> None

Write at path under the prefix from contents.

read async

read(path: str) -> bytes

Return the whole file at path under the prefix.

read_stream

read_stream(path: str) -> AsyncIterator[bytes]

Return the file at path under the prefix, in chunks.

delete async

delete(path: str) -> None

Delete the file at path under the prefix.

delete_directory async

delete_directory(path: str) -> None

Delete the directory at path under the prefix, and its tree.

create_directory async

create_directory(path: str, config: Config) -> None

Create the directory at path under the prefix.

set_visibility async

set_visibility(path: str, visibility: Visibility) -> None

Change who may read what is at path under the prefix.

visibility async

visibility(path: str) -> FileAttributes

Return who may read the file at path, described at the caller's path.

mime_type async

mime_type(path: str) -> FileAttributes

Return what the file at path holds, described at the caller's path.

last_modified async

last_modified(path: str) -> FileAttributes

Return when the file at path last changed, at the caller's path.

file_size async

file_size(path: str) -> FileAttributes

Return how many bytes the file at path holds, at the caller's path.

list_contents

list_contents(path: str, deep: bool) -> AsyncIterator[StorageAttributes]

Return what is under path, every entry back in the caller's terms.

move async

move(source: str, destination: str, config: Config) -> None

Move source to destination, both under the prefix.

copy async

copy(source: str, destination: str, config: Config) -> None

Copy source to destination, both under the prefix.

checksum async

checksum(path: str, config: Config) -> str

Return the digest the wrapped adapter keeps for path, if it keeps one.

Raises:

  • ChecksumAlgorithmNotSupportedError –

    When the wrapped adapter keeps no digest at all — the same answer it would give for an algorithm it does not hold, and what makes a storage read the file and compute the digest itself.

public_url async

public_url(path: str, config: Config) -> str

Return the lasting address the wrapped adapter gives the prefixed path.

Raises:

temporary_url async

temporary_url(path: str, expires_at: datetime, config: Config) -> str

Return the expiring address the wrapped adapter signs for the prefixed path.

Raises:

close async

close() -> None

Close the wrapped adapter; a prefix holds nothing of its own.

PortableVisibilityConverter

PortableVisibilityConverter(*, file_public: int = 420, file_private: int = 384, directory_public: int = 493, directory_private: int = 448, default_for_directories: Visibility = Visibility.PRIVATE)

Four modes and a default, chosen to work the same wherever they land.

The defaults are the ones an ordinary deployment wants: a public file is readable by everyone and writable by its owner, a private one is the owner's alone, and each directory adds the execute bit it needs to be entered. Directories default to private, because a write creates them without being asked and the safe answer to an unasked question is the narrower one.

Every value is adjustable, since what "public" has to mean depends on who serves the files: a web server running as another user needs the group or world bits that a site handing nothing out would rather not grant.

Reading a mode back is the inexact direction, and the rule is deliberately blunt: a mode reads as private when it is exactly the mode this converter writes for private, and as public otherwise. That keeps the two directions in agreement — set a visibility, read it back, get the same answer — while making no claim about a file this storage never wrote, which carries whatever umask the process that created it had. The only thing this converter can honestly say about such a file is that its bits are not the ones it locks a file with.

Record the four modes and which one an unasked-for directory gets.

The modes are kept as two lookup tables rather than four fields so that translating a visibility is a total mapping: a visibility this converter was not built for fails loudly on the lookup instead of silently taking one of the two branches an if would offer.

Parameters:

  • file_public (int, default: 420 ) –

    The mode a file readable by everyone carries.

  • file_private (int, default: 384 ) –

    The mode a file readable by its owner alone carries, and the only mode read back as private.

  • directory_public (int, default: 493 ) –

    The mode a directory everyone may enter carries.

  • directory_private (int, default: 448 ) –

    The mode a directory its owner alone may enter carries, and the only directory mode read back as private.

  • default_for_directories (Visibility, default: PRIVATE ) –

    The visibility a directory gets when the call that created it named none.

for_file

for_file(visibility: Visibility) -> int

Return the mode a file of this visibility carries.

for_directory

for_directory(visibility: Visibility) -> int

Return the mode a directory of this visibility carries.

inverse_for_file

inverse_for_file(mode: int) -> Visibility

Return private when mode is exactly the private file mode, else public.

inverse_for_directory

inverse_for_directory(mode: int) -> Visibility

Return private when mode is exactly the private directory mode, else public.

default_for_directories

default_for_directories() -> int

Return the mode a directory nobody named a visibility for carries.

ReadOnlyAdapter

ReadOnlyAdapter(inner: StorageAdapterInterface)

Bases: StorageAdapterInterface, ChecksumProviderInterface, PublicUrlGeneratorInterface, TemporaryUrlGeneratorInterface

Another adapter, published without any of its ways of changing anything.

What an application hands out when a backend is someone else's to write: an archive it may serve but not amend, a bucket another service owns, a directory of assets a build step fills. Every read goes through untouched and every attempt to change something fails at once, so a mistake surfaces here rather than as a permission error from the backend hours later — or, worse, as a write that succeeded where it should not have.

Each refusal is the failure of the operation that was asked for, with the reason naming why, so code that already handles "the write did not happen" needs nothing new to handle this.

The three capabilities a backend may have — a digest it keeps, a lasting address, a signed one — are read-only themselves, so they are passed on when the wrapped adapter has them. This adapter answers to all three regardless, because whether the one underneath does is not known until it is there; when it does not, each says so in the terms of that capability: a checksum reports that no such digest is kept, which is a storage's cue to read the file and compute one, and the two url capabilities refuse.

Wrap inner and keep only the half of it that reads.

Parameters:

  • inner (StorageAdapterInterface) –

    The adapter that holds the files. It is never asked to change anything, so it may well be one whose credentials could.

file_exists async

file_exists(path: str) -> bool

Return whether a file is at path, as the wrapped adapter sees it.

directory_exists async

directory_exists(path: str) -> bool

Return whether a directory is at path, as the wrapped adapter sees it.

read async

read(path: str) -> bytes

Return the whole file at path.

read_stream

read_stream(path: str) -> AsyncIterator[bytes]

Return the file at path in chunks, straight from the wrapped adapter.

list_contents

list_contents(path: str, deep: bool) -> AsyncIterator[StorageAttributes]

Return what is under path, straight from the wrapped adapter.

visibility async

visibility(path: str) -> FileAttributes

Return who may read the file at path.

mime_type async

mime_type(path: str) -> FileAttributes

Return what the file at path holds.

last_modified async

last_modified(path: str) -> FileAttributes

Return when the file at path last changed.

file_size async

file_size(path: str) -> FileAttributes

Return how many bytes the file at path holds.

write async

write(path: str, contents: bytes, config: Config) -> None

Refuse: nothing is written through a read-only adapter.

write_stream async

write_stream(path: str, contents: AsyncIterable[bytes], config: Config) -> None

Refuse before a single chunk is read, leaving the source untouched.

delete async

delete(path: str) -> None

Refuse, even for a path that holds nothing: a delete is a change.

delete_directory async

delete_directory(path: str) -> None

Refuse: a read-only adapter deletes no tree.

create_directory async

create_directory(path: str, config: Config) -> None

Refuse: a read-only adapter makes no directory.

set_visibility async

set_visibility(path: str, visibility: Visibility) -> None

Refuse: who may read a file is the owning application's to decide.

move async

move(source: str, destination: str, config: Config) -> None

Refuse: a move would both write and delete.

copy async

copy(source: str, destination: str, config: Config) -> None

Refuse: the destination is under the same read-only adapter.

checksum async

checksum(path: str, config: Config) -> str

Return the digest the wrapped adapter keeps for path, if it keeps one.

Raises:

  • ChecksumAlgorithmNotSupportedError –

    When the wrapped adapter keeps no digest at all — the same answer it would give for an algorithm it does not hold, and what makes a storage read the file and compute the digest itself.

public_url async

public_url(path: str, config: Config) -> str

Return the lasting address the wrapped adapter gives path.

Raises:

temporary_url async

temporary_url(path: str, expires_at: datetime, config: Config) -> str

Return the expiring address the wrapped adapter signs for path.

Raises:

close async

close() -> None

Close the wrapped adapter; a read-only view holds nothing of its own.

S3Adapter

S3Adapter(bucket: str, prefix: str = '', *, region: str | None = None, endpoint_url: str | None = None, key: str | None = None, secret: str | None = None, token: str | None = None, anon: bool = False, client_kwargs: Mapping[str, object] | None = None, config_kwargs: Mapping[str, object] | None = None, write_options: Mapping[str, object] | None = None, mime_type_detector: MimeTypeDetectorInterface | None = None)

Bases: FsspecAdapter

An S3 bucket, and an optional key prefix within it, as one storage.

What an object store adds to the shared base is all here. A directory is a zero-byte marker object, since the store keeps none of its own, and those markers are hidden from listings and answered for by an existence check that knows to look for them. Visibility is a canned access list — public-read or private — set on write and copy and read back from the object's grants. A checksum is the entity tag the store already holds, handed out only when "etag" is asked for so the storage falls back to hashing the bytes for anything else. And a file has two addresses: a lasting public one composed from the bucket and region, and a temporary signed one the backend produces.

Nothing is opened by construction: the backend package is imported and the client built on the first operation, so an application may declare a bucket it never touches and pay nothing for it. Credentials handed in are kept for that first build and never reach a message or a representation.

Remember the bucket, the prefix and the credentials, and open nothing.

Parameters:

  • bucket (str) –

    The bucket every object is stored in.

  • prefix (str, default: '' ) –

    A key prefix under which this storage's objects live, so one bucket may hold several storages. Empty for the whole bucket.

  • region (str | None, default: None ) –

    The bucket's region, used to compose a public url; defaults to us-east-1 there when left out.

  • endpoint_url (str | None, default: None ) –

    An alternate endpoint — a compatible store, a local test server — used for both requests and composed public urls.

  • key (str | None, default: None ) –

    The access key id, or None to let the backend find one.

  • secret (str | None, default: None ) –

    The secret access key, paired with key.

  • token (str | None, default: None ) –

    A session token, for temporary credentials.

  • anon (bool, default: False ) –

    Whether to reach the store without credentials.

  • client_kwargs (Mapping[str, object] | None, default: None ) –

    Extra keyword settings for the underlying client, merged after the region.

  • config_kwargs (Mapping[str, object] | None, default: None ) –

    Extra client configuration for the underlying client.

  • write_options (Mapping[str, object] | None, default: None ) –

    Default write settings; only the keys this backend understands are forwarded, the rest dropped.

  • mime_type_detector (MimeTypeDetectorInterface | None, default: None ) –

    How a media type is guessed from a name when a write names none; a name-only detector by default.

write async

write(path: str, contents: bytes, config: Config) -> None

Write contents at path, its media type guessed from the name.

write_stream async

write_stream(path: str, contents: AsyncIterable[bytes], config: Config) -> None

Stream contents to path, its media type guessed from the name.

create_directory async

create_directory(path: str, config: Config) -> None

Write the zero-byte marker object that stands in for the directory.

directory_exists async

directory_exists(path: str) -> bool

Return whether a marker or any object lives under the directory path.

visibility async

visibility(path: str) -> FileAttributes

Return who may read path, read from the object's access list.

Raises:

set_visibility async

set_visibility(path: str, visibility: Visibility) -> None

Set path's canned access list to the one visibility means.

Raises:

checksum async

checksum(path: str, config: Config) -> str

Return the entity tag the store holds for path, quotes stripped.

Parameters:

  • path (str) –

    The file to digest, already normalized.

  • config (Config) –

    The options in force; only "etag" is answered from here.

Returns:

  • str –

    The entity tag, without the quotes the store wraps it in.

Raises:

public_url async

public_url(path: str, config: Config) -> str

Return a lasting address for path, composed from bucket and region.

Parameters:

  • path (str) –

    The file to address, already normalized.

  • config (Config) –

    The options in force; unread here.

Returns:

  • str –

    An absolute url. When an endpoint was configured the address is

  • str –

    path-style under it; otherwise it is the virtual-hosted address in

  • str –

    the bucket's region, us-east-1 when none was named.

temporary_url async

temporary_url(path: str, expires_at: datetime, config: Config) -> str

Return a signed address for path, good until expires_at.

Parameters:

  • path (str) –

    The file to address, already normalized.

  • expires_at (datetime) –

    When the address stops working; a timezone-aware moment the storage has already vouched for.

  • config (Config) –

    The options in force; unread here.

Returns:

  • str –

    An absolute, signed url.

Raises:

StorageAdapterInterface

Bases: Protocol

One backend's side of the bargain: paths in, bytes and facts out.

This is deliberately narrower than what a storage offers its callers. A storage is the front door — it normalizes paths, layers options, refuses what it can refuse and falls back where it can. An adapter is handed the result of all that: a path already cleaned, and a :class:~xtr_storage.Config already merged. It never has to guess what a caller meant.

That is what makes a new backend a small piece of work, and what lets one conformance suite judge every backend by the same behaviour.

Anything a backend genuinely cannot do raises FeatureNotSupportedError rather than pretending: a silent no-op would leave a caller believing a file is private when anyone can read it.

The three metadata methods return :class:~xtr_storage.FileAttributes rather than the bare value, so an adapter that learned four facts from one round trip can hand all four back and let the storage keep what it needs.

file_exists async

file_exists(path: str) -> bool

Return whether a file — not a directory — is at path.

Raises:

directory_exists async

directory_exists(path: str) -> bool

Return whether a directory — not a file — is at path.

Raises:

write async

write(path: str, contents: bytes, config: Config) -> None

Write contents at path, making the directories on the way.

Parameters:

  • path (str) –

    Where the file goes.

  • contents (bytes) –

    What it holds.

  • config (Config) –

    The options in force, a visibility and a content type among the ones an adapter may read.

Raises:

write_stream async

write_stream(path: str, contents: AsyncIterable[bytes], config: Config) -> None

Write at path from chunks, without holding the whole file in memory.

Only an async source arrives here: a storage has already turned an open file or a plain iterable into one.

Parameters:

  • path (str) –

    Where the file goes.

  • contents (AsyncIterable[bytes]) –

    The chunks, in order.

  • config (Config) –

    The options in force.

Raises:

read async

read(path: str) -> bytes

Return the whole contents of the file at path.

Raises:

read_stream

read_stream(path: str) -> AsyncIterator[bytes]

Return the file's contents in chunks, reaching the backend as they are taken.

Not awaited, so an adapter may implement it as an async generator.

Raises:

delete async

delete(path: str) -> None

Delete the file at path, treating a missing file as already deleted.

Raises:

delete_directory async

delete_directory(path: str) -> None

Delete the directory at path and everything under it.

A directory that is not there is already deleted.

Raises:

create_directory async

create_directory(path: str, config: Config) -> None

Create the directory at path, and the ones leading to it.

Parameters:

  • path (str) –

    The directory to create.

  • config (Config) –

    The options in force, a directory visibility among them.

Raises:

set_visibility async

set_visibility(path: str, visibility: Visibility) -> None

Change who may read what is at path.

Raises:

visibility async

visibility(path: str) -> FileAttributes

Return who may read the file at path, as an attributes record.

Raises:

mime_type async

mime_type(path: str) -> FileAttributes

Return what the file at path holds, as an attributes record.

Raises:

last_modified async

last_modified(path: str) -> FileAttributes

Return when the file at path last changed, as an attributes record.

Raises:

file_size async

file_size(path: str) -> FileAttributes

Return the size of the file at path, as an attributes record.

Raises:

list_contents

list_contents(path: str, deep: bool) -> AsyncIterator[StorageAttributes]

Return what is under path, one entry at a time.

Not awaited, so an adapter may implement it as an async generator and a storage may wrap it in a lazy listing. A directory that is not there yields nothing rather than raising: a backend where directories are only implied cannot tell the two apart.

Parameters:

  • path (str) –

    The directory to look in; the adapter's root when empty.

  • deep (bool) –

    Whether to descend into the directories found.

move async

move(source: str, destination: str, config: Config) -> None

Move the file at source to destination, overwriting it.

Raises:

copy async

copy(source: str, destination: str, config: Config) -> None

Copy the file at source to destination, overwriting it.

Raises:

close async

close() -> None

Release whatever the adapter holds open, and stay usable afterwards.

A backend reached over the network keeps a session; one on local disk keeps nothing and does nothing here. Closing twice is not an error.

VisibilityConverterInterface

Bases: Protocol

Turns a visibility into permission bits, and permission bits back into one.

A filesystem has no notion of public and private — it has a number, and the number a deployment calls public is its own decision: one site serves files through a web server running as another user and needs the group bit, another hands nothing out and wants the narrowest mode that still works. A converter is where that decision lives, so the adapter above it only ever speaks of visibility.

Files and directories are asked separately because a directory needs its execute bit to be entered at all, so the mode that makes a file readable makes a directory useless.

The inverse direction is a guess and is meant to be: a filesystem holds no record of what a mode was intended to mean, and a file written by another process carries whatever that process's umask left. An implementation says which modes it recognises as private and reads the rest as public, so that setting a visibility and reading it back agrees with itself.

for_file

for_file(visibility: Visibility) -> int

Return the permission bits a file of this visibility should carry.

Parameters:

  • visibility (Visibility) –

    Who should be able to read the file.

Returns:

  • int –

    The mode, as the number a permission change takes.

for_directory

for_directory(visibility: Visibility) -> int

Return the permission bits a directory of this visibility should carry.

Parameters:

  • visibility (Visibility) –

    Who should be able to list and enter the directory.

Returns:

  • int –

    The mode, as the number a permission change takes.

inverse_for_file

inverse_for_file(mode: int) -> Visibility

Return the visibility a file's permission bits stand for.

Parameters:

  • mode (int) –

    The file's permission bits, with any type bits already removed.

Returns:

  • Visibility –

    The visibility those bits are read as.

inverse_for_directory

inverse_for_directory(mode: int) -> Visibility

Return the visibility a directory's permission bits stand for.

Parameters:

  • mode (int) –

    The directory's permission bits, with any type bits already removed.

Returns:

  • Visibility –

    The visibility those bits are read as.

default_for_directories

default_for_directories() -> int

Return the permission bits a directory nobody asked about should carry.

A write creates the directories on the way to its file, and the call that asked for it usually says nothing about them. This is the answer for that case, so those directories get a mode the deployment chose rather than whatever the process umask happens to be.

ChecksumProviderInterface

Bases: Protocol

An adapter that can answer for a digest without sending the file back.

Object stores keep a digest beside every object, and asking for it is one small request; computing the same digest means downloading the whole file. So an adapter that has one says so by satisfying this, and a storage checks at runtime before falling back to reading the bytes itself.

Optional by design: an adapter that does not implement this is not lacking anything — the fallback gives the same answer, more slowly.

checksum async

checksum(path: str, config: Config) -> str

Return the digest the backend holds for the file at path.

Parameters:

  • path (str) –

    The file to digest, already normalized.

  • config (Config) –

    The options in force, naming the algorithm among them.

Returns:

  • str –

    The digest, in the form the backend keeps it.

Raises:

Config

Config(options: Mapping[str, object] | None = None)

The options one operation runs with, layered rather than changed.

Options reach an adapter from two places: the ones a storage was built with, and the ones a single call passes. Both are open mappings — an adapter reads the keys below, and forwards the rest to its backend — so the layering, not a fixed field list, is what this class is for.

Nothing is ever changed in place. extend and with_defaults answer the only two questions there are — who wins, the newcomer or the incumbent — and each returns a new configuration, so a storage can hand its own options to one call without that call's options leaking into the next.

defaults = Config({Config.VISIBILITY: "private"})
for_this_call = defaults.extend({Config.VISIBILITY: "public"})  # public wins

The typed accessors are the reading half: a value arriving from a configuration file is text, and every caller would otherwise parse it again, differently.

Take a copy of options, so a caller's dictionary stays the caller's.

Parameters:

  • options (Mapping[str, object] | None, default: None ) –

    What this configuration holds; nothing when left out.

VISIBILITY class-attribute instance-attribute

VISIBILITY: Final[str] = 'visibility'

Who may read what is written: a :class:~xtr_storage.Visibility or its name.

DIRECTORY_VISIBILITY class-attribute instance-attribute

DIRECTORY_VISIBILITY: Final[str] = 'directory_visibility'

The same, for directories a write or a create has to make on the way.

RETAIN_VISIBILITY class-attribute instance-attribute

RETAIN_VISIBILITY: Final[str] = 'retain_visibility'

Whether a copy or a move keeps what the source had. True unless said otherwise.

MOVE_IDENTICAL_PATH class-attribute instance-attribute

MOVE_IDENTICAL_PATH: Final[str] = 'move_identical_path'

What a move from a path to itself does: an :class:~xtr_storage.IdenticalPathPolicy.

Nothing, unless said otherwise.

COPY_IDENTICAL_PATH class-attribute instance-attribute

COPY_IDENTICAL_PATH: Final[str] = 'copy_identical_path'

The same, for a copy.

CHECKSUM_ALGORITHM class-attribute instance-attribute

CHECKSUM_ALGORITHM: Final[str] = 'checksum_algorithm'

Which digest a checksum is taken with. "md5" unless said otherwise.

PUBLIC_URL class-attribute instance-attribute

PUBLIC_URL: Final[str] = 'public_url'

Where files are reachable from: one prefix, or several to spread them over.

ALLOW_RELATIVE_PATH_TRAVERSAL class-attribute instance-attribute

ALLOW_RELATIVE_PATH_TRAVERSAL: Final[str] = 'allow_relative_path_traversal'

Whether .. in a path may climb. True unless said otherwise.

get

get(key: str, default: object = None) -> object

Return the option key holds, or default when it holds none.

Parameters:

  • key (str) –

    The option's name.

  • default (object, default: None ) –

    What to answer when the option was never set.

Returns:

  • object –

    The value, untouched and unparsed.

extend

extend(options: Mapping[str, object]) -> Config

Return these options with options laid over them — the newcomer wins.

Parameters:

Returns:

  • Config –

    A new configuration; this one is unchanged.

with_defaults

with_defaults(defaults: Mapping[str, object]) -> Config

Return these options over defaults — what is already set wins.

Parameters:

  • defaults (Mapping[str, object]) –

    What to fall back on for options nobody set.

Returns:

  • Config –

    A new configuration; this one is unchanged.

with_setting

with_setting(key: str, value: object) -> Config

Return these options with key set to value.

Parameters:

  • key (str) –

    The option's name.

  • value (object) –

    What it is worth from now on.

Returns:

  • Config –

    A new configuration; this one is unchanged.

without_settings

without_settings(*keys: str) -> Config

Return these options with keys dropped, ignoring any that is not set.

Dropping rather than overwriting matters where absence means something of its own: a storage's default visibility must be gone, not None, for a copy that is asked to keep the source's.

Parameters:

  • *keys (str, default: () ) –

    The options to leave out.

Returns:

  • Config –

    A new configuration; this one is unchanged.

to_dict

to_dict() -> dict[str, object]

Return the options as a plain dictionary, copied so callers may keep it.

visibility_option

visibility_option(key: str) -> Visibility | None

Return the visibility key holds, or None when it holds none.

None is an answer, not a failure: it says the caller made no demand, and the adapter is free to use the backend's default.

Parameters:

  • key (str) –

    The option's name, usually :attr:VISIBILITY.

Returns:

  • Visibility | None –

    The visibility asked for, or None.

Raises:

bool_option

bool_option(key: str, default: bool) -> bool

Return the flag key holds, or default when it holds none.

Parameters:

  • key (str) –

    The option's name.

  • default (bool) –

    What the flag is worth where nobody set it.

Returns:

  • bool –

    The flag.

Raises:

identical_path_policy

identical_path_policy(key: str) -> IdenticalPathPolicy

Return the policy key holds, ignoring the operation when it holds none.

Parameters:

  • key (str) –

    :attr:COPY_IDENTICAL_PATH or :attr:MOVE_IDENTICAL_PATH.

Returns:

Raises:

str_option

str_option(key: str, default: str) -> str

Return the text key holds, or default when it holds none.

Parameters:

  • key (str) –

    The option's name.

  • default (str) –

    What to answer where nobody set it.

Returns:

  • str –

    The text.

Raises:

DirectoryAttributes dataclass

DirectoryAttributes(path: str, visibility: Visibility | None = None, last_modified: int | None = None, extra_metadata: Mapping[str, object] = dict[str, object]())

One directory, as a listing describes it.

There is no size and no MIME type here, and that is the point of a separate type: a directory has neither, so asking is a mistake the type system can catch rather than a field that is forever None.

Attributes:

  • path (str) –

    Where the directory is, relative to the storage's root, with no slash at either end — a backend that names directories a/b/ and one that names them a/b describe the same place.

  • visibility (Visibility | None) –

    Who may read it.

  • last_modified (int | None) –

    When it last changed, in whole seconds since the epoch.

  • extra_metadata (Mapping[str, object]) –

    Whatever else the backend said about it.

type property

type: str

What this entry is, for code sorting a listing without a type check.

is_file property

is_file: bool

Whether this entry is a file. Always false here.

is_dir property

is_dir: bool

Whether this entry is a directory. Always true here.

with_path

with_path(path: str) -> DirectoryAttributes

Return the same directory described at another path.

What a prefixing adapter or a mount manager needs: the backend answered about its own path, and the caller must be told about theirs.

Parameters:

  • path (str) –

    The path to describe this directory at.

Returns:

to_dict

to_dict() -> dict[str, object]

Return every field under its name, for a log line, a cache or a payload.

DirectoryListing

DirectoryListing(source: Callable[[], AsyncIterator[T]])

Entries a storage would yield, described rather than fetched.

A directory on an object store can hold more entries than fit in memory, and a caller usually wants a handful of them. So this holds no entries: it holds a source factory — a zero-argument callable that starts a fresh read — and :meth:filter and :meth:map return a new listing wrapping that source in another step. Nothing is read, no backend is called and no error is raised until something iterates.

listing = storage.list_contents("photos", deep=True)  # no call made yet
names = listing.filter(lambda entry: entry.is_file).map(lambda entry: entry.path)
async for name in names:  # the backend is read here, once
    print(name)

Iterating twice calls the source factory twice, which re-reads the backend: a listing is a question, and asking it again may rightly get a new answer. Hold :meth:to_list instead when one answer must serve twice.

Two operations cannot stay lazy. :meth:sort_by_path has to see every entry before it can yield the first, and :meth:to_list exists to gather them; both read the whole listing into memory when they run.

Take the factory that starts one read of the entries.

Parameters:

  • source (Callable[[], AsyncIterator[T]]) –

    Called with no arguments each time iteration begins, and expected to return a fresh iterator. A factory rather than an iterator so the listing can be read more than once, and so building it costs nothing.

filter

filter(predicate: Callable[[T], bool]) -> DirectoryListing[T]

Return the entries predicate accepts, deciding as they arrive.

Nothing is read here: the returned listing remembers the question and asks it of each entry while it is being iterated.

Parameters:

  • predicate (Callable[[T], bool]) –

    Answers whether one entry belongs in the result.

Returns:

map

map(transform: Callable[[T], R]) -> DirectoryListing[R]

Return each entry as transform describes it, one at a time.

Nothing is read here either — the transformation runs per entry during iteration, so a listing of a million files costs one entry of memory.

Parameters:

  • transform (Callable[[T], R]) –

    Turns one entry into whatever the caller wants instead, a path or a payload of its own.

Returns:

  • DirectoryListing[R] –

    A new listing over the transformed entries; this one is unchanged.

sort_by_path

sort_by_path() -> DirectoryListing[PathT]

Return the entries ordered by path — the one step that cannot stream.

Backends list in whatever order suits them, and code that compares two listings, or shows one to a person, needs an order it chose. The last entry may be the first alphabetically, so the returned listing reads the whole source into memory before it yields anything. Keep it late in a chain: a :meth:filter placed before it is what bounds the cost.

Returns:

  • DirectoryListing[PathT] –

    A new listing which, each time it is iterated, reads the source in

  • DirectoryListing[PathT] –

    full and yields its entries ordered by :attr:path.

to_list async

to_list() -> list[T]

Read every entry and return them, for when one answer must serve twice.

This is where a listing stops being a plan: it reads the source to the end and keeps the result, so nothing else has to worry that iterating again would ask the backend again.

Returns:

  • list[T] –

    Every entry, in the order the listing yields them.

ChecksumAlgorithmNotSupportedError

ChecksumAlgorithmNotSupportedError(location: str, algorithm: str)

Bases: UnableToProvideChecksumError

The backend keeps checksums, but not in the algorithm that was asked for.

An object store hands out the digests it happens to hold — an entity tag, an MD5 it recorded on upload — and nothing else. A storage treats this as "ask me differently", not as a failure: it reads the file and computes the digest itself. It surfaces only when a caller went to the backend directly.

Attributes:

  • location (str) –

    The path a checksum was wanted for, as the caller gave it.

  • algorithm (str) –

    The algorithm that was asked for.

  • reason (str) –

    Why there is none.

Record the path and the algorithm the backend does not keep.

CorruptedPathDetectedError

CorruptedPathDetectedError(path: str)

Bases: StorageError

A path held a control or formatting character.

Anything Unicode files under "other" — a null byte, an escape, a zero-width joiner — is refused rather than passed on. Such a character survives a comparison but not a filesystem, a url or a log line, so a path holding one means two names that look identical and are not.

Attributes:

  • path (str) –

    The path that was refused, as the caller gave it.

Record the path that cannot be used.

FeatureNotSupportedError

FeatureNotSupportedError(feature: Feature, adapter: str)

Bases: StorageError, NotImplementedError

The backend behind this adapter has no such capability.

Kept apart from every operation failure on purpose: a failed operation may work next time, this one never will. It says "configure something else", not "try again" — so a retry loop catching operation failures leaves it alone, and a caller that can do without the capability catches this one class and goes on.

Also a :class:NotImplementedError, which is what an implementation that does not implement something raises.

Attributes:

  • feature (Feature) –

    The capability that is missing.

  • adapter (str) –

    The name of the adapter class that does not have it.

Record which capability is missing from which adapter.

InvalidArgumentError

InvalidArgumentError(reason: str)

Bases: StorageError, ValueError

A storage, an adapter or a generator was given an argument it cannot use.

Raised where the argument is given — an empty prefix, a checksum algorithm this machine does not have, an expiry without a time zone — rather than on the first attempt to reach a backend.

Also a :class:ValueError, so code that already guards its configuration with except ValueError keeps working without learning a new exception.

Attributes:

  • reason (str) –

    What is wrong with the argument.

Record what is wrong with the argument.

InvalidStreamError

InvalidStreamError(received: str)

Bases: StorageError, TypeError

A write was given contents it cannot stream.

A stream is bytes arriving in pieces — an async iterable, a plain iterable or an open binary file. Text is not one: it would have to be encoded, and guessing which encoding is how files end up unreadable. The caller encodes and hands in bytes.

Also a :class:TypeError, because that is what handing in the wrong kind of thing is.

Attributes:

  • received (str) –

    The name of the type that was handed in.

Record what was handed in instead of a stream.

InvalidVisibilityError

InvalidVisibilityError(visibility: str)

Bases: StorageError, ValueError

A value was handed in as a visibility and is not one.

Visibility travels as a plain string through configuration files and request bodies, so it is parsed on the way in and anything unknown is refused there — before an adapter turns it into permissions or an access policy it cannot take back.

Also a :class:ValueError, so code already guarding its configuration with except ValueError keeps working.

Attributes:

  • visibility (str) –

    The value that was refused.

Record the value that is not a visibility.

MissingBackendError

MissingBackendError(package: str, extra: str)

Bases: StorageError, ImportError

The package this adapter talks to its backend through is not installed.

Backends live behind extras so an application installs only the ones it reaches. An adapter imports its package the first time it is used, and turns a failed import into this — an error naming the extra to install, rather than a traceback about a module nobody asked for.

Also an :class:ImportError, so it reads as what it is.

Attributes:

  • package (str) –

    The package that is missing.

  • extra (str) –

    The extra of this distribution that installs it.

Record the missing package and the extra that brings it.

PathTraversalDetectedError

PathTraversalDetectedError(path: str)

Bases: StorageError

A path walked up past the root of the storage.

Every path is resolved before it reaches a backend, and one whose .. segments take it above the root is refused there — a storage answers for what it holds and for nothing beside it. A storage may also refuse .. outright, which raises this for any path carrying one.

Attributes:

  • path (str) –

    The path that was refused, as the caller gave it.

Record the path that climbed out.

StorageError

Bases: Exception

Base class for every error raised by this library.

Catch this to handle anything storing a file can go wrong with; catch a subclass to handle one cause. Every subclass carries the data a caller needs as typed attributes and composes its own message from them, so nothing has to be parsed back out of a string.

A message is built from paths and from the reason the backend gave. It never carries a key, a token or a connection string: an error travels into logs and responses, and credentials must not travel with it.

StorageOperationFailedError

Bases: StorageError

Something a storage was asked to do failed; the operation says what.

This is the line between "the backend refused this attempt" and every other kind of error — a path that is not a path, a feature the backend does not have, an argument that makes no sense. Code that retries, or that turns a failure into a response, catches this one class and reads :attr:operation rather than listing every subclass.

Whatever the backend raised is chained as __cause__.

Attributes:

  • operation (Operation) –

    What was being attempted. Every concrete subclass sets it; reading it off this class itself raises :class:AttributeError, which is how a subclass that forgot to set it is found.

SymbolicLinkEncounteredError

SymbolicLinkEncounteredError(location: str)

Bases: StorageError

A listing ran into a symbolic link and was told to refuse them.

A link can point anywhere, including outside the storage, so following one would quietly hand out what the storage does not hold. An adapter that meets one either skips it or raises this, depending on how it was built.

Attributes:

  • location (str) –

    The link that was met, as the storage names it.

Record the link that was refused.

UnableToCheckDirectoryExistenceError

UnableToCheckDirectoryExistenceError(location: str)

Bases: UnableToCheckExistenceError

Asking whether a directory is there failed.

The class name is what tells a reader a directory was being looked for; the message stays the same as its parent's so the two read alike in a log.

Attributes:

  • location (str) –

    The path that was checked, as the caller gave it.

UnableToCheckExistenceError

UnableToCheckExistenceError(location: str)

Bases: StorageOperationFailedError

Asking whether a path is there failed.

Not an answer of "no": the backend could not be asked at all. The two subclasses say whether a file or a directory was being looked for, and this class catches both.

Attributes:

  • location (str) –

    The path that was checked, as the caller gave it.

Record the path that could not be checked.

UnableToCheckFileExistenceError

UnableToCheckFileExistenceError(location: str)

Bases: UnableToCheckExistenceError

Asking whether a file is there failed.

The class name is what tells a reader a file was being looked for; the message stays the same as its parent's so the two read alike in a log.

Attributes:

  • location (str) –

    The path that was checked, as the caller gave it.

UnableToCopyFileError

UnableToCopyFileError(source: str, destination: str, reason: str = '')

Bases: StorageOperationFailedError

Copying a file failed; the destination may be half-written.

A copy across backends is a read streamed into a write, so a failure part way through leaves whatever reached the destination. The source is never touched.

Attributes:

  • source (str) –

    The path that was copied from, as the caller gave it.

  • destination (str) –

    The path that was copied to, as the caller gave it.

  • reason (str) –

    What the backend said, empty when it said nothing.

Record both paths and why the copy failed.

UnableToCreateDirectoryError

UnableToCreateDirectoryError(location: str, reason: str = '')

Bases: StorageOperationFailedError

Creating a directory failed.

Attributes:

  • location (str) –

    The directory that was created, as the caller gave it.

  • reason (str) –

    What the backend said, empty when it said nothing.

Record the directory and why creating it failed.

UnableToDeleteDirectoryError

UnableToDeleteDirectoryError(location: str, reason: str = '')

Bases: StorageOperationFailedError

Deleting a directory and everything under it failed.

The deletion is not atomic anywhere: part of the tree may be gone. A caller that must end up with nothing there deletes again rather than assuming either outcome.

Attributes:

  • location (str) –

    The directory that was deleted, as the caller gave it.

  • reason (str) –

    What the backend said, empty when it said nothing.

Record the directory and why deleting it failed.

UnableToDeleteFileError

UnableToDeleteFileError(location: str, reason: str = '')

Bases: StorageOperationFailedError

Deleting a file failed while the file was there.

Deleting what is not there is not a failure: a delete asks for the file to be gone, and it is. This is raised only when the backend refused.

Attributes:

  • location (str) –

    The path that was deleted, as the caller gave it.

  • reason (str) –

    What the backend said, empty when it said nothing.

Record the path and why deleting it failed.

UnableToGeneratePublicUrlError

UnableToGeneratePublicUrlError(location: str, reason: str = '')

Bases: StorageError

A file has no address anyone can fetch it from.

Usually configuration rather than breakage: nothing told the storage what its files are served under, and the backend does not know either. A chain of generators raises this once every one of them has declined.

Attributes:

  • location (str) –

    The path a url was wanted for, as the caller gave it.

  • reason (str) –

    Why there is none, empty when nothing more is known.

Record the path and why it has no public url.

UnableToGenerateTemporaryUrlError

UnableToGenerateTemporaryUrlError(location: str, reason: str = '')

Bases: StorageError

A file has no address that lets the bearer fetch it until a deadline.

A backend signs such a url with the credentials it holds, so this covers both "this backend does not sign" and "signing failed". The message never repeats what was signed with.

Attributes:

  • location (str) –

    The path a url was wanted for, as the caller gave it.

  • reason (str) –

    Why there is none, empty when nothing more is known.

Record the path and why it has no temporary url.

UnableToListContentsError

UnableToListContentsError(location: str, deep: bool)

Bases: StorageOperationFailedError

Listing a directory failed, at its start or part way through.

A listing is lazy, so this can surface on any step of the iteration, not only on the call that asked for it. :attr:deep is carried because a shallow listing and a deep one are different requests to the backend and fail for different reasons.

Attributes:

  • location (str) –

    The directory that was listed, as the caller gave it.

  • deep (bool) –

    Whether everything below it was being listed too.

Record the directory and how far the listing reached.

UnableToMoveFileError

UnableToMoveFileError(source: str, destination: str, reason: str = '')

Bases: StorageOperationFailedError

Moving a file failed; the source may still be there.

A move across backends is a copy followed by a delete, so a failure can leave the file in both places or in neither. The error names both paths for that reason: which one to look at depends on where it stopped.

Attributes:

  • source (str) –

    The path that was moved from, as the caller gave it.

  • destination (str) –

    The path that was moved to, as the caller gave it.

  • reason (str) –

    What the backend said, empty when it said nothing.

Record both paths and why the move failed.

UnableToProvideChecksumError

UnableToProvideChecksumError(location: str, reason: str = '')

Bases: StorageError

A file's checksum could neither be read from the backend nor computed.

Not an operation failure: a checksum is a capability, and a storage that cannot get one from the backend falls back to reading the file and hashing it. This is what is left when that fails too.

Attributes:

  • location (str) –

    The path a checksum was wanted for, as the caller gave it.

  • reason (str) –

    Why there is none, empty when nothing more is known.

Record the path and why no checksum could be produced.

UnableToReadFileError

UnableToReadFileError(location: str, reason: str = '')

Bases: StorageOperationFailedError

A file could not be read: it is not there, or the backend refused.

Missing and unreadable are one error on purpose. An object store answers both the same way, so a caller that wants to know whether a file is there asks file_exists rather than reading it and reading the failure.

Attributes:

  • location (str) –

    The path that was read, as the caller gave it.

  • reason (str) –

    What the backend said, empty when it said nothing.

Record the path and why reading it failed.

UnableToResolveMountError

UnableToResolveMountError(location: str, reason: str)

Bases: StorageError

A location could not be routed to one of the mounted storages.

Raised before anything is attempted: the location carries no mount name, carries an empty one, or names a storage nothing was mounted under. It is a mistake in the location, not a failure of the storage behind it, which is why it is not one of the operation failures.

Attributes:

  • location (str) –

    The location that was routed, as the caller gave it.

  • reason (str) –

    What is wrong with it.

Record the location and why it routes nowhere.

UnableToRetrieveMetadataError

UnableToRetrieveMetadataError(location: str, metadata_type: str, reason: str = '')

Bases: StorageOperationFailedError

A file's size, type, visibility or last write could not be read.

One error for all four, with :attr:metadata_type saying which: they fail together — a missing file has none of them — and a caller asking for two would otherwise catch two exception types for one cause. The classmethods build each kind, so no call site has to spell the name.

Attributes:

  • location (str) –

    The path the metadata was read for, as the caller gave it.

  • metadata_type (str) –

    Which piece was asked for — last_modified, visibility, file_size or mime_type.

  • reason (str) –

    What the backend said, empty when it said nothing.

Record the path, what was asked for, and why it could not be read.

last_modified classmethod

last_modified(location: str, reason: str = '') -> Self

Build the failure for the moment a file was last written.

visibility classmethod

visibility(location: str, reason: str = '') -> Self

Build the failure for who may read a file.

file_size classmethod

file_size(location: str, reason: str = '') -> Self

Build the failure for how many bytes a file holds.

mime_type classmethod

mime_type(location: str, reason: str = '') -> Self

Build the failure for what kind of content a file holds.

UnableToSetVisibilityError

UnableToSetVisibilityError(location: str, reason: str = '')

Bases: StorageOperationFailedError

Changing who may read a file or a directory failed.

A backend that has no notion of visibility at all raises :class:~xtr_storage.exception.FeatureNotSupportedError instead: this one means the backend has it and the attempt failed.

Attributes:

  • location (str) –

    The path whose visibility was set, as the caller gave it.

  • reason (str) –

    What the backend said, empty when it said nothing.

Record the path and why its visibility could not be set.

UnableToWriteFileError

UnableToWriteFileError(location: str, reason: str = '')

Bases: StorageOperationFailedError

Writing a file failed, whether or not anything was written.

A backend that fails midway may leave a partial object behind; the error says where to look, not what survived, because only the backend knows.

Attributes:

  • location (str) –

    The path that was written to, as the caller gave it.

  • reason (str) –

    What the backend said, empty when it said nothing.

Record the path and why writing it failed.

Feature

Bases: StrEnum

A capability an adapter either has or does not.

An adapter that cannot do one of these raises :class:~xtr_storage.exception.FeatureNotSupportedError naming it, rather than failing the operation: the difference is between "this backend never does that" and "that attempt failed", and only the first is worth changing the configuration over.

Visibility is the only capability named this way. A checksum, a public url and a temporary url each refuse with their own error — respectively :class:~xtr_storage.exception.ChecksumAlgorithmNotSupportedError, :class:~xtr_storage.exception.UnableToGeneratePublicUrlError and :class:~xtr_storage.exception.UnableToGenerateTemporaryUrlError — so there is nothing for them to name here.

FileAttributes dataclass

FileAttributes(path: str, file_size: int | None = None, visibility: Visibility | None = None, last_modified: int | None = None, mime_type: str | None = None, extra_metadata: Mapping[str, object] = dict[str, object]())

One file, as a listing or a metadata answer describes it.

Everything but the path is optional because backends answer different questions for the same file: a listing may carry sizes and no MIME type, and a metadata call fills in the one field it was asked for, leaving the rest alone rather than paying for a second round trip. None therefore means "not asked, or not known here" — never "empty".

Attributes:

  • path (str) –

    Where the file is, relative to the storage's root, with no leading slash: one file has one path, whoever wrote it.

  • file_size (int | None) –

    Its size in bytes.

  • visibility (Visibility | None) –

    Who may read it.

  • last_modified (int | None) –

    When it last changed, in whole seconds since the epoch.

  • mime_type (str | None) –

    What it holds.

  • extra_metadata (Mapping[str, object]) –

    Whatever else the backend said about it.

type property

type: str

What this entry is, for code sorting a listing without a type check.

is_file property

is_file: bool

Whether this entry is a file. Always true here.

is_dir property

is_dir: bool

Whether this entry is a directory. Always false here.

with_path

with_path(path: str) -> FileAttributes

Return the same file described at another path.

What a prefixing adapter or a mount manager needs: the backend answered about its own path, and the caller must be told about theirs, with every other field the backend gave kept intact.

Parameters:

  • path (str) –

    The path to describe this file at.

Returns:

to_dict

to_dict() -> dict[str, object]

Return every field under its name, for a log line, a cache or a payload.

IdenticalPathPolicy

Bases: StrEnum

What to do when a copy or a move is asked to go from a path to itself.

Backends disagree: one rewrites the file with itself, another refuses, a third truncates it on the way. The caller decides instead of the backend, so the same code behaves the same however the bytes are stored.

:attr:IGNORE is what a call that chose nothing gets, because it is the only answer that cannot destroy the file. A move an object store cannot do in one step is a copy followed by a delete of the source — and when the two paths are one path, the delete removes what the copy just wrote. The two other answers stay available to a caller who wants them, named per call or per storage.

The members are their strings, so a configuration carries "fail".

Attributes:

  • TRY –

    Hand it to the adapter and live with the backend's answer, whatever that is — a rewrite, a refusal, or a file that is no longer there.

  • FAIL –

    Refuse before the backend is touched.

  • IGNORE –

    Return as though it had been done, touching nothing — unless there is no file at the path, which is reported as a missing source like any other transfer, because nothing was done and no file appeared. The default.

LinkHandling

Bases: StrEnum

What an operation does with a symbolic link it meets.

A link may point anywhere, including outside the directory a storage is rooted in, so following one would hand out a file that storage never held. Neither answer is right for everyone — a deployment that plants links on purpose wants them left out quietly, one that plants none wants to hear about the first — so the adapter is told which to give.

Both answers keep the storage inside its root. This setting decides what a listing does with a link; neither value makes the tree above the root reachable through one, so a read, a write or a delete naming a path a link carries outside the root is refused either way.

Attributes:

  • SKIP –

    Leave links out of the listing. A link is hidden, not a door: one that resolves outside the root is still refused on a direct operation.

  • DISALLOW –

    Raise on the first one met.

ExtensionMimeTypeDetector

ExtensionMimeTypeDetector()

Bases: MimeTypeDetectorInterface

Guesses a media type from the extension, deterministically across machines.

The standard library's shared registry reads system files (such as /etc/mime.types) on first use, so the same extension can resolve differently on two machines. This detector instead owns a private MimeTypes(filenames=()) instance: passing no filenames keeps it from reading any system file, while it still loads the interpreter's built-in table, so the mapping depends only on the standard library, not on the host. A handful of extensions are then added explicitly to close the remaining gap between interpreter versions.

The lookup is case-insensitive — the underlying table folds the extension — and an extension it does not recognise yields None rather than a guess.

Build the private table and register the explicit extensions.

detect_from_path

detect_from_path(path: str) -> str | None

Return the media type path's extension names, or None.

MimeTypeDetectorInterface

Bases: Protocol

Turns a path into a media type without ever opening the file.

An adapter asks for a file's media type when the backend does not store one of its own. The path is all it has — the bytes may live on another machine and fetching them only to sniff a type would be a second round trip — so a detector decides from the name and returns None when the name says nothing it recognises.

detect_from_path

detect_from_path(path: str) -> str | None

Return the media type path names, or None when it is unknown.

Parameters:

  • path (str) –

    The location of the file, whether or not it exists. Only its trailing extension is read.

Returns:

  • str | None –

    A media type such as "image/svg+xml", or None when the

  • str | None –

    extension maps to nothing known.

MountManager

MountManager(storages: Mapping[str, StorageOperatorInterface], config: Mapping[str, object] | None = None)

Several storages behind one, picked by the name a location starts with.

An application that keeps uploads in one place, generated reports in another and a scratch area in a third ends up passing three storages around, and every function that touches a file has to be told which one it is for. A mount manager is the one object instead: the storages are named once, and from then on the name travels with the path — "uploads://avatars/1.png" — so a location is enough to act on.

Every location reads <name>://<path>. There is no default mount, because a location that quietly went somewhere would be exactly the bug this exists to remove, so one naming no mount, an empty mount or a mount nothing was put under is refused with :class:~xtr_storage.exception.UnableToResolveMountError before anything is attempted.

Routing is all this does, bar one thing: a copy or a move between two different storages, which no backend can be asked to perform, and which :meth:copy and :meth:move describe.

Mount storages under their names and fix the standing options.

Parameters:

  • storages (Mapping[str, StorageOperatorInterface]) –

    What each mount name stands for. Copied, so a later change to the caller's mapping cannot re-route live code.

  • config (Mapping[str, object] | None, default: None ) –

    Options every call through this manager starts from. A call's own are laid over them, and the result reaches the owning storage as that call's options.

file_exists async

file_exists(location: str) -> bool

Return whether a file is at location.

directory_exists async

directory_exists(location: str) -> bool

Return whether a directory is at location.

has async

has(location: str) -> bool

Return whether anything at all is at location, file or directory.

read async

read(location: str) -> bytes

Return the whole contents of the file at location.

read_stream

read_stream(location: str) -> AsyncIterator[bytes]

Return the file's contents in chunks, from the storage that holds it.

list_contents

list_contents(location: str, deep: bool = False) -> DirectoryListing[StorageAttributes]

Return a lazy listing of what is under location.

Every entry comes back at its full location — "uploads://a/b.png" rather than "a/b.png" — so what a listing yields can be handed straight back to :meth:read or :meth:copy without the caller having to remember which mount it came from.

The mount is resolved here rather than on first iteration: routing nowhere is a mistake in the call, and an error is worth more where the call is than wherever the listing is eventually read. There is no default location: a bare "" names no mount, so the caller always says which storage to list.

last_modified async

last_modified(path: str) -> int

Return when the file at path last changed, in whole epoch seconds.

file_size async

file_size(path: str) -> int

Return the size in bytes of the file at path.

mime_type async

mime_type(path: str) -> str

Return what the file at path holds.

visibility async

visibility(path: str) -> Visibility

Return who may read the file at path.

write async

write(location: str, contents: bytes, config: Mapping[str, object] | None = None) -> None

Write contents at location, replacing whatever was there.

write_stream async

write_stream(location: str, contents: AsyncIterable[bytes] | Iterable[bytes] | BinaryIO, config: Mapping[str, object] | None = None) -> None

Write at location from a source read in pieces.

set_visibility async

set_visibility(path: str, visibility: Visibility | str) -> None

Change who may read the file or directory at path.

delete async

delete(location: str) -> None

Delete the file at location, treating a missing one as already gone.

delete_directory async

delete_directory(location: str) -> None

Delete the directory at location and everything under it.

create_directory async

create_directory(location: str, config: Mapping[str, object] | None = None) -> None

Create the directory at location, and the ones leading to it.

copy async

copy(source: str, destination: str, config: Mapping[str, object] | None = None) -> None

Copy the file at source to destination, overwriting it.

Between two locations on the same storage this is that storage's own copy, which is where a backend does the work itself instead of sending every byte out and back. Between two storages it cannot be: the file is read out of one and written into the other a chunk at a time, and its visibility is read separately and named in the write, because nothing else carries it across. A visibility the call names wins over the source's, retain_visibility: False keeps none, a source that has no notion of visibility contributes none, and a destination that has none ignores what it is given.

Raises:

  • UnableToResolveMountError –

    When either location routes nowhere.

  • UnableToCopyFileError –

    When the copy failed. Across two storages the read, the write and the visibility lookup all end here, carrying what actually failed as __cause__.

move async

move(source: str, destination: str, config: Mapping[str, object] | None = None) -> None

Move the file at source to destination, overwriting it.

Between two locations on the same storage this is that storage's own move. Between two storages it is a copy followed by a delete, which is the only thing it can be, and the delete is what makes it a move rather than a copy: when it fails the file is still at source, and this says so instead of reporting a move that did not happen. Nothing is rolled back — the copy stands — so a caller that retries moves the same file again rather than losing it.

Raises:

public_url async

public_url(path: str, config: Mapping[str, object] | None = None) -> str

Return a lasting address for the file at path, from its own storage.

temporary_url async

temporary_url(path: str, expires_at: datetime, config: Mapping[str, object] | None = None) -> str

Return an expiring address for the file at path, from its own storage.

checksum async

checksum(path: str, config: Mapping[str, object] | None = None) -> str

Return a digest of the file at path, from the storage that holds it.

close async

close() -> None

Close every mounted storage, whatever any one of them does about it.

One storage refusing to close must not leave the rest open — a pool nobody shut is what outlives the process — so each is closed in turn, and every failure is raised together, in an :class:ExceptionGroup, once they all have been, so none is lost behind another. A storage with nothing to close is passed over.

Operation

Bases: StrEnum

The operation an :class:~xtr_storage.exception.StorageOperationFailedError was in.

Every failed operation carries one of these as a class attribute, so code catching the one base error can still branch on what was attempted instead of listing a dozen exception types. The values are stable, lower-case names, fit to log or to put in a response body.

PathNormalizerInterface

Bases: Protocol

Resolves a caller's path to the single form every backend agrees on.

A path reaches the library the way a caller wrote it — with backslashes, doubled slashes, . and .. segments, leading and trailing slashes. A backend must never see that variety: two paths that name the same file have to arrive identical, and one that climbs above the root must be caught before a filesystem or an object store is asked about it. Every path crosses this boundary exactly once, so the backends downstream receive plain paths and never guard against these cases themselves.

normalize_path

normalize_path(path: str) -> str

Return path reduced to its plain, root-relative form.

Parameters:

  • path (str) –

    The path as the caller gave it.

Returns:

  • str –

    The path with separators unified, redundant segments removed and no

  • str –

    leading or trailing separator.

PathPrefixer

PathPrefixer(prefix: str, separator: str = '/')

Joins a caller's path to a backend's root and strips it back off again.

A backend often stores everything under one root — a directory, a bucket key prefix — that the caller never sees. This prefixer owns that translation in one place: every path going down to the backend is prefixed, every path coming back up in a listing has the prefix removed, so the rest of the library works in caller-relative paths and the root lives here alone.

The root is stored once, in a normalized shape, so joining is a plain string concatenation with no per-call trimming. A non-empty root always ends with the separator; an empty root stays empty so a prefixer with no root is a transparent pass-through; a root that is only the separator stays the separator, keeping a leading-slash convention intact.

Store the root in the shape every method below relies on.

Parameters:

  • prefix (str) –

    The root every path is placed under. Trailing separators are folded so "a" and "a/" behave identically.

  • separator (str, default: '/' ) –

    The character that joins segments in this backend.

prefix_path

prefix_path(path: str) -> str

Place path under the root, dropping any leading separator it carries.

strip_prefix

strip_prefix(path: str) -> str

Return path with the stored root removed from its front.

prefix_directory_path

prefix_directory_path(path: str) -> str

Place a directory path under the root, ending it with the separator.

An empty result — an empty root joined to an empty path — stays empty rather than becoming a lone separator, and a result that already ends with the separator is left as it is.

WhitespacePathNormalizer

WhitespacePathNormalizer(allow_relative_path_traversal: bool = True)

Bases: PathNormalizerInterface

Reduces a path to one canonical form and refuses the paths a backend can't trust.

The name keeps the intent explicit: the normalizer folds away the "whitespace" of path writing — the backslashes, the doubled and edge separators, the . and .. segments — so that two spellings of the same file collapse to one string. It resolves in memory, never touching a filesystem, which is why a .. is settled by counting segments rather than by asking where a symlink points.

Two paths are refused rather than resolved. One that carries a control or formatting character (anything Unicode files under category C) cannot be a real name — such a character survives a comparison but not a filesystem, a url or a log line — so it raises rather than travelling on. One whose .. segments walk above the root would name a file the storage does not answer for, so it raises too; a storage may also forbid .. outright, in which case any parent segment is refused whether or not there is something to pop.

Record whether a resolvable .. segment is allowed.

Parameters:

  • allow_relative_path_traversal (bool, default: True ) –

    When true, a .. pops the segment before it and only a .. with nothing left to pop is refused. When false, any .. is refused outright.

normalize_path

normalize_path(path: str) -> str

Return path in canonical form, or raise if it cannot be trusted.

Storage

Storage(adapter: StorageAdapterInterface, config: Mapping[str, object] | None = None, *, path_normalizer: PathNormalizerInterface | None = None, public_url_generator: PublicUrlGeneratorInterface | None = None, temporary_url_generator: TemporaryUrlGeneratorInterface | None = None)

One backend behind the whole reading-and-writing contract.

A storage is what an application holds and passes around: it turns a caller's path into a backend operation, and it is the single place a path is cleaned, options are layered, a stream is made uniform and a capability a backend lacks is either worked around or refused. The adapter it wraps receives the result of all that and never has to second-guess a caller.

The options a storage is built with are the standing defaults; the mapping a single call passes is laid over them for that call alone, so one upload can be public without changing what the next one does.

Wrap adapter and fix the defaults every call starts from.

Parameters:

  • adapter (StorageAdapterInterface) –

    The backend to drive. It is handed normalized paths and a merged :class:~xtr_storage.Config, never a raw caller path.

  • config (Mapping[str, object] | None, default: None ) –

    The standing options a call's own options are laid over.

  • path_normalizer (PathNormalizerInterface | None, default: None ) –

    How caller paths are reduced to the form the backend sees; a whitespace normalizer by default, reading whether .. may climb from the config.

  • public_url_generator (PublicUrlGeneratorInterface | None, default: None ) –

    A generator put in front of the backend for lasting addresses; consulted before the backend's own.

  • temporary_url_generator (TemporaryUrlGeneratorInterface | None, default: None ) –

    The same, for expiring addresses.

file_exists async

file_exists(location: str) -> bool

Return whether a file is at location.

directory_exists async

directory_exists(location: str) -> bool

Return whether a directory is at location.

has async

has(location: str) -> bool

Return whether anything at all is at location, file or directory.

read async

read(location: str) -> bytes

Return the whole contents of the file at location.

The whole file lands in memory, so reach for :meth:read_stream instead whenever the size is a caller's to decide — an upload, a remote object — rather than something this application wrote and knows the bound of.

read_stream

read_stream(location: str) -> AsyncIterator[bytes]

Return the file's contents in chunks, reaching the backend as they are taken.

list_contents

list_contents(location: str = '', deep: bool = False) -> DirectoryListing[StorageAttributes]

Return a lazy listing of what is under location.

Nothing is read here: the returned listing reaches the backend only once it is iterated, and any failure that surfaces then — whatever the backend raised — arrives as an :class:UnableToListContentsError carrying it as __cause__. A :class:FeatureNotSupportedError passes through unwrapped, so a caller that can list without descending can tell "this backend cannot" apart from "this listing failed".

last_modified async

last_modified(path: str) -> int

Return when the file at path last changed, in whole epoch seconds.

file_size async

file_size(path: str) -> int

Return the size in bytes of the file at path.

mime_type async

mime_type(path: str) -> str

Return what the file at path holds.

visibility async

visibility(path: str) -> Visibility

Return who may read the file at path.

write async

write(location: str, contents: bytes, config: Mapping[str, object] | None = None) -> None

Write contents at location, replacing whatever was there.

write_stream async

write_stream(location: str, contents: AsyncIterable[bytes] | Iterable[bytes] | BinaryIO, config: Mapping[str, object] | None = None) -> None

Write at location from a source read in pieces.

An async source is used as it is; a plain iterable is wrapped in one; an open binary file is rewound when it can be and read a megabyte at a time off the event loop. Text is refused, because encoding it is the caller's decision, not a guess this makes.

set_visibility async

set_visibility(path: str, visibility: Visibility | str) -> None

Change who may read the file or directory at path.

delete async

delete(location: str) -> None

Delete the file at location, treating a missing one as already gone.

delete_directory async

delete_directory(location: str) -> None

Delete the directory at location and everything under it.

create_directory async

create_directory(location: str, config: Mapping[str, object] | None = None) -> None

Create the directory at location, and the ones leading to it.

move async

move(source: str, destination: str, config: Mapping[str, object] | None = None) -> None

Move the file at source to destination, overwriting it.

Raises:

  • UnableToMoveFileError –

    Under the fail policy when the two paths are the same, and under the ignore policy when they are the same and no file is there: nothing was moved, so answering success would tell the caller a file exists that does not.

copy async

copy(source: str, destination: str, config: Mapping[str, object] | None = None) -> None

Copy the file at source to destination, overwriting it.

Raises:

  • UnableToCopyFileError –

    Under the fail policy when the two paths are the same, and under the ignore policy when they are the same and no file is there: nothing was copied, so answering success would tell the caller a file exists that does not.

checksum async

checksum(path: str, config: Mapping[str, object] | None = None) -> str

Return a digest of the file at path.

A backend that keeps its own digest answers from it; otherwise, or when it keeps none of the algorithm asked for, the file's bytes are streamed through :mod:hashlib here.

public_url async

public_url(path: str, config: Mapping[str, object] | None = None) -> str

Return a lasting address the file at path is reachable at.

An injected generator is asked first, then one built from the public_url option, and only then the backend's own — a caller that supplied nothing and a backend that knows nothing leave a storage with no address to give.

temporary_url async

temporary_url(path: str, expires_at: datetime, config: Mapping[str, object] | None = None) -> str

Return an address for the file at path that stops working at expires_at.

The deadline has to name one instant everywhere, so a moment carrying no timezone is refused before any generator is asked.

close async

close() -> None

Release whatever the adapter holds open; it stays usable afterwards.

StorageOperatorInterface

Bases: StorageReaderInterface, StorageWriterInterface, Protocol

Everything a storage does, and what code depending on one asks for.

Reading and writing arrive by inheritance — a storage that can do both is the ordinary case, and the two halves exist so that narrower code can ask for one. What is added here is the third thing a storage is used for: handing out an address a browser can fetch a file from, without the application serving the bytes itself.

Whether an address can be produced at all depends on the backend and on what the storage was configured with, so both methods may refuse.

public_url async

public_url(path: str, config: Mapping[str, object] | None = None) -> str

Return a lasting address the file at path is reachable at.

Parameters:

  • path (str) –

    The file to address.

  • config (Mapping[str, object] | None, default: None ) –

    Options for this call alone.

Returns:

  • str –

    An absolute URL, which says nothing about whether the file is

  • str –

    actually readable by whoever follows it — that is its visibility.

Raises:

temporary_url async

temporary_url(path: str, expires_at: datetime, config: Mapping[str, object] | None = None) -> str

Return an address for the file at path that stops working later.

How a private file is handed to a browser for one download: the address carries its own permission, and the permission runs out.

Parameters:

  • path (str) –

    The file to address.

  • expires_at (datetime) –

    When the address stops working. Must carry a timezone, since a moment without one names a different instant on every machine that reads it.

  • config (Mapping[str, object] | None, default: None ) –

    Options for this call alone.

Returns:

  • str –

    An absolute URL, valid until expires_at.

Raises:

checksum async

checksum(path: str, config: Mapping[str, object] | None = None) -> str

Return a digest of the file at path, for comparing it with another.

Parameters:

  • path (str) –

    The file to digest.

  • config (Mapping[str, object] | None, default: None ) –

    Options for this call alone, the algorithm among them.

Returns:

  • str –

    The digest in lowercase hexadecimal, unless the algorithm asked for

  • str –

    is one the backend keeps in a form of its own.

Raises:

StorageReaderInterface

Bases: Protocol

Reads files and tells what is there, without ever changing it.

Asking for this rather than the whole storage is how a caller says it only looks: a template renderer, a download endpoint, a report that walks a directory. The type then refuses the write that a later change might otherwise slip in.

Paths are text, relative to the storage's root, separated by / and carrying no leading slash. Every method that reaches a backend is awaited; :meth:list_contents is not, because it hands back a plan for reading rather than the reading itself.

file_exists async

file_exists(location: str) -> bool

Return whether a file is at location.

Parameters:

  • location (str) –

    Where to look.

Returns:

  • bool –

    True when a file is there — a directory at that path answers False.

Raises:

directory_exists async

directory_exists(location: str) -> bool

Return whether a directory is at location.

Parameters:

  • location (str) –

    Where to look.

Returns:

  • bool –

    True when a directory is there — a file at that path answers False.

Raises:

has async

has(location: str) -> bool

Return whether anything at all is at location, file or directory.

Parameters:

  • location (str) –

    Where to look.

Returns:

  • bool –

    True when either a file or a directory is there.

Raises:

read async

read(location: str) -> bytes

Return the whole contents of the file at location.

Parameters:

  • location (str) –

    The file to read.

Returns:

  • bytes –

    Every byte of it, in memory. Prefer :meth:read_stream for a file

  • bytes –

    whose size the caller does not control.

Raises:

read_stream

read_stream(location: str) -> AsyncIterator[bytes]

Return the file's contents in chunks, so its size need not fit in memory.

Not awaited: the call itself does nothing, and the backend is reached as the chunks are taken.

Parameters:

  • location (str) –

    The file to read.

Returns:

Raises:

  • UnableToReadFileError –

    When the file is missing or unreadable — raised while iterating, not when calling.

list_contents

list_contents(location: str, deep: bool = False) -> DirectoryListing[StorageAttributes]

Return what is under location, as a listing that has read nothing yet.

Not awaited, because nothing has happened: the returned listing can be narrowed and transformed first, and reaches the backend only once something iterates it.

Parameters:

  • location (str) –

    The directory to look in. An implementation that has a single root — a :class:~xtr_storage.Storage — may default it to that root; one routing by name — a :class:~xtr_storage.MountManager — requires it, since an empty location names no mount.

  • deep (bool, default: False ) –

    Whether to descend into the directories found, rather than naming them and stopping.

Returns:

Raises:

last_modified async

last_modified(path: str) -> int

Return when the file at path last changed.

Parameters:

  • path (str) –

    The file to ask about.

Returns:

  • int –

    Whole seconds since the epoch, in UTC.

Raises:

file_size async

file_size(path: str) -> int

Return the size of the file at path.

Parameters:

  • path (str) –

    The file to ask about.

Returns:

  • int –

    Its size in bytes.

Raises:

mime_type async

mime_type(path: str) -> str

Return what the file at path holds.

Parameters:

  • path (str) –

    The file to ask about.

Returns:

  • str –

    A media type such as "image/svg+xml", from what the backend

  • str –

    stored alongside the file or, failing that, from its extension.

Raises:

visibility async

visibility(path: str) -> Visibility

Return who may read the file at path.

Parameters:

  • path (str) –

    The file to ask about.

Returns:

  • Visibility –

    The visibility the backend reports.

Raises:

StorageWriterInterface

Bases: Protocol

Writes, deletes, moves and copies — everything that leaves a mark.

The counterpart of :class:~xtr_storage.StorageReaderInterface, kept apart so an upload handler can be handed the ability to write without being handed the ability to read every other file in the storage.

Each method takes an optional mapping of options which is laid over the ones the storage was built with, for that call alone: a visibility, a content type, whatever the backend understands. Paths are text, relative to the storage's root, separated by /.

write async

write(location: str, contents: bytes, config: Mapping[str, object] | None = None) -> None

Write contents at location, replacing whatever was there.

Missing directories on the way are created.

Parameters:

  • location (str) –

    Where the file goes.

  • contents (bytes) –

    What it holds.

  • config (Mapping[str, object] | None, default: None ) –

    Options for this call alone.

Raises:

write_stream async

write_stream(location: str, contents: AsyncIterable[bytes] | Iterable[bytes] | BinaryIO, config: Mapping[str, object] | None = None) -> None

Write at location from a source read in pieces.

For contents that should not be held in memory at once, and for the upload handles a web framework hands over.

Parameters:

  • location (str) –

    Where the file goes.

  • contents (AsyncIterable[bytes] | Iterable[bytes] | BinaryIO) –

    Chunks to write in order, from an async source, a plain iterable, or an open binary file, which is rewound first.

  • config (Mapping[str, object] | None, default: None ) –

    Options for this call alone.

Raises:

set_visibility async

set_visibility(path: str, visibility: Visibility | str) -> None

Change who may read the file or directory at path.

Parameters:

  • path (str) –

    What to change.

  • visibility (Visibility | str) –

    The visibility to set, or its name.

Raises:

delete async

delete(location: str) -> None

Delete the file at location.

Deleting a file that is not there is not an error: the caller asked for it to be gone, and it is.

Parameters:

  • location (str) –

    The file to delete.

Raises:

delete_directory async

delete_directory(location: str) -> None

Delete the directory at location and everything under it.

A directory that is not there is not an error, for the same reason.

Parameters:

  • location (str) –

    The directory to delete.

Raises:

create_directory async

create_directory(location: str, config: Mapping[str, object] | None = None) -> None

Create the directory at location, and the ones leading to it.

Creating one that already exists is not an error.

Parameters:

  • location (str) –

    The directory to create.

  • config (Mapping[str, object] | None, default: None ) –

    Options for this call alone, a directory visibility among them.

Raises:

move async

move(source: str, destination: str, config: Mapping[str, object] | None = None) -> None

Move the file at source to destination, overwriting it.

Asked to move a file onto itself, nothing is done unless the options say otherwise: see :class:~xtr_storage.IdenticalPathPolicy.

Parameters:

  • source (str) –

    The file to move.

  • destination (str) –

    Where it ends up.

  • config (Mapping[str, object] | None, default: None ) –

    Options for this call alone; what an identical source and destination does is one of them.

Raises:

  • UnableToMoveFileError –

    When the backend refused the move, or the two paths are the same and the options say that is a mistake.

copy async

copy(source: str, destination: str, config: Mapping[str, object] | None = None) -> None

Copy the file at source to destination, overwriting it.

Asked to copy a file onto itself, nothing is done unless the options say otherwise: see :class:~xtr_storage.IdenticalPathPolicy.

Parameters:

  • source (str) –

    The file to copy.

  • destination (str) –

    Where the copy goes.

  • config (Mapping[str, object] | None, default: None ) –

    Options for this call alone; what an identical source and destination does is one of them.

Raises:

  • UnableToCopyFileError –

    When the backend refused the copy, or the two paths are the same and the options say that is a mistake.

ChainedPublicUrlGenerator

ChainedPublicUrlGenerator(generators: Sequence[PublicUrlGeneratorInterface])

Bases: PublicUrlGeneratorInterface

Tries generators in order, taking the first address any of them will give.

A store may be reachable more than one way — a fast delivery host for most files, a signing generator for the few that need it — and which applies is decided per file rather than up front. Each generator declines a file it cannot address by raising, which is this chain's cue to try the next; only when every one has declined does the chain itself decline.

Fix the generators to consult, in the order they are consulted.

Parameters:

public_url async

public_url(path: str, config: Config) -> str

Return the first address any generator gives for path.

Parameters:

  • path (str) –

    The file to address, already normalized.

  • config (Config) –

    The options in force, passed on to each generator unchanged.

Returns:

  • str –

    The address of the first generator that does not decline.

Raises:

PrefixPublicUrlGenerator

PrefixPublicUrlGenerator(prefix: str)

Bases: PublicUrlGeneratorInterface

Joins every path to one address, the way a single delivery host serves a store.

The commonest case: one content delivery host publishes the whole store, so a file's address is that host followed by the file's path. The prefix owns that join, reusing the same path arithmetic the adapters use to sit under a root, so a trailing slash on the prefix or a leading slash on the path never doubles or drops one.

Fix the address every path is placed under.

Parameters:

  • prefix (str) –

    The host and any base path files are reachable from, such as "https://cdn.example.com/assets/".

public_url async

public_url(path: str, config: Config) -> str

Return path joined to the fixed prefix.

Parameters:

  • path (str) –

    The file to address, already normalized.

  • config (Config) –

    The options in force; the prefix is fixed, so they change nothing here.

Returns:

  • str –

    The prefix followed by the path, with exactly one separator between.

PublicUrlGeneratorInterface

Bases: Protocol

Says where a stored file is reachable from, so the application need not serve it.

Two kinds of thing satisfy this. An adapter whose backend owns an address for every object it holds, which a storage discovers at runtime. And a standalone generator put in front of a storage — a content delivery host the files are published under, which the backend knows nothing about.

Producing an address is not a permission check: the address may well lead to a file nobody but its owner may read.

public_url async

public_url(path: str, config: Config) -> str

Return a lasting address for the file at path.

Parameters:

  • path (str) –

    The file to address, already normalized.

  • config (Config) –

    The options in force.

Returns:

  • str –

    An absolute URL.

Raises:

  • UnableToGeneratePublicUrlError –

    When no address can be built for this file — which a chain of generators takes as its cue to try the next one.

ShardedPrefixPublicUrlGenerator

ShardedPrefixPublicUrlGenerator(prefixes: Sequence[str])

Bases: PublicUrlGeneratorInterface

Picks one of several addresses per file, so no single host serves everything.

Several delivery hosts publish the same store; spreading files over them keeps any one from carrying the whole load. The choice is a hash of the path rather than a counter, so the same file always resolves to the same host — an address stays quotable and a browser's cache stays warm — while different files land evenly across the hosts.

Fix the addresses files are spread over.

Parameters:

  • prefixes (Sequence[str]) –

    The hosts, each a base a file may be reachable from. Order matters, since a path's host is chosen by its position.

Raises:

  • InvalidArgumentError –

    When no prefix is given, since there would be nothing to place a file under.

public_url async

public_url(path: str, config: Config) -> str

Return path joined to the host its hash selects.

Parameters:

  • path (str) –

    The file to address, already normalized.

  • config (Config) –

    The options in force; the hosts are fixed, so they change nothing here.

Returns:

  • str –

    The chosen host followed by the path, with exactly one separator

  • str –

    between.

TemporaryUrlGeneratorInterface

Bases: Protocol

Hands out an address that carries its own permission, and loses it in time.

How a private file reaches a browser without the application streaming the bytes: the address is signed, whoever holds it may fetch that one file, and after the moment it was signed for it leads nowhere.

Usually satisfied by an adapter whose backend can sign — a storage looks for that at runtime — but a generator of one's own may be put in front instead.

temporary_url async

temporary_url(path: str, expires_at: datetime, config: Config) -> str

Return an address for the file at path, good until expires_at.

Parameters:

  • path (str) –

    The file to address, already normalized.

  • expires_at (datetime) –

    When the address stops working. A storage has already refused a moment carrying no timezone, so this names one instant everywhere.

  • config (Config) –

    The options in force.

Returns:

  • str –

    An absolute URL.

Raises:

Visibility

Bases: StrEnum

Whether a stored file is readable by anyone, or by its owner alone.

Two values rather than a backend's permission model: a mode bit on a local file and an access list on an object store have nothing in common, and code that stores files only ever cares which of the two it asked for. Each adapter translates them, and one that cannot say raises rather than guess.

The members are their strings, so an option mapping, a configuration file or an environment variable carries "public" and is understood without a conversion step — while code that reads them still gets a closed set.

parse classmethod

parse(value: Visibility | str) -> Visibility

Return the member value stands for.

The single door for anything arriving from outside — a config option, a bundle definition, a caller's keyword — so a typo is refused once, here, instead of reaching an adapter as an unknown permission.

Parameters:

  • value (Visibility | str) –

    A member, or the string one carries.

Returns:

Raises: