xtr_storage ¶
Files on local disk, in memory or in object stores, behind one async interface.
A Storage wraps one adapter and turns paths into reads, writes, listings,
copies and moves. The adapter decides where the bytes live — a directory on this
machine, a dictionary in this process, a bucket on an object store — so code that
stores files is written once and moved between them by its constructor.
Every call that touches a backend is awaited.
StorageAttributes ¶
One entry of a listing.
A closed union rather than a base class with optional fields: the two sides
answer different questions — only a file has a size — and code that must handle
both is made to say which it is holding, through is_file or a match.
FsspecAdapter ¶
Bases: StorageAdapterInterface, ABC
Everything an fsspec backend does, bar the one line that builds it.
A subclass implements :meth:_create_filesystem and inherits a working
adapter. Where its backend behaves differently — an object store that has
no real directories, one that keeps a media type of its own — it overrides
a hook rather than a whole method: :meth:_write_options to forward write
settings, :meth:_is_hidden_entry to hide the markers it lists,
:meth:_session_closer to shut a client down. Visibility is off by default,
:meth:visibility and :meth:set_visibility raising until a subclass that
has it overrides them.
Remember the root and the detector, and open nothing.
Parameters:
-
root(str, default:'') –What every path is stored under — a directory, a key prefix. Empty means the backend's own root, so the adapter is a pass-through.
-
mime_type_detector(MimeTypeDetectorInterface | None, default:None) –How a media type is guessed from a name when the backend keeps none; a name-only detector by default.
read_stream
async
¶
Yield the file at path a mebibyte at a time, reaching back for each.
The size is read once, up front, and the ranges are cut from it: a
stream is a sequence of independent ranged reads, not a held handle.
A file rewritten while it is being streamed therefore hands back a
mixture — ranges from before the rewrite and ranges from after, or a
short last range when it shrank — and one deleted mid-stream fails on
the next range with :class:UnableToReadFileError. No backend this
library speaks to offers a consistent snapshot across separate reads,
so a caller who needs one writes a new object and swaps the name rather
than rewriting a file something is reading.
write
async
¶
Write contents at path in one call.
write_stream
async
¶
Write contents at path, in memory while small, through a temp above.
A stream that fits in :data:_SPOOL_MAX_SIZE is gathered in memory and
written in one :meth:pipe_file; one that grows past it spills, from that
point on, into a named temporary file the backend reads with
:meth:put_file, so an upload of any size lands without ever holding the
whole file in memory.
file_exists
async
¶
Return whether a file — not a directory — is at path.
directory_exists
async
¶
Return whether a directory — not a file — is at path.
delete
async
¶
Delete the file at path, treating a missing one as already gone.
delete_directory
async
¶
Delete the directory at path and its tree, missing being no failure.
create_directory
async
¶
Create the directory at path and its parents, once being enough.
list_contents ¶
Return what is under path, one entry at a time.
move
async
¶
Move source onto destination, then settle its visibility.
copy
async
¶
Copy source onto destination, then settle its visibility.
last_modified
async
¶
Return when the file at path last changed, in whole epoch seconds.
mime_type
async
¶
Return what the file at path holds, from the backend or the name.
visibility
async
¶
Raise: a plain filesystem adapter has no notion of visibility.
set_visibility
async
¶
Raise: a plain filesystem adapter has no notion of visibility.
close
async
¶
Close the backend's session, if one was opened; safe to call again.
The bridge resets rather than being dropped, so the filesystem it wraps — and, for the in-memory backend, the files that live in it — survive the close while the networked session is released: a storage closed between requests is usable again, not stuck closed and not emptied.
GcsAdapter ¶
GcsAdapter(bucket: str, prefix: str = '', *, project: str | None = None, token: str | None = None, endpoint_url: str | None = None, write_options: Mapping[str, object] | None = None, mime_type_detector: MimeTypeDetectorInterface | None = None)
Bases: FsspecAdapter
Files in a Google Cloud Storage bucket, behind the shared adapter contract.
The bucket, and an optional key prefix within it, are the root every path is placed under. Directories are the object-store fiction the shared helpers keep up; checksums, public URLs and signed URLs are real; visibility is refused, because the store offers no single answer to map onto.
Satisfies the checksum, public-url and temporary-url capability protocols structurally — by having their methods, not by inheriting them — so a storage discovers each at runtime.
Remember where and how to reach the bucket, and open nothing yet.
Parameters:
-
bucket(str) –The bucket every path lives in.
-
prefix(str, default:'') –A key prefix within the bucket that the caller never sees, so several storages may share one bucket without colliding.
-
project(str | None, default:None) –The Google Cloud project the bucket belongs to, for the operations that need one; the backend's own default otherwise.
-
token(str | None, default:None) –How the backend authenticates — a path to a credentials file, a token, or
"anon"for an unauthenticated emulator. -
endpoint_url(str | None, default:None) –Where to reach the store, for pointing at an emulator; the real service otherwise.
-
write_options(Mapping[str, object] | None, default:None) –Settings every write forwards unless a call overrides them — a content type, a cache-control header, custom metadata.
-
mime_type_detector(MimeTypeDetectorInterface | None, default:None) –How a media type is guessed from a name when the object carries none; a name-only detector by default.
write
async
¶
Write contents at path, refusing a visibility this store cannot keep.
write_stream
async
¶
Write path from chunks, refusing a visibility this store cannot keep.
create_directory
async
¶
Write the zero-byte marker that stands in for the directory at path.
The store has no directories, so an empty one is a key ending in a slash; the bucket root always exists, so a request for it does nothing. A visibility named for the directory is refused, as it is on a write.
directory_exists
async
¶
Return whether a marker or any key stands under path.
checksum
async
¶
Return the digest the store keeps beside the object at path.
The store records two: an MD5, handed back as the hex of the base64 the store keeps it in, and a CRC32C, handed back in the base64 the store keeps it in. Any other algorithm — and an object the store kept no digest of the asked-for kind for — is declined, which a storage takes as its cue to read the bytes and compute one itself.
Raises:
-
ChecksumAlgorithmNotSupportedError–When the algorithm is neither
"md5"nor"crc32c", or the object carries no such digest. -
UnableToProvideChecksumError–When the store could not be asked.
public_url
async
¶
Return the lasting address of the object at path.
The address is the emulator endpoint, or the public service, followed by the bucket and the object's key, the key percent-escaped but for the slashes that structure it.
temporary_url
async
¶
Return a signed address for path, good until expires_at.
The deadline is turned into a whole number of seconds from now, never
below one, and the store signs for that long. Anything the backend
raises while signing becomes an
:class:~xtr_storage.exception.UnableToGenerateTemporaryUrlError, the
original kept as its cause.
Raises:
-
UnableToGenerateTemporaryUrlError–When the store could not sign.
GenericFsspecAdapter ¶
GenericFsspecAdapter(filesystem: AbstractFileSystem, root: str = '', *, closer: Callable[[AbstractFileSystem], Awaitable[None]] | None = None)
Bases: FsspecAdapter
A storage adapter over a filesystem somebody else built.
The instance is used exactly as given and never reconfigured: it is the
caller's, built the way they wanted, and — unless a closer is handed over
with it — it stays open when this adapter is closed so the rest of their code
may go on using it. A caller that wants the filesystem shut with the adapter
(the bundle, which builds the filesystem itself and owns its lifecycle) passes
a closer, which the bridge runs once on close when a session was opened.
Reading, writing, listing, moving and metadata all work — they are the same calls whatever the backend. Visibility is not offered: see the module docstring for why.
Keep the filesystem and the root, and touch neither.
Parameters:
-
filesystem(AbstractFileSystem) –The filesystem to store through, already built.
-
root(str, default:'') –What every path is stored under — a directory, a key prefix. Empty stores at the filesystem's own root.
-
closer(Callable[[AbstractFileSystem], Awaitable[None]] | None, default:None) –What closes the filesystem's session on
close, for an owner that built it;Noneleaves it open, since the caller who built it owns it.
InMemoryAdapter ¶
InMemoryAdapter(default_visibility: Visibility = Visibility.PUBLIC, *, mime_type_detector: MimeTypeDetectorInterface | None = None)
Bases: FsspecAdapter
Files in a dictionary, each with a visibility remembered beside it.
Every file this adapter is given is readable from it and from nowhere else: two adapters in one process share no store, so a test that writes has nothing to undo and cannot colour the next one.
Visibility is real here rather than refused. There is no permission to set on a dictionary entry, so the answer is simply remembered: a write records what its options asked for, or the adapter's default where they asked for nothing; a copy and a move carry it across; a delete forgets it, as does deleting a directory for everything beneath it.
Take the visibility a write falls back on, and hold no store yet.
Parameters:
-
default_visibility(Visibility, default:PUBLIC) –What a file is worth when nothing said otherwise — readable by anyone, matching the openness of a store that only this process can reach anyway.
-
mime_type_detector(MimeTypeDetectorInterface | None, default:None) –How a media type is guessed from a name; a name-only detector by default.
write
async
¶
Write contents at path and remember who may read it.
write_stream
async
¶
Write path from chunks and remember who may read it.
move
async
¶
Move the file, and leave no visibility behind at the old path.
The base has already given the destination the visibility the rule calls for — the one the call named, or the source's — by the time this drops the source's, so what travelled is kept and what stayed is gone.
delete_directory
async
¶
Delete the directory and forget the visibility of everything under it.
visibility
async
¶
Return who may read the file at path.
Raises:
-
UnableToRetrieveMetadataError–When no file is stored there.
set_visibility
async
¶
Record who may read the file at path.
Raises:
-
UnableToSetVisibilityError–When no file is stored there. A remembered answer about a file that does not exist would outlive nothing and mislead the next write.
LocalAdapter ¶
LocalAdapter(directory: str | PathLike[str], *, visibility: VisibilityConverterInterface | None = None, link_handling: LinkHandling = LinkHandling.DISALLOW, mime_type_detector: MimeTypeDetectorInterface | None = None)
Bases: FsspecAdapter
A directory on this machine, everything under it and nothing above it.
The three things a real filesystem adds to the shared base are all here. Visibility becomes permission bits, through a converter the deployment may replace. Directories a write needs on the way are created with the mode that converter names rather than whatever the process umask leaves, since a storage told to keep its directories private must not depend on the shell that started the application. And symbolic links, which could point anywhere including outside the directory this storage is rooted in, are either left out of listings or refused outright.
Staying inside the root is enforced on every operation, not only on
listings, and under either way of handling a link: each caller path is
resolved through the filesystem before the backend is reached, and one that
a link — its own last segment, or a directory on the way — carries outside
the root is refused with
:class:~xtr_storage.exception.SymbolicLinkEncounteredError, because such a
path names a file the storage does not answer for. Skipping links decides
what a listing shows; it does not make the tree above the root reachable
through one.
Nothing is touched until the first operation: the root directory is created by the first write, so an application may declare a storage for a disk that is not mounted yet and pay nothing until it uses it.
Remember the directory and the rules, and touch the disk for nothing.
Parameters:
-
directory(str | PathLike[str]) –The directory every path is relative to. Made absolute here — reading the working directory, not the disk — because the paths a listing hands back are absolute and the root has to be strippable from them again. One that is itself a symbolic link is honoured — the link is the operator's own choice — and only links met below it are refused.
-
visibility(VisibilityConverterInterface | None, default:None) –How a visibility becomes permission bits; the portable converter and its defaults when left out.
-
link_handling(LinkHandling, default:DISALLOW) –What an operation does with a symbolic link it meets. Refusing by default: a link is rarely meant to be part of a storage, and silence about one is worse than a failure. Either way, no link is followed out of the root.
-
mime_type_detector(MimeTypeDetectorInterface | None, default:None) –How a media type is guessed from a name, since a filesystem stores none.
write
async
¶
Write the bytes at path, directories and permissions as asked.
The bytes go to a hidden temporary file beside the target, which then replaces it in one step: a reader sees the old file or the new one, never part of either, and a failed write leaves the old file.
write_stream
async
¶
Stream the bytes to path, replacing it whole once the stream ends.
create_directory
async
¶
Create the directory at path and its parents, with their own mode.
The mode comes from the call's directory visibility, or from the converter's default for directories when the call named none. Only the directories this call creates are given it: one that was already there keeps what it had, so creating the same directory twice is not a way to change its permissions.
visibility
async
¶
Return who may read what is at path, from its permission bits.
set_visibility
async
¶
Change the permission bits at path to the ones visibility means.
PathPrefixedAdapter ¶
Bases: StorageAdapterInterface, ChecksumProviderInterface, PublicUrlGeneratorInterface, TemporaryUrlGeneratorInterface
Another adapter, shown as if its files began one path further down.
Two applications sharing a bucket, one tenant per directory, a test suite that wants its own corner of a real backend: each is the same need, to hand out a storage whose root is somewhere inside another one. The caller works in plain paths and never learns the prefix, which is what makes the prefix changeable — it is configuration, not something spelled out at every call site and impossible to move later.
The translation goes both ways, and both halves matter. Paths go down with
the prefix on. Listings come back with it off, because an entry a caller
cannot pass back to the same storage is worse than no entry. And a failure
the wrapped adapter raises is told again about the path the caller used,
keeping the original as its __cause__ so the prefixed path is still
there for whoever is debugging the backend.
Only the failures of an operation are retold that way. Anything else — a capability the backend does not have, an argument that makes no sense — passes through as it is: it is not about a path.
Wrap inner so everything it is asked for sits under prefix.
Parameters:
-
inner(StorageAdapterInterface) –The adapter that holds the files, under whatever root of its own it already has.
-
prefix(str) –The path this adapter's root is at, inside
inner.
Raises:
-
InvalidArgumentError–When
prefixnames no path. A prefix that prefixes nothing would make this a pass-through, and a storage quietly writing to a shared root instead of its own corner is the kind of mistake that is found by finding the files. -
PathTraversalDetectedError–When
prefixcarries a..segment. -
CorruptedPathDetectedError–When
prefixcarries a control character.
file_exists
async
¶
Return whether a file is at path under the prefix.
directory_exists
async
¶
Return whether a directory is at path under the prefix.
write
async
¶
Write contents at path under the prefix.
write_stream
async
¶
Write at path under the prefix from contents.
read_stream ¶
Return the file at path under the prefix, in chunks.
delete_directory
async
¶
Delete the directory at path under the prefix, and its tree.
create_directory
async
¶
Create the directory at path under the prefix.
set_visibility
async
¶
Change who may read what is at path under the prefix.
visibility
async
¶
Return who may read the file at path, described at the caller's path.
mime_type
async
¶
Return what the file at path holds, described at the caller's path.
last_modified
async
¶
Return when the file at path last changed, at the caller's path.
file_size
async
¶
Return how many bytes the file at path holds, at the caller's path.
list_contents ¶
Return what is under path, every entry back in the caller's terms.
move
async
¶
Move source to destination, both under the prefix.
copy
async
¶
Copy source to destination, both under the prefix.
checksum
async
¶
Return the digest the wrapped adapter keeps for path, if it keeps one.
Raises:
-
ChecksumAlgorithmNotSupportedError–When the wrapped adapter keeps no digest at all — the same answer it would give for an algorithm it does not hold, and what makes a storage read the file and compute the digest itself.
public_url
async
¶
Return the lasting address the wrapped adapter gives the prefixed path.
Raises:
-
UnableToGeneratePublicUrlError–When the wrapped adapter has no address to give.
temporary_url
async
¶
Return the expiring address the wrapped adapter signs for the prefixed path.
Raises:
-
UnableToGenerateTemporaryUrlError–When the wrapped adapter signs nothing.
PortableVisibilityConverter ¶
PortableVisibilityConverter(*, file_public: int = 420, file_private: int = 384, directory_public: int = 493, directory_private: int = 448, default_for_directories: Visibility = Visibility.PRIVATE)
Four modes and a default, chosen to work the same wherever they land.
The defaults are the ones an ordinary deployment wants: a public file is readable by everyone and writable by its owner, a private one is the owner's alone, and each directory adds the execute bit it needs to be entered. Directories default to private, because a write creates them without being asked and the safe answer to an unasked question is the narrower one.
Every value is adjustable, since what "public" has to mean depends on who serves the files: a web server running as another user needs the group or world bits that a site handing nothing out would rather not grant.
Reading a mode back is the inexact direction, and the rule is deliberately blunt: a mode reads as private when it is exactly the mode this converter writes for private, and as public otherwise. That keeps the two directions in agreement — set a visibility, read it back, get the same answer — while making no claim about a file this storage never wrote, which carries whatever umask the process that created it had. The only thing this converter can honestly say about such a file is that its bits are not the ones it locks a file with.
Record the four modes and which one an unasked-for directory gets.
The modes are kept as two lookup tables rather than four fields so that
translating a visibility is a total mapping: a visibility this converter
was not built for fails loudly on the lookup instead of silently taking
one of the two branches an if would offer.
Parameters:
-
file_public(int, default:420) –The mode a file readable by everyone carries.
-
file_private(int, default:384) –The mode a file readable by its owner alone carries, and the only mode read back as private.
-
directory_public(int, default:493) –The mode a directory everyone may enter carries.
-
directory_private(int, default:448) –The mode a directory its owner alone may enter carries, and the only directory mode read back as private.
-
default_for_directories(Visibility, default:PRIVATE) –The visibility a directory gets when the call that created it named none.
for_file ¶
Return the mode a file of this visibility carries.
for_directory ¶
Return the mode a directory of this visibility carries.
inverse_for_file ¶
Return private when mode is exactly the private file mode, else public.
inverse_for_directory ¶
Return private when mode is exactly the private directory mode, else public.
default_for_directories ¶
Return the mode a directory nobody named a visibility for carries.
ReadOnlyAdapter ¶
Bases: StorageAdapterInterface, ChecksumProviderInterface, PublicUrlGeneratorInterface, TemporaryUrlGeneratorInterface
Another adapter, published without any of its ways of changing anything.
What an application hands out when a backend is someone else's to write: an archive it may serve but not amend, a bucket another service owns, a directory of assets a build step fills. Every read goes through untouched and every attempt to change something fails at once, so a mistake surfaces here rather than as a permission error from the backend hours later — or, worse, as a write that succeeded where it should not have.
Each refusal is the failure of the operation that was asked for, with the reason naming why, so code that already handles "the write did not happen" needs nothing new to handle this.
The three capabilities a backend may have — a digest it keeps, a lasting address, a signed one — are read-only themselves, so they are passed on when the wrapped adapter has them. This adapter answers to all three regardless, because whether the one underneath does is not known until it is there; when it does not, each says so in the terms of that capability: a checksum reports that no such digest is kept, which is a storage's cue to read the file and compute one, and the two url capabilities refuse.
Wrap inner and keep only the half of it that reads.
Parameters:
-
inner(StorageAdapterInterface) –The adapter that holds the files. It is never asked to change anything, so it may well be one whose credentials could.
file_exists
async
¶
Return whether a file is at path, as the wrapped adapter sees it.
directory_exists
async
¶
Return whether a directory is at path, as the wrapped adapter sees it.
read_stream ¶
Return the file at path in chunks, straight from the wrapped adapter.
list_contents ¶
Return what is under path, straight from the wrapped adapter.
last_modified
async
¶
Return when the file at path last changed.
file_size
async
¶
Return how many bytes the file at path holds.
write
async
¶
Refuse: nothing is written through a read-only adapter.
write_stream
async
¶
Refuse before a single chunk is read, leaving the source untouched.
delete
async
¶
Refuse, even for a path that holds nothing: a delete is a change.
delete_directory
async
¶
Refuse: a read-only adapter deletes no tree.
create_directory
async
¶
Refuse: a read-only adapter makes no directory.
set_visibility
async
¶
Refuse: who may read a file is the owning application's to decide.
move
async
¶
Refuse: a move would both write and delete.
copy
async
¶
Refuse: the destination is under the same read-only adapter.
checksum
async
¶
Return the digest the wrapped adapter keeps for path, if it keeps one.
Raises:
-
ChecksumAlgorithmNotSupportedError–When the wrapped adapter keeps no digest at all — the same answer it would give for an algorithm it does not hold, and what makes a storage read the file and compute the digest itself.
public_url
async
¶
Return the lasting address the wrapped adapter gives path.
Raises:
-
UnableToGeneratePublicUrlError–When the wrapped adapter has no address to give.
temporary_url
async
¶
Return the expiring address the wrapped adapter signs for path.
Raises:
-
UnableToGenerateTemporaryUrlError–When the wrapped adapter signs nothing.
S3Adapter ¶
S3Adapter(bucket: str, prefix: str = '', *, region: str | None = None, endpoint_url: str | None = None, key: str | None = None, secret: str | None = None, token: str | None = None, anon: bool = False, client_kwargs: Mapping[str, object] | None = None, config_kwargs: Mapping[str, object] | None = None, write_options: Mapping[str, object] | None = None, mime_type_detector: MimeTypeDetectorInterface | None = None)
Bases: FsspecAdapter
An S3 bucket, and an optional key prefix within it, as one storage.
What an object store adds to the shared base is all here. A directory is a
zero-byte marker object, since the store keeps none of its own, and those
markers are hidden from listings and answered for by an existence check that
knows to look for them. Visibility is a canned access list — public-read or
private — set on write and copy and read back from the object's grants.
A checksum is the entity tag the store already holds, handed out only when
"etag" is asked for so the storage falls back to hashing the bytes for
anything else. And a file has two addresses: a lasting public one composed
from the bucket and region, and a temporary signed one the backend produces.
Nothing is opened by construction: the backend package is imported and the client built on the first operation, so an application may declare a bucket it never touches and pay nothing for it. Credentials handed in are kept for that first build and never reach a message or a representation.
Remember the bucket, the prefix and the credentials, and open nothing.
Parameters:
-
bucket(str) –The bucket every object is stored in.
-
prefix(str, default:'') –A key prefix under which this storage's objects live, so one bucket may hold several storages. Empty for the whole bucket.
-
region(str | None, default:None) –The bucket's region, used to compose a public url; defaults to
us-east-1there when left out. -
endpoint_url(str | None, default:None) –An alternate endpoint — a compatible store, a local test server — used for both requests and composed public urls.
-
key(str | None, default:None) –The access key id, or
Noneto let the backend find one. -
secret(str | None, default:None) –The secret access key, paired with
key. -
token(str | None, default:None) –A session token, for temporary credentials.
-
anon(bool, default:False) –Whether to reach the store without credentials.
-
client_kwargs(Mapping[str, object] | None, default:None) –Extra keyword settings for the underlying client, merged after the region.
-
config_kwargs(Mapping[str, object] | None, default:None) –Extra client configuration for the underlying client.
-
write_options(Mapping[str, object] | None, default:None) –Default write settings; only the keys this backend understands are forwarded, the rest dropped.
-
mime_type_detector(MimeTypeDetectorInterface | None, default:None) –How a media type is guessed from a name when a write names none; a name-only detector by default.
write
async
¶
Write contents at path, its media type guessed from the name.
write_stream
async
¶
Stream contents to path, its media type guessed from the name.
create_directory
async
¶
Write the zero-byte marker object that stands in for the directory.
directory_exists
async
¶
Return whether a marker or any object lives under the directory path.
visibility
async
¶
Return who may read path, read from the object's access list.
Raises:
-
UnableToRetrieveMetadataError–When the object is missing or its access list could not be read.
set_visibility
async
¶
Set path's canned access list to the one visibility means.
Raises:
-
UnableToSetVisibilityError–When the object is missing or the change was refused.
checksum
async
¶
Return the entity tag the store holds for path, quotes stripped.
Parameters:
-
path(str) –The file to digest, already normalized.
-
config(Config) –The options in force; only
"etag"is answered from here.
Returns:
-
str–The entity tag, without the quotes the store wraps it in.
Raises:
-
ChecksumAlgorithmNotSupportedError–When any algorithm but
"etag"is asked for — the storage's cue to hash the bytes instead. -
UnableToProvideChecksumError–When the store kept no entity tag.
-
UnableToRetrieveMetadataError–When the object is missing.
public_url
async
¶
temporary_url
async
¶
Return a signed address for path, good until expires_at.
Parameters:
-
path(str) –The file to address, already normalized.
-
expires_at(datetime) –When the address stops working; a timezone-aware moment the storage has already vouched for.
-
config(Config) –The options in force; unread here.
Returns:
-
str–An absolute, signed url.
Raises:
-
UnableToGenerateTemporaryUrlError–When the address could not be signed.
StorageAdapterInterface ¶
Bases: Protocol
One backend's side of the bargain: paths in, bytes and facts out.
This is deliberately narrower than what a storage offers its callers. A
storage is the front door — it normalizes paths, layers options, refuses
what it can refuse and falls back where it can. An adapter is handed the
result of all that: a path already cleaned, and a :class:~xtr_storage.Config
already merged. It never has to guess what a caller meant.
That is what makes a new backend a small piece of work, and what lets one conformance suite judge every backend by the same behaviour.
Anything a backend genuinely cannot do raises FeatureNotSupportedError
rather than pretending: a silent no-op would leave a caller believing a
file is private when anyone can read it.
The three metadata methods return :class:~xtr_storage.FileAttributes
rather than the bare value, so an adapter that learned four facts from one
round trip can hand all four back and let the storage keep what it needs.
file_exists
async
¶
Return whether a file — not a directory — is at path.
Raises:
-
UnableToCheckFileExistenceError–When the backend could not say.
directory_exists
async
¶
Return whether a directory — not a file — is at path.
Raises:
-
UnableToCheckDirectoryExistenceError–When the backend could not say.
write
async
¶
Write contents at path, making the directories on the way.
Parameters:
-
path(str) –Where the file goes.
-
contents(bytes) –What it holds.
-
config(Config) –The options in force, a visibility and a content type among the ones an adapter may read.
Raises:
-
UnableToWriteFileError–When the backend refused the write.
write_stream
async
¶
Write at path from chunks, without holding the whole file in memory.
Only an async source arrives here: a storage has already turned an open file or a plain iterable into one.
Parameters:
-
path(str) –Where the file goes.
-
contents(AsyncIterable[bytes]) –The chunks, in order.
-
config(Config) –The options in force.
Raises:
-
UnableToWriteFileError–When the backend refused the write.
read
async
¶
Return the whole contents of the file at path.
Raises:
-
UnableToReadFileError–When the file is missing or unreadable.
read_stream ¶
Return the file's contents in chunks, reaching the backend as they are taken.
Not awaited, so an adapter may implement it as an async generator.
Raises:
-
UnableToReadFileError–While iterating, when the file is missing or unreadable.
delete
async
¶
Delete the file at path, treating a missing file as already deleted.
Raises:
-
UnableToDeleteFileError–When the backend refused the deletion.
delete_directory
async
¶
Delete the directory at path and everything under it.
A directory that is not there is already deleted.
Raises:
-
UnableToDeleteDirectoryError–When the backend refused the deletion.
create_directory
async
¶
Create the directory at path, and the ones leading to it.
Parameters:
-
path(str) –The directory to create.
-
config(Config) –The options in force, a directory visibility among them.
Raises:
-
UnableToCreateDirectoryError–When the backend refused.
set_visibility
async
¶
Change who may read what is at path.
Raises:
-
UnableToSetVisibilityError–When the backend refused the change.
-
FeatureNotSupportedError–When the backend has no notion of visibility at all.
visibility
async
¶
Return who may read the file at path, as an attributes record.
Raises:
-
UnableToRetrieveMetadataError–When the backend does not know.
-
FeatureNotSupportedError–When the backend has no notion of visibility at all.
mime_type
async
¶
Return what the file at path holds, as an attributes record.
Raises:
-
UnableToRetrieveMetadataError–When neither the backend nor the file's name says.
last_modified
async
¶
Return when the file at path last changed, as an attributes record.
Raises:
-
UnableToRetrieveMetadataError–When the backend does not know.
file_size
async
¶
Return the size of the file at path, as an attributes record.
Raises:
-
UnableToRetrieveMetadataError–When the backend does not know, or
pathnames a directory.
list_contents ¶
Return what is under path, one entry at a time.
Not awaited, so an adapter may implement it as an async generator and a storage may wrap it in a lazy listing. A directory that is not there yields nothing rather than raising: a backend where directories are only implied cannot tell the two apart.
Parameters:
move
async
¶
Move the file at source to destination, overwriting it.
Raises:
-
UnableToMoveFileError–When the backend refused the move.
copy
async
¶
Copy the file at source to destination, overwriting it.
Raises:
-
UnableToCopyFileError–When the backend refused the copy.
close
async
¶
Release whatever the adapter holds open, and stay usable afterwards.
A backend reached over the network keeps a session; one on local disk keeps nothing and does nothing here. Closing twice is not an error.
VisibilityConverterInterface ¶
Bases: Protocol
Turns a visibility into permission bits, and permission bits back into one.
A filesystem has no notion of public and private — it has a number, and the number a deployment calls public is its own decision: one site serves files through a web server running as another user and needs the group bit, another hands nothing out and wants the narrowest mode that still works. A converter is where that decision lives, so the adapter above it only ever speaks of visibility.
Files and directories are asked separately because a directory needs its execute bit to be entered at all, so the mode that makes a file readable makes a directory useless.
The inverse direction is a guess and is meant to be: a filesystem holds no record of what a mode was intended to mean, and a file written by another process carries whatever that process's umask left. An implementation says which modes it recognises as private and reads the rest as public, so that setting a visibility and reading it back agrees with itself.
for_file ¶
Return the permission bits a file of this visibility should carry.
Parameters:
-
visibility(Visibility) –Who should be able to read the file.
Returns:
-
int–The mode, as the number a permission change takes.
for_directory ¶
Return the permission bits a directory of this visibility should carry.
Parameters:
-
visibility(Visibility) –Who should be able to list and enter the directory.
Returns:
-
int–The mode, as the number a permission change takes.
inverse_for_file ¶
Return the visibility a file's permission bits stand for.
Parameters:
-
mode(int) –The file's permission bits, with any type bits already removed.
Returns:
-
Visibility–The visibility those bits are read as.
inverse_for_directory ¶
Return the visibility a directory's permission bits stand for.
Parameters:
-
mode(int) –The directory's permission bits, with any type bits already removed.
Returns:
-
Visibility–The visibility those bits are read as.
default_for_directories ¶
Return the permission bits a directory nobody asked about should carry.
A write creates the directories on the way to its file, and the call that asked for it usually says nothing about them. This is the answer for that case, so those directories get a mode the deployment chose rather than whatever the process umask happens to be.
ChecksumProviderInterface ¶
Bases: Protocol
An adapter that can answer for a digest without sending the file back.
Object stores keep a digest beside every object, and asking for it is one small request; computing the same digest means downloading the whole file. So an adapter that has one says so by satisfying this, and a storage checks at runtime before falling back to reading the bytes itself.
Optional by design: an adapter that does not implement this is not lacking anything — the fallback gives the same answer, more slowly.
checksum
async
¶
Return the digest the backend holds for the file at path.
Parameters:
-
path(str) –The file to digest, already normalized.
-
config(Config) –The options in force, naming the algorithm among them.
Returns:
-
str–The digest, in the form the backend keeps it.
Raises:
-
ChecksumAlgorithmNotSupportedError–When the backend keeps no digest of that algorithm — a storage takes this as its cue to compute one from the file's bytes instead.
-
UnableToProvideChecksumError–When the backend could not be asked.
Config ¶
The options one operation runs with, layered rather than changed.
Options reach an adapter from two places: the ones a storage was built with, and the ones a single call passes. Both are open mappings — an adapter reads the keys below, and forwards the rest to its backend — so the layering, not a fixed field list, is what this class is for.
Nothing is ever changed in place. extend and with_defaults answer
the only two questions there are — who wins, the newcomer or the incumbent
— and each returns a new configuration, so a storage can hand its own
options to one call without that call's options leaking into the next.
defaults = Config({Config.VISIBILITY: "private"})
for_this_call = defaults.extend({Config.VISIBILITY: "public"}) # public wins
The typed accessors are the reading half: a value arriving from a configuration file is text, and every caller would otherwise parse it again, differently.
Take a copy of options, so a caller's dictionary stays the caller's.
Parameters:
-
options(Mapping[str, object] | None, default:None) –What this configuration holds; nothing when left out.
VISIBILITY
class-attribute
instance-attribute
¶
Who may read what is written: a :class:~xtr_storage.Visibility or its name.
DIRECTORY_VISIBILITY
class-attribute
instance-attribute
¶
The same, for directories a write or a create has to make on the way.
RETAIN_VISIBILITY
class-attribute
instance-attribute
¶
Whether a copy or a move keeps what the source had. True unless said otherwise.
MOVE_IDENTICAL_PATH
class-attribute
instance-attribute
¶
What a move from a path to itself does: an :class:~xtr_storage.IdenticalPathPolicy.
Nothing, unless said otherwise.
COPY_IDENTICAL_PATH
class-attribute
instance-attribute
¶
The same, for a copy.
CHECKSUM_ALGORITHM
class-attribute
instance-attribute
¶
Which digest a checksum is taken with. "md5" unless said otherwise.
PUBLIC_URL
class-attribute
instance-attribute
¶
Where files are reachable from: one prefix, or several to spread them over.
ALLOW_RELATIVE_PATH_TRAVERSAL
class-attribute
instance-attribute
¶
Whether .. in a path may climb. True unless said otherwise.
get ¶
extend ¶
with_defaults ¶
with_setting ¶
without_settings ¶
Return these options with keys dropped, ignoring any that is not set.
Dropping rather than overwriting matters where absence means something
of its own: a storage's default visibility must be gone, not None,
for a copy that is asked to keep the source's.
Parameters:
-
*keys(str, default:()) –The options to leave out.
Returns:
-
Config–A new configuration; this one is unchanged.
to_dict ¶
Return the options as a plain dictionary, copied so callers may keep it.
visibility_option ¶
Return the visibility key holds, or None when it holds none.
None is an answer, not a failure: it says the caller made no
demand, and the adapter is free to use the backend's default.
Parameters:
-
key(str) –The option's name, usually :attr:
VISIBILITY.
Returns:
-
Visibility | None–The visibility asked for, or
None.
Raises:
-
InvalidArgumentError–When the option holds something that is not text.
-
InvalidVisibilityError–When the text names no visibility.
bool_option ¶
Return the flag key holds, or default when it holds none.
Parameters:
Returns:
-
bool–The flag.
Raises:
-
InvalidArgumentError–When the option holds anything but true or false.
identical_path_policy ¶
Return the policy key holds, ignoring the operation when it holds none.
Parameters:
-
key(str) –:attr:
COPY_IDENTICAL_PATHor :attr:MOVE_IDENTICAL_PATH.
Returns:
-
IdenticalPathPolicy–The policy asked for, or :attr:
IdenticalPathPolicy.IGNORE.
Raises:
-
InvalidArgumentError–When the option names no policy, or is not text.
str_option ¶
Return the text key holds, or default when it holds none.
Parameters:
Returns:
-
str–The text.
Raises:
-
InvalidArgumentError–When the option holds something that is not text.
DirectoryAttributes
dataclass
¶
DirectoryAttributes(path: str, visibility: Visibility | None = None, last_modified: int | None = None, extra_metadata: Mapping[str, object] = dict[str, object]())
One directory, as a listing describes it.
There is no size and no MIME type here, and that is the point of a separate
type: a directory has neither, so asking is a mistake the type system can
catch rather than a field that is forever None.
Attributes:
-
path(str) –Where the directory is, relative to the storage's root, with no slash at either end — a backend that names directories
a/b/and one that names thema/bdescribe the same place. -
visibility(Visibility | None) –Who may read it.
-
last_modified(int | None) –When it last changed, in whole seconds since the epoch.
-
extra_metadata(Mapping[str, object]) –Whatever else the backend said about it.
with_path ¶
Return the same directory described at another path.
What a prefixing adapter or a mount manager needs: the backend answered about its own path, and the caller must be told about theirs.
Parameters:
-
path(str) –The path to describe this directory at.
Returns:
-
DirectoryAttributes–A new instance; this one is unchanged.
to_dict ¶
Return every field under its name, for a log line, a cache or a payload.
DirectoryListing ¶
Entries a storage would yield, described rather than fetched.
A directory on an object store can hold more entries than fit in memory,
and a caller usually wants a handful of them. So this holds no entries: it
holds a source factory — a zero-argument callable that starts a fresh
read — and :meth:filter and :meth:map return a new listing wrapping
that source in another step. Nothing is read, no backend is called and no
error is raised until something iterates.
listing = storage.list_contents("photos", deep=True) # no call made yet
names = listing.filter(lambda entry: entry.is_file).map(lambda entry: entry.path)
async for name in names: # the backend is read here, once
print(name)
Iterating twice calls the source factory twice, which re-reads the backend:
a listing is a question, and asking it again may rightly get a new answer.
Hold :meth:to_list instead when one answer must serve twice.
Two operations cannot stay lazy. :meth:sort_by_path has to see every
entry before it can yield the first, and :meth:to_list exists to gather
them; both read the whole listing into memory when they run.
Take the factory that starts one read of the entries.
Parameters:
-
source(Callable[[], AsyncIterator[T]]) –Called with no arguments each time iteration begins, and expected to return a fresh iterator. A factory rather than an iterator so the listing can be read more than once, and so building it costs nothing.
filter ¶
Return the entries predicate accepts, deciding as they arrive.
Nothing is read here: the returned listing remembers the question and asks it of each entry while it is being iterated.
Parameters:
Returns:
-
DirectoryListing[T]–A new listing; this one is unchanged.
map ¶
Return each entry as transform describes it, one at a time.
Nothing is read here either — the transformation runs per entry during iteration, so a listing of a million files costs one entry of memory.
Parameters:
-
transform(Callable[[T], R]) –Turns one entry into whatever the caller wants instead, a path or a payload of its own.
Returns:
-
DirectoryListing[R]–A new listing over the transformed entries; this one is unchanged.
sort_by_path ¶
Return the entries ordered by path — the one step that cannot stream.
Backends list in whatever order suits them, and code that compares two
listings, or shows one to a person, needs an order it chose. The last
entry may be the first alphabetically, so the returned listing reads
the whole source into memory before it yields anything. Keep it late in
a chain: a :meth:filter placed before it is what bounds the cost.
Returns:
-
DirectoryListing[PathT]–A new listing which, each time it is iterated, reads the source in
-
DirectoryListing[PathT]–full and yields its entries ordered by :attr:
path.
to_list
async
¶
Read every entry and return them, for when one answer must serve twice.
This is where a listing stops being a plan: it reads the source to the end and keeps the result, so nothing else has to worry that iterating again would ask the backend again.
Returns:
-
list[T]–Every entry, in the order the listing yields them.
ChecksumAlgorithmNotSupportedError ¶
Bases: UnableToProvideChecksumError
The backend keeps checksums, but not in the algorithm that was asked for.
An object store hands out the digests it happens to hold — an entity tag, an MD5 it recorded on upload — and nothing else. A storage treats this as "ask me differently", not as a failure: it reads the file and computes the digest itself. It surfaces only when a caller went to the backend directly.
Attributes:
-
location(str) –The path a checksum was wanted for, as the caller gave it.
-
algorithm(str) –The algorithm that was asked for.
-
reason(str) –Why there is none.
Record the path and the algorithm the backend does not keep.
CorruptedPathDetectedError ¶
Bases: StorageError
A path held a control or formatting character.
Anything Unicode files under "other" — a null byte, an escape, a zero-width joiner — is refused rather than passed on. Such a character survives a comparison but not a filesystem, a url or a log line, so a path holding one means two names that look identical and are not.
Attributes:
-
path(str) –The path that was refused, as the caller gave it.
Record the path that cannot be used.
FeatureNotSupportedError ¶
Bases: StorageError, NotImplementedError
The backend behind this adapter has no such capability.
Kept apart from every operation failure on purpose: a failed operation may work next time, this one never will. It says "configure something else", not "try again" — so a retry loop catching operation failures leaves it alone, and a caller that can do without the capability catches this one class and goes on.
Also a :class:NotImplementedError, which is what an implementation that
does not implement something raises.
Attributes:
-
feature(Feature) –The capability that is missing.
-
adapter(str) –The name of the adapter class that does not have it.
Record which capability is missing from which adapter.
InvalidArgumentError ¶
Bases: StorageError, ValueError
A storage, an adapter or a generator was given an argument it cannot use.
Raised where the argument is given — an empty prefix, a checksum algorithm this machine does not have, an expiry without a time zone — rather than on the first attempt to reach a backend.
Also a :class:ValueError, so code that already guards its configuration
with except ValueError keeps working without learning a new exception.
Attributes:
-
reason(str) –What is wrong with the argument.
Record what is wrong with the argument.
InvalidStreamError ¶
Bases: StorageError, TypeError
A write was given contents it cannot stream.
A stream is bytes arriving in pieces — an async iterable, a plain iterable or an open binary file. Text is not one: it would have to be encoded, and guessing which encoding is how files end up unreadable. The caller encodes and hands in bytes.
Also a :class:TypeError, because that is what handing in the wrong kind
of thing is.
Attributes:
-
received(str) –The name of the type that was handed in.
Record what was handed in instead of a stream.
InvalidVisibilityError ¶
Bases: StorageError, ValueError
A value was handed in as a visibility and is not one.
Visibility travels as a plain string through configuration files and request bodies, so it is parsed on the way in and anything unknown is refused there — before an adapter turns it into permissions or an access policy it cannot take back.
Also a :class:ValueError, so code already guarding its configuration
with except ValueError keeps working.
Attributes:
-
visibility(str) –The value that was refused.
Record the value that is not a visibility.
MissingBackendError ¶
Bases: StorageError, ImportError
The package this adapter talks to its backend through is not installed.
Backends live behind extras so an application installs only the ones it reaches. An adapter imports its package the first time it is used, and turns a failed import into this — an error naming the extra to install, rather than a traceback about a module nobody asked for.
Also an :class:ImportError, so it reads as what it is.
Attributes:
-
package(str) –The package that is missing.
-
extra(str) –The extra of this distribution that installs it.
Record the missing package and the extra that brings it.
PathTraversalDetectedError ¶
Bases: StorageError
A path walked up past the root of the storage.
Every path is resolved before it reaches a backend, and one whose ..
segments take it above the root is refused there — a storage answers for
what it holds and for nothing beside it. A storage may also refuse ..
outright, which raises this for any path carrying one.
Attributes:
-
path(str) –The path that was refused, as the caller gave it.
Record the path that climbed out.
StorageError ¶
Bases: Exception
Base class for every error raised by this library.
Catch this to handle anything storing a file can go wrong with; catch a subclass to handle one cause. Every subclass carries the data a caller needs as typed attributes and composes its own message from them, so nothing has to be parsed back out of a string.
A message is built from paths and from the reason the backend gave. It never carries a key, a token or a connection string: an error travels into logs and responses, and credentials must not travel with it.
StorageOperationFailedError ¶
Bases: StorageError
Something a storage was asked to do failed; the operation says what.
This is the line between "the backend refused this attempt" and every
other kind of error — a path that is not a path, a feature the backend
does not have, an argument that makes no sense. Code that retries, or that
turns a failure into a response, catches this one class and reads
:attr:operation rather than listing every subclass.
Whatever the backend raised is chained as __cause__.
Attributes:
-
operation(Operation) –What was being attempted. Every concrete subclass sets it; reading it off this class itself raises :class:
AttributeError, which is how a subclass that forgot to set it is found.
SymbolicLinkEncounteredError ¶
Bases: StorageError
A listing ran into a symbolic link and was told to refuse them.
A link can point anywhere, including outside the storage, so following one would quietly hand out what the storage does not hold. An adapter that meets one either skips it or raises this, depending on how it was built.
Attributes:
-
location(str) –The link that was met, as the storage names it.
Record the link that was refused.
UnableToCheckDirectoryExistenceError ¶
Bases: UnableToCheckExistenceError
Asking whether a directory is there failed.
The class name is what tells a reader a directory was being looked for; the message stays the same as its parent's so the two read alike in a log.
Attributes:
-
location(str) –The path that was checked, as the caller gave it.
UnableToCheckExistenceError ¶
Bases: StorageOperationFailedError
Asking whether a path is there failed.
Not an answer of "no": the backend could not be asked at all. The two subclasses say whether a file or a directory was being looked for, and this class catches both.
Attributes:
-
location(str) –The path that was checked, as the caller gave it.
Record the path that could not be checked.
UnableToCheckFileExistenceError ¶
Bases: UnableToCheckExistenceError
Asking whether a file is there failed.
The class name is what tells a reader a file was being looked for; the message stays the same as its parent's so the two read alike in a log.
Attributes:
-
location(str) –The path that was checked, as the caller gave it.
UnableToCopyFileError ¶
Bases: StorageOperationFailedError
Copying a file failed; the destination may be half-written.
A copy across backends is a read streamed into a write, so a failure part way through leaves whatever reached the destination. The source is never touched.
Attributes:
-
source(str) –The path that was copied from, as the caller gave it.
-
destination(str) –The path that was copied to, as the caller gave it.
-
reason(str) –What the backend said, empty when it said nothing.
Record both paths and why the copy failed.
UnableToCreateDirectoryError ¶
Bases: StorageOperationFailedError
Creating a directory failed.
Attributes:
-
location(str) –The directory that was created, as the caller gave it.
-
reason(str) –What the backend said, empty when it said nothing.
Record the directory and why creating it failed.
UnableToDeleteDirectoryError ¶
Bases: StorageOperationFailedError
Deleting a directory and everything under it failed.
The deletion is not atomic anywhere: part of the tree may be gone. A caller that must end up with nothing there deletes again rather than assuming either outcome.
Attributes:
-
location(str) –The directory that was deleted, as the caller gave it.
-
reason(str) –What the backend said, empty when it said nothing.
Record the directory and why deleting it failed.
UnableToDeleteFileError ¶
Bases: StorageOperationFailedError
Deleting a file failed while the file was there.
Deleting what is not there is not a failure: a delete asks for the file to be gone, and it is. This is raised only when the backend refused.
Attributes:
-
location(str) –The path that was deleted, as the caller gave it.
-
reason(str) –What the backend said, empty when it said nothing.
Record the path and why deleting it failed.
UnableToGeneratePublicUrlError ¶
Bases: StorageError
A file has no address anyone can fetch it from.
Usually configuration rather than breakage: nothing told the storage what its files are served under, and the backend does not know either. A chain of generators raises this once every one of them has declined.
Attributes:
-
location(str) –The path a url was wanted for, as the caller gave it.
-
reason(str) –Why there is none, empty when nothing more is known.
Record the path and why it has no public url.
UnableToGenerateTemporaryUrlError ¶
Bases: StorageError
A file has no address that lets the bearer fetch it until a deadline.
A backend signs such a url with the credentials it holds, so this covers both "this backend does not sign" and "signing failed". The message never repeats what was signed with.
Attributes:
-
location(str) –The path a url was wanted for, as the caller gave it.
-
reason(str) –Why there is none, empty when nothing more is known.
Record the path and why it has no temporary url.
UnableToListContentsError ¶
Bases: StorageOperationFailedError
Listing a directory failed, at its start or part way through.
A listing is lazy, so this can surface on any step of the iteration, not
only on the call that asked for it. :attr:deep is carried because a
shallow listing and a deep one are different requests to the backend and
fail for different reasons.
Attributes:
-
location(str) –The directory that was listed, as the caller gave it.
-
deep(bool) –Whether everything below it was being listed too.
Record the directory and how far the listing reached.
UnableToMoveFileError ¶
Bases: StorageOperationFailedError
Moving a file failed; the source may still be there.
A move across backends is a copy followed by a delete, so a failure can leave the file in both places or in neither. The error names both paths for that reason: which one to look at depends on where it stopped.
Attributes:
-
source(str) –The path that was moved from, as the caller gave it.
-
destination(str) –The path that was moved to, as the caller gave it.
-
reason(str) –What the backend said, empty when it said nothing.
Record both paths and why the move failed.
UnableToProvideChecksumError ¶
Bases: StorageError
A file's checksum could neither be read from the backend nor computed.
Not an operation failure: a checksum is a capability, and a storage that cannot get one from the backend falls back to reading the file and hashing it. This is what is left when that fails too.
Attributes:
-
location(str) –The path a checksum was wanted for, as the caller gave it.
-
reason(str) –Why there is none, empty when nothing more is known.
Record the path and why no checksum could be produced.
UnableToReadFileError ¶
Bases: StorageOperationFailedError
A file could not be read: it is not there, or the backend refused.
Missing and unreadable are one error on purpose. An object store answers
both the same way, so a caller that wants to know whether a file is there
asks file_exists rather than reading it and reading the failure.
Attributes:
-
location(str) –The path that was read, as the caller gave it.
-
reason(str) –What the backend said, empty when it said nothing.
Record the path and why reading it failed.
UnableToResolveMountError ¶
Bases: StorageError
A location could not be routed to one of the mounted storages.
Raised before anything is attempted: the location carries no mount name, carries an empty one, or names a storage nothing was mounted under. It is a mistake in the location, not a failure of the storage behind it, which is why it is not one of the operation failures.
Attributes:
-
location(str) –The location that was routed, as the caller gave it.
-
reason(str) –What is wrong with it.
Record the location and why it routes nowhere.
UnableToRetrieveMetadataError ¶
Bases: StorageOperationFailedError
A file's size, type, visibility or last write could not be read.
One error for all four, with :attr:metadata_type saying which: they fail
together — a missing file has none of them — and a caller asking for two
would otherwise catch two exception types for one cause. The classmethods
build each kind, so no call site has to spell the name.
Attributes:
-
location(str) –The path the metadata was read for, as the caller gave it.
-
metadata_type(str) –Which piece was asked for —
last_modified,visibility,file_sizeormime_type. -
reason(str) –What the backend said, empty when it said nothing.
Record the path, what was asked for, and why it could not be read.
last_modified
classmethod
¶
Build the failure for the moment a file was last written.
visibility
classmethod
¶
Build the failure for who may read a file.
file_size
classmethod
¶
Build the failure for how many bytes a file holds.
mime_type
classmethod
¶
Build the failure for what kind of content a file holds.
UnableToSetVisibilityError ¶
Bases: StorageOperationFailedError
Changing who may read a file or a directory failed.
A backend that has no notion of visibility at all raises
:class:~xtr_storage.exception.FeatureNotSupportedError instead: this one
means the backend has it and the attempt failed.
Attributes:
-
location(str) –The path whose visibility was set, as the caller gave it.
-
reason(str) –What the backend said, empty when it said nothing.
Record the path and why its visibility could not be set.
UnableToWriteFileError ¶
Bases: StorageOperationFailedError
Writing a file failed, whether or not anything was written.
A backend that fails midway may leave a partial object behind; the error says where to look, not what survived, because only the backend knows.
Attributes:
-
location(str) –The path that was written to, as the caller gave it.
-
reason(str) –What the backend said, empty when it said nothing.
Record the path and why writing it failed.
Feature ¶
Bases: StrEnum
A capability an adapter either has or does not.
An adapter that cannot do one of these raises
:class:~xtr_storage.exception.FeatureNotSupportedError naming it, rather
than failing the operation: the difference is between "this backend never
does that" and "that attempt failed", and only the first is worth changing
the configuration over.
Visibility is the only capability named this way. A checksum, a public url
and a temporary url each refuse with their own error — respectively
:class:~xtr_storage.exception.ChecksumAlgorithmNotSupportedError,
:class:~xtr_storage.exception.UnableToGeneratePublicUrlError and
:class:~xtr_storage.exception.UnableToGenerateTemporaryUrlError — so there
is nothing for them to name here.
FileAttributes
dataclass
¶
FileAttributes(path: str, file_size: int | None = None, visibility: Visibility | None = None, last_modified: int | None = None, mime_type: str | None = None, extra_metadata: Mapping[str, object] = dict[str, object]())
One file, as a listing or a metadata answer describes it.
Everything but the path is optional because backends answer different
questions for the same file: a listing may carry sizes and no MIME type,
and a metadata call fills in the one field it was asked for, leaving the
rest alone rather than paying for a second round trip. None therefore
means "not asked, or not known here" — never "empty".
Attributes:
-
path(str) –Where the file is, relative to the storage's root, with no leading slash: one file has one path, whoever wrote it.
-
file_size(int | None) –Its size in bytes.
-
visibility(Visibility | None) –Who may read it.
-
last_modified(int | None) –When it last changed, in whole seconds since the epoch.
-
mime_type(str | None) –What it holds.
-
extra_metadata(Mapping[str, object]) –Whatever else the backend said about it.
with_path ¶
Return the same file described at another path.
What a prefixing adapter or a mount manager needs: the backend answered about its own path, and the caller must be told about theirs, with every other field the backend gave kept intact.
Parameters:
-
path(str) –The path to describe this file at.
Returns:
-
FileAttributes–A new instance; this one is unchanged.
to_dict ¶
Return every field under its name, for a log line, a cache or a payload.
IdenticalPathPolicy ¶
Bases: StrEnum
What to do when a copy or a move is asked to go from a path to itself.
Backends disagree: one rewrites the file with itself, another refuses, a third truncates it on the way. The caller decides instead of the backend, so the same code behaves the same however the bytes are stored.
:attr:IGNORE is what a call that chose nothing gets, because it is the
only answer that cannot destroy the file. A move an object store cannot do
in one step is a copy followed by a delete of the source — and when the two
paths are one path, the delete removes what the copy just wrote. The two
other answers stay available to a caller who wants them, named per call or
per storage.
The members are their strings, so a configuration carries "fail".
Attributes:
-
TRY–Hand it to the adapter and live with the backend's answer, whatever that is — a rewrite, a refusal, or a file that is no longer there.
-
FAIL–Refuse before the backend is touched.
-
IGNORE–Return as though it had been done, touching nothing — unless there is no file at the path, which is reported as a missing source like any other transfer, because nothing was done and no file appeared. The default.
LinkHandling ¶
Bases: StrEnum
What an operation does with a symbolic link it meets.
A link may point anywhere, including outside the directory a storage is rooted in, so following one would hand out a file that storage never held. Neither answer is right for everyone — a deployment that plants links on purpose wants them left out quietly, one that plants none wants to hear about the first — so the adapter is told which to give.
Both answers keep the storage inside its root. This setting decides what a listing does with a link; neither value makes the tree above the root reachable through one, so a read, a write or a delete naming a path a link carries outside the root is refused either way.
Attributes:
-
SKIP–Leave links out of the listing. A link is hidden, not a door: one that resolves outside the root is still refused on a direct operation.
-
DISALLOW–Raise on the first one met.
ExtensionMimeTypeDetector ¶
Bases: MimeTypeDetectorInterface
Guesses a media type from the extension, deterministically across machines.
The standard library's shared registry reads system files (such as
/etc/mime.types) on first use, so the same extension can resolve
differently on two machines. This detector instead owns a private
MimeTypes(filenames=()) instance: passing no filenames keeps it from
reading any system file, while it still loads the interpreter's built-in
table, so the mapping depends only on the standard library, not on the host.
A handful of extensions are then added explicitly to close the remaining gap
between interpreter versions.
The lookup is case-insensitive — the underlying table folds the extension —
and an extension it does not recognise yields None rather than a guess.
Build the private table and register the explicit extensions.
detect_from_path ¶
Return the media type path's extension names, or None.
MimeTypeDetectorInterface ¶
Bases: Protocol
Turns a path into a media type without ever opening the file.
An adapter asks for a file's media type when the backend does not store one
of its own. The path is all it has — the bytes may live on another machine
and fetching them only to sniff a type would be a second round trip — so a
detector decides from the name and returns None when the name says
nothing it recognises.
MountManager ¶
MountManager(storages: Mapping[str, StorageOperatorInterface], config: Mapping[str, object] | None = None)
Several storages behind one, picked by the name a location starts with.
An application that keeps uploads in one place, generated reports in
another and a scratch area in a third ends up passing three storages
around, and every function that touches a file has to be told which one it
is for. A mount manager is the one object instead: the storages are named
once, and from then on the name travels with the path —
"uploads://avatars/1.png" — so a location is enough to act on.
Every location reads <name>://<path>. There is no default mount,
because a location that quietly went somewhere would be exactly the bug
this exists to remove, so one naming no mount, an empty mount or a mount
nothing was put under is refused with
:class:~xtr_storage.exception.UnableToResolveMountError before anything
is attempted.
Routing is all this does, bar one thing: a copy or a move between two
different storages, which no backend can be asked to perform, and which
:meth:copy and :meth:move describe.
Mount storages under their names and fix the standing options.
Parameters:
-
storages(Mapping[str, StorageOperatorInterface]) –What each mount name stands for. Copied, so a later change to the caller's mapping cannot re-route live code.
-
config(Mapping[str, object] | None, default:None) –Options every call through this manager starts from. A call's own are laid over them, and the result reaches the owning storage as that call's options.
directory_exists
async
¶
Return whether a directory is at location.
has
async
¶
Return whether anything at all is at location, file or directory.
read_stream ¶
Return the file's contents in chunks, from the storage that holds it.
list_contents ¶
Return a lazy listing of what is under location.
Every entry comes back at its full location — "uploads://a/b.png"
rather than "a/b.png" — so what a listing yields can be handed
straight back to :meth:read or :meth:copy without the caller having
to remember which mount it came from.
The mount is resolved here rather than on first iteration: routing
nowhere is a mistake in the call, and an error is worth more where the
call is than wherever the listing is eventually read. There is no default
location: a bare "" names no mount, so the caller always says which
storage to list.
last_modified
async
¶
Return when the file at path last changed, in whole epoch seconds.
write
async
¶
Write contents at location, replacing whatever was there.
write_stream
async
¶
write_stream(location: str, contents: AsyncIterable[bytes] | Iterable[bytes] | BinaryIO, config: Mapping[str, object] | None = None) -> None
Write at location from a source read in pieces.
set_visibility
async
¶
Change who may read the file or directory at path.
delete
async
¶
Delete the file at location, treating a missing one as already gone.
delete_directory
async
¶
Delete the directory at location and everything under it.
create_directory
async
¶
Create the directory at location, and the ones leading to it.
copy
async
¶
Copy the file at source to destination, overwriting it.
Between two locations on the same storage this is that storage's own
copy, which is where a backend does the work itself instead of sending
every byte out and back. Between two storages it cannot be: the file is
read out of one and written into the other a chunk at a time, and its
visibility is read separately and named in the write, because nothing
else carries it across. A visibility the call names wins over the
source's, retain_visibility: False keeps none, a source that has no
notion of visibility contributes none, and a destination that has none
ignores what it is given.
Raises:
-
UnableToResolveMountError–When either location routes nowhere.
-
UnableToCopyFileError–When the copy failed. Across two storages the read, the write and the visibility lookup all end here, carrying what actually failed as
__cause__.
move
async
¶
Move the file at source to destination, overwriting it.
Between two locations on the same storage this is that storage's own
move. Between two storages it is a copy followed by a delete, which is
the only thing it can be, and the delete is what makes it a move rather
than a copy: when it fails the file is still at source, and this
says so instead of reporting a move that did not happen. Nothing is
rolled back — the copy stands — so a caller that retries moves the same
file again rather than losing it.
Raises:
-
UnableToResolveMountError–When either location routes nowhere.
-
UnableToMoveFileError–When either half failed, carrying what actually failed as
__cause__.
public_url
async
¶
Return a lasting address for the file at path, from its own storage.
temporary_url
async
¶
Return an expiring address for the file at path, from its own storage.
checksum
async
¶
Return a digest of the file at path, from the storage that holds it.
close
async
¶
Close every mounted storage, whatever any one of them does about it.
One storage refusing to close must not leave the rest open — a pool
nobody shut is what outlives the process — so each is closed in turn,
and every failure is raised together, in an :class:ExceptionGroup, once
they all have been, so none is lost behind another. A storage with
nothing to close is passed over.
Operation ¶
Bases: StrEnum
The operation an :class:~xtr_storage.exception.StorageOperationFailedError was in.
Every failed operation carries one of these as a class attribute, so code catching the one base error can still branch on what was attempted instead of listing a dozen exception types. The values are stable, lower-case names, fit to log or to put in a response body.
PathNormalizerInterface ¶
Bases: Protocol
Resolves a caller's path to the single form every backend agrees on.
A path reaches the library the way a caller wrote it — with backslashes,
doubled slashes, . and .. segments, leading and trailing slashes.
A backend must never see that variety: two paths that name the same file
have to arrive identical, and one that climbs above the root must be caught
before a filesystem or an object store is asked about it. Every path crosses
this boundary exactly once, so the backends downstream receive plain paths
and never guard against these cases themselves.
PathPrefixer ¶
Joins a caller's path to a backend's root and strips it back off again.
A backend often stores everything under one root — a directory, a bucket key prefix — that the caller never sees. This prefixer owns that translation in one place: every path going down to the backend is prefixed, every path coming back up in a listing has the prefix removed, so the rest of the library works in caller-relative paths and the root lives here alone.
The root is stored once, in a normalized shape, so joining is a plain string concatenation with no per-call trimming. A non-empty root always ends with the separator; an empty root stays empty so a prefixer with no root is a transparent pass-through; a root that is only the separator stays the separator, keeping a leading-slash convention intact.
Store the root in the shape every method below relies on.
Parameters:
-
prefix(str) –The root every path is placed under. Trailing separators are folded so
"a"and"a/"behave identically. -
separator(str, default:'/') –The character that joins segments in this backend.
prefix_path ¶
Place path under the root, dropping any leading separator it carries.
strip_prefix ¶
Return path with the stored root removed from its front.
prefix_directory_path ¶
Place a directory path under the root, ending it with the separator.
An empty result — an empty root joined to an empty path — stays empty rather than becoming a lone separator, and a result that already ends with the separator is left as it is.
WhitespacePathNormalizer ¶
Bases: PathNormalizerInterface
Reduces a path to one canonical form and refuses the paths a backend can't trust.
The name keeps the intent explicit: the normalizer folds away the
"whitespace" of path writing — the backslashes, the doubled and edge
separators, the . and .. segments — so that two spellings of the
same file collapse to one string. It resolves in memory, never touching a
filesystem, which is why a .. is settled by counting segments rather than
by asking where a symlink points.
Two paths are refused rather than resolved. One that carries a control or
formatting character (anything Unicode files under category C) cannot be
a real name — such a character survives a comparison but not a filesystem, a
url or a log line — so it raises rather than travelling on. One whose ..
segments walk above the root would name a file the storage does not answer
for, so it raises too; a storage may also forbid .. outright, in which
case any parent segment is refused whether or not there is something to pop.
Record whether a resolvable .. segment is allowed.
Parameters:
-
allow_relative_path_traversal(bool, default:True) –When true, a
..pops the segment before it and only a..with nothing left to pop is refused. When false, any..is refused outright.
normalize_path ¶
Return path in canonical form, or raise if it cannot be trusted.
Storage ¶
Storage(adapter: StorageAdapterInterface, config: Mapping[str, object] | None = None, *, path_normalizer: PathNormalizerInterface | None = None, public_url_generator: PublicUrlGeneratorInterface | None = None, temporary_url_generator: TemporaryUrlGeneratorInterface | None = None)
One backend behind the whole reading-and-writing contract.
A storage is what an application holds and passes around: it turns a caller's path into a backend operation, and it is the single place a path is cleaned, options are layered, a stream is made uniform and a capability a backend lacks is either worked around or refused. The adapter it wraps receives the result of all that and never has to second-guess a caller.
The options a storage is built with are the standing defaults; the mapping a single call passes is laid over them for that call alone, so one upload can be public without changing what the next one does.
Wrap adapter and fix the defaults every call starts from.
Parameters:
-
adapter(StorageAdapterInterface) –The backend to drive. It is handed normalized paths and a merged :class:
~xtr_storage.Config, never a raw caller path. -
config(Mapping[str, object] | None, default:None) –The standing options a call's own options are laid over.
-
path_normalizer(PathNormalizerInterface | None, default:None) –How caller paths are reduced to the form the backend sees; a whitespace normalizer by default, reading whether
..may climb from the config. -
public_url_generator(PublicUrlGeneratorInterface | None, default:None) –A generator put in front of the backend for lasting addresses; consulted before the backend's own.
-
temporary_url_generator(TemporaryUrlGeneratorInterface | None, default:None) –The same, for expiring addresses.
directory_exists
async
¶
Return whether a directory is at location.
has
async
¶
Return whether anything at all is at location, file or directory.
read
async
¶
Return the whole contents of the file at location.
The whole file lands in memory, so reach for :meth:read_stream instead
whenever the size is a caller's to decide — an upload, a remote object —
rather than something this application wrote and knows the bound of.
read_stream ¶
Return the file's contents in chunks, reaching the backend as they are taken.
list_contents ¶
Return a lazy listing of what is under location.
Nothing is read here: the returned listing reaches the backend only
once it is iterated, and any failure that surfaces then — whatever the
backend raised — arrives as an :class:UnableToListContentsError
carrying it as __cause__. A :class:FeatureNotSupportedError
passes through unwrapped, so a caller that can list without descending
can tell "this backend cannot" apart from "this listing failed".
last_modified
async
¶
Return when the file at path last changed, in whole epoch seconds.
write
async
¶
Write contents at location, replacing whatever was there.
write_stream
async
¶
write_stream(location: str, contents: AsyncIterable[bytes] | Iterable[bytes] | BinaryIO, config: Mapping[str, object] | None = None) -> None
Write at location from a source read in pieces.
An async source is used as it is; a plain iterable is wrapped in one; an open binary file is rewound when it can be and read a megabyte at a time off the event loop. Text is refused, because encoding it is the caller's decision, not a guess this makes.
set_visibility
async
¶
Change who may read the file or directory at path.
delete
async
¶
Delete the file at location, treating a missing one as already gone.
delete_directory
async
¶
Delete the directory at location and everything under it.
create_directory
async
¶
Create the directory at location, and the ones leading to it.
move
async
¶
Move the file at source to destination, overwriting it.
Raises:
-
UnableToMoveFileError–Under the
failpolicy when the two paths are the same, and under theignorepolicy when they are the same and no file is there: nothing was moved, so answering success would tell the caller a file exists that does not.
copy
async
¶
Copy the file at source to destination, overwriting it.
Raises:
-
UnableToCopyFileError–Under the
failpolicy when the two paths are the same, and under theignorepolicy when they are the same and no file is there: nothing was copied, so answering success would tell the caller a file exists that does not.
checksum
async
¶
Return a digest of the file at path.
A backend that keeps its own digest answers from it; otherwise, or when
it keeps none of the algorithm asked for, the file's bytes are streamed
through :mod:hashlib here.
public_url
async
¶
Return a lasting address the file at path is reachable at.
An injected generator is asked first, then one built from the
public_url option, and only then the backend's own — a caller that
supplied nothing and a backend that knows nothing leave a storage with
no address to give.
temporary_url
async
¶
Return an address for the file at path that stops working at expires_at.
The deadline has to name one instant everywhere, so a moment carrying no timezone is refused before any generator is asked.
StorageOperatorInterface ¶
Bases: StorageReaderInterface, StorageWriterInterface, Protocol
Everything a storage does, and what code depending on one asks for.
Reading and writing arrive by inheritance — a storage that can do both is the ordinary case, and the two halves exist so that narrower code can ask for one. What is added here is the third thing a storage is used for: handing out an address a browser can fetch a file from, without the application serving the bytes itself.
Whether an address can be produced at all depends on the backend and on what the storage was configured with, so both methods may refuse.
public_url
async
¶
Return a lasting address the file at path is reachable at.
Parameters:
-
path(str) –The file to address.
-
config(Mapping[str, object] | None, default:None) –Options for this call alone.
Returns:
-
str–An absolute URL, which says nothing about whether the file is
-
str–actually readable by whoever follows it — that is its visibility.
Raises:
-
UnableToGeneratePublicUrlError–When nothing was configured to build one, or building it failed.
temporary_url
async
¶
Return an address for the file at path that stops working later.
How a private file is handed to a browser for one download: the address carries its own permission, and the permission runs out.
Parameters:
-
path(str) –The file to address.
-
expires_at(datetime) –When the address stops working. Must carry a timezone, since a moment without one names a different instant on every machine that reads it.
-
config(Mapping[str, object] | None, default:None) –Options for this call alone.
Returns:
-
str–An absolute URL, valid until
expires_at.
Raises:
-
InvalidArgumentError–When
expires_atcarries no timezone. -
UnableToGenerateTemporaryUrlError–When nothing was configured to build one, or building it failed.
checksum
async
¶
Return a digest of the file at path, for comparing it with another.
Parameters:
-
path(str) –The file to digest.
-
config(Mapping[str, object] | None, default:None) –Options for this call alone, the algorithm among them.
Returns:
-
str–The digest in lowercase hexadecimal, unless the algorithm asked for
-
str–is one the backend keeps in a form of its own.
Raises:
-
InvalidArgumentError–When the algorithm asked for is unknown.
-
UnableToProvideChecksumError–When the file could not be digested.
StorageReaderInterface ¶
Bases: Protocol
Reads files and tells what is there, without ever changing it.
Asking for this rather than the whole storage is how a caller says it only looks: a template renderer, a download endpoint, a report that walks a directory. The type then refuses the write that a later change might otherwise slip in.
Paths are text, relative to the storage's root, separated by / and
carrying no leading slash. Every method that reaches a backend is awaited;
:meth:list_contents is not, because it hands back a plan for reading
rather than the reading itself.
file_exists
async
¶
Return whether a file is at location.
Parameters:
-
location(str) –Where to look.
Returns:
-
bool–True when a file is there — a directory at that path answers False.
Raises:
-
UnableToCheckFileExistenceError–When the backend could not say.
directory_exists
async
¶
Return whether a directory is at location.
Parameters:
-
location(str) –Where to look.
Returns:
-
bool–True when a directory is there — a file at that path answers False.
Raises:
-
UnableToCheckDirectoryExistenceError–When the backend could not say.
has
async
¶
Return whether anything at all is at location, file or directory.
Parameters:
-
location(str) –Where to look.
Returns:
-
bool–True when either a file or a directory is there.
Raises:
-
UnableToCheckExistenceError–When the backend could not say.
read
async
¶
Return the whole contents of the file at location.
Parameters:
-
location(str) –The file to read.
Returns:
-
bytes–Every byte of it, in memory. Prefer :meth:
read_streamfor a file -
bytes–whose size the caller does not control.
Raises:
-
UnableToReadFileError–When the file is missing or unreadable.
read_stream ¶
Return the file's contents in chunks, so its size need not fit in memory.
Not awaited: the call itself does nothing, and the backend is reached as the chunks are taken.
Parameters:
-
location(str) –The file to read.
Returns:
-
AsyncIterator[bytes]–An iterator over the file's bytes, in order, in chunks of whatever
-
AsyncIterator[bytes]–size the adapter finds efficient.
Raises:
-
UnableToReadFileError–When the file is missing or unreadable — raised while iterating, not when calling.
list_contents ¶
Return what is under location, as a listing that has read nothing yet.
Not awaited, because nothing has happened: the returned listing can be narrowed and transformed first, and reaches the backend only once something iterates it.
Parameters:
-
location(str) –The directory to look in. An implementation that has a single root — a :class:
~xtr_storage.Storage— may default it to that root; one routing by name — a :class:~xtr_storage.MountManager— requires it, since an empty location names no mount. -
deep(bool, default:False) –Whether to descend into the directories found, rather than naming them and stopping.
Returns:
-
DirectoryListing[StorageAttributes]–A lazy listing of files and directories.
Raises:
-
UnableToListContentsError–When the backend could not be read — raised while iterating, not when calling.
last_modified
async
¶
Return when the file at path last changed.
Parameters:
-
path(str) –The file to ask about.
Returns:
-
int–Whole seconds since the epoch, in UTC.
Raises:
-
UnableToRetrieveMetadataError–When the backend does not know.
file_size
async
¶
Return the size of the file at path.
Parameters:
-
path(str) –The file to ask about.
Returns:
-
int–Its size in bytes.
Raises:
-
UnableToRetrieveMetadataError–When the backend does not know, or
pathnames a directory.
mime_type
async
¶
Return what the file at path holds.
Parameters:
-
path(str) –The file to ask about.
Returns:
-
str–A media type such as
"image/svg+xml", from what the backend -
str–stored alongside the file or, failing that, from its extension.
Raises:
-
UnableToRetrieveMetadataError–When neither says.
visibility
async
¶
Return who may read the file at path.
Parameters:
-
path(str) –The file to ask about.
Returns:
-
Visibility–The visibility the backend reports.
Raises:
-
UnableToRetrieveMetadataError–When the backend does not know.
-
FeatureNotSupportedError–When the backend has no notion of visibility at all.
StorageWriterInterface ¶
Bases: Protocol
Writes, deletes, moves and copies — everything that leaves a mark.
The counterpart of :class:~xtr_storage.StorageReaderInterface, kept apart
so an upload handler can be handed the ability to write without being
handed the ability to read every other file in the storage.
Each method takes an optional mapping of options which is laid over the
ones the storage was built with, for that call alone: a visibility, a
content type, whatever the backend understands. Paths are text, relative to
the storage's root, separated by /.
write
async
¶
Write contents at location, replacing whatever was there.
Missing directories on the way are created.
Parameters:
-
location(str) –Where the file goes.
-
contents(bytes) –What it holds.
-
config(Mapping[str, object] | None, default:None) –Options for this call alone.
Raises:
-
UnableToWriteFileError–When the backend refused the write.
write_stream
async
¶
write_stream(location: str, contents: AsyncIterable[bytes] | Iterable[bytes] | BinaryIO, config: Mapping[str, object] | None = None) -> None
Write at location from a source read in pieces.
For contents that should not be held in memory at once, and for the upload handles a web framework hands over.
Parameters:
-
location(str) –Where the file goes.
-
contents(AsyncIterable[bytes] | Iterable[bytes] | BinaryIO) –Chunks to write in order, from an async source, a plain iterable, or an open binary file, which is rewound first.
-
config(Mapping[str, object] | None, default:None) –Options for this call alone.
Raises:
-
InvalidStreamError–When
contentsis none of the three. -
UnableToWriteFileError–When the backend refused the write.
set_visibility
async
¶
Change who may read the file or directory at path.
Parameters:
-
path(str) –What to change.
-
visibility(Visibility | str) –The visibility to set, or its name.
Raises:
-
InvalidVisibilityError–When the name stands for no visibility.
-
UnableToSetVisibilityError–When the backend refused the change.
-
FeatureNotSupportedError–When the backend has no notion of visibility at all.
delete
async
¶
Delete the file at location.
Deleting a file that is not there is not an error: the caller asked for it to be gone, and it is.
Parameters:
-
location(str) –The file to delete.
Raises:
-
UnableToDeleteFileError–When the backend refused the deletion.
delete_directory
async
¶
Delete the directory at location and everything under it.
A directory that is not there is not an error, for the same reason.
Parameters:
-
location(str) –The directory to delete.
Raises:
-
UnableToDeleteDirectoryError–When the backend refused the deletion.
create_directory
async
¶
Create the directory at location, and the ones leading to it.
Creating one that already exists is not an error.
Parameters:
-
location(str) –The directory to create.
-
config(Mapping[str, object] | None, default:None) –Options for this call alone, a directory visibility among them.
Raises:
-
UnableToCreateDirectoryError–When the backend refused.
move
async
¶
Move the file at source to destination, overwriting it.
Asked to move a file onto itself, nothing is done unless the options say
otherwise: see :class:~xtr_storage.IdenticalPathPolicy.
Parameters:
-
source(str) –The file to move.
-
destination(str) –Where it ends up.
-
config(Mapping[str, object] | None, default:None) –Options for this call alone; what an identical source and destination does is one of them.
Raises:
-
UnableToMoveFileError–When the backend refused the move, or the two paths are the same and the options say that is a mistake.
copy
async
¶
Copy the file at source to destination, overwriting it.
Asked to copy a file onto itself, nothing is done unless the options say
otherwise: see :class:~xtr_storage.IdenticalPathPolicy.
Parameters:
-
source(str) –The file to copy.
-
destination(str) –Where the copy goes.
-
config(Mapping[str, object] | None, default:None) –Options for this call alone; what an identical source and destination does is one of them.
Raises:
-
UnableToCopyFileError–When the backend refused the copy, or the two paths are the same and the options say that is a mistake.
ChainedPublicUrlGenerator ¶
Bases: PublicUrlGeneratorInterface
Tries generators in order, taking the first address any of them will give.
A store may be reachable more than one way — a fast delivery host for most files, a signing generator for the few that need it — and which applies is decided per file rather than up front. Each generator declines a file it cannot address by raising, which is this chain's cue to try the next; only when every one has declined does the chain itself decline.
Fix the generators to consult, in the order they are consulted.
Parameters:
-
generators(Sequence[PublicUrlGeneratorInterface]) –The generators to try; an empty run always declines, since there is nothing to ask.
public_url
async
¶
Return the first address any generator gives for path.
Parameters:
-
path(str) –The file to address, already normalized.
-
config(Config) –The options in force, passed on to each generator unchanged.
Returns:
-
str–The address of the first generator that does not decline.
Raises:
-
UnableToGeneratePublicUrlError–When every generator declined, or there were none to ask.
PrefixPublicUrlGenerator ¶
Bases: PublicUrlGeneratorInterface
Joins every path to one address, the way a single delivery host serves a store.
The commonest case: one content delivery host publishes the whole store, so a file's address is that host followed by the file's path. The prefix owns that join, reusing the same path arithmetic the adapters use to sit under a root, so a trailing slash on the prefix or a leading slash on the path never doubles or drops one.
Fix the address every path is placed under.
Parameters:
-
prefix(str) –The host and any base path files are reachable from, such as
"https://cdn.example.com/assets/".
public_url
async
¶
PublicUrlGeneratorInterface ¶
Bases: Protocol
Says where a stored file is reachable from, so the application need not serve it.
Two kinds of thing satisfy this. An adapter whose backend owns an address for every object it holds, which a storage discovers at runtime. And a standalone generator put in front of a storage — a content delivery host the files are published under, which the backend knows nothing about.
Producing an address is not a permission check: the address may well lead to a file nobody but its owner may read.
public_url
async
¶
Return a lasting address for the file at path.
Parameters:
Returns:
-
str–An absolute URL.
Raises:
-
UnableToGeneratePublicUrlError–When no address can be built for this file — which a chain of generators takes as its cue to try the next one.
ShardedPrefixPublicUrlGenerator ¶
Bases: PublicUrlGeneratorInterface
Picks one of several addresses per file, so no single host serves everything.
Several delivery hosts publish the same store; spreading files over them keeps any one from carrying the whole load. The choice is a hash of the path rather than a counter, so the same file always resolves to the same host — an address stays quotable and a browser's cache stays warm — while different files land evenly across the hosts.
Fix the addresses files are spread over.
Parameters:
-
prefixes(Sequence[str]) –The hosts, each a base a file may be reachable from. Order matters, since a path's host is chosen by its position.
Raises:
-
InvalidArgumentError–When no prefix is given, since there would be nothing to place a file under.
TemporaryUrlGeneratorInterface ¶
Bases: Protocol
Hands out an address that carries its own permission, and loses it in time.
How a private file reaches a browser without the application streaming the bytes: the address is signed, whoever holds it may fetch that one file, and after the moment it was signed for it leads nowhere.
Usually satisfied by an adapter whose backend can sign — a storage looks for that at runtime — but a generator of one's own may be put in front instead.
temporary_url
async
¶
Return an address for the file at path, good until expires_at.
Parameters:
-
path(str) –The file to address, already normalized.
-
expires_at(datetime) –When the address stops working. A storage has already refused a moment carrying no timezone, so this names one instant everywhere.
-
config(Config) –The options in force.
Returns:
-
str–An absolute URL.
Raises:
-
UnableToGenerateTemporaryUrlError–When the address could not be built or signed.
Visibility ¶
Bases: StrEnum
Whether a stored file is readable by anyone, or by its owner alone.
Two values rather than a backend's permission model: a mode bit on a local file and an access list on an object store have nothing in common, and code that stores files only ever cares which of the two it asked for. Each adapter translates them, and one that cannot say raises rather than guess.
The members are their strings, so an option mapping, a configuration file
or an environment variable carries "public" and is understood without a
conversion step — while code that reads them still gets a closed set.
parse
classmethod
¶
Return the member value stands for.
The single door for anything arriving from outside — a config option, a bundle definition, a caller's keyword — so a typo is refused once, here, instead of reaching an adapter as an unknown permission.
Parameters:
-
value(Visibility | str) –A member, or the string one carries.
Returns:
-
Visibility–The member named by
value.
Raises:
-
InvalidVisibilityError–When no member carries that string.