IF Commons GitHub

AnyFS S3 store

0002Active

0002-anyfs-s3-store · owner if · domain commons

Requirements

FR-001checkpassing

A bucket name MUST be made only by parseBucketName, which MUST accept 3 to 63 lower case letters, digits, hyphens and dots, beginning and ending with a letter or a digit, and MUST refuse two dots together, a dot beside a hyphen, a name written as an address of the internet, and the reserved prefixes and suffixes of S3.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/keys.test.ts: the names of buckets

Proved by: anyfs-s3/suite/keys.test.ts

Written in: anyfs-s3/src/bucket.ts, anyfs-s3/src/mod.ts

Tests: passed: suite/keys.test.ts a bucket name is 3 to 63 lower case letters, digits, hyphens and dots, beginning and endin

FR-002checkpassing

The key of a path MUST be the prefix of the place, a / and the path (the path alone below the root), written in a URL segment by segment as RFC 3986 says with the / kept; a key that is not the key of a path of the store (outside the prefix, not a path, not in normalization form C, or a name that begins with .anyfs-) MUST be a finding and MUST NOT be listed or read, and a key listed with encoding-type=url MUST be read back whatever it holds.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/keys.test.ts: keys, paths, encoding and findings

Proved by: anyfs-s3/suite/keys.test.ts

Written in: anyfs-s3/src/keys.ts

Tests: passed: suite/keys.test.ts a key is encoded in a URL segment by segment as RFC 3986 says, the slash kept, and a liste; passed: suite/keys.test.ts a key that is not the key of a path of the store is a finding that says which kind: outsid; passed: suite/keys.test.ts property: any path below any prefix goes to a key and back to the same path, and the encod; passed: suite/keys.test.ts the key of a path is the prefix, a slash and the path (the path alone below the root), and

FR-003checkpassing

The XML of an answer MUST be read by a reader that takes one root element, its children, its text and the five named and the numeric entities, and MUST refuse a document type, a comment, CDATA, an instruction, a character XML 1.0 does not allow, text after the root, and a document past a bound of size, depth or elements; a listing and an error MUST be read from it.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/xml.test.ts: the strict reader and its refusals

Proved by: anyfs-s3/suite/xml.test.ts

Written in: anyfs-s3/src/xml.ts

Tests: passed: suite/xml.test.ts a document past a bound of size, depth or number of elements is refused, and one at the bo; passed: suite/xml.test.ts a listing is read as S3 writes it: the keys as given, the entity tags without their quotes; passed: suite/xml.test.ts an error is read as its code, message and request id, and an answer that is neither a list; passed: suite/xml.test.ts hostile documents are refused: a document type with entities (the billion laughs and the e; passed: suite/xml.test.ts property: reading text never throws, and a name or a text made of any characters that XML and 1 more

FR-004checkpassing

Every failure of the service MUST be of one of thirteen kinds, by its code where it is known and by its status where it is not, and every kind MUST be a problem of AnyFS by a total table: a failed condition is a conflict only where the path, the expectation and what the object is at are known, a throttle and a fault of the server MUST be retryable, a refusal of the keys MUST NOT be, and a sentence MUST hold no more than the service said.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/errors.test.ts: the total table of errors

Proved by: anyfs-s3/suite/errors.test.ts

Written in: anyfs-s3/src/errors.ts

Tests: passed: suite/errors.test.ts a failure is of the kind its code says where the code is known, and of the kind its status; passed: suite/errors.test.ts a sentence says what the service answered, cut short, and the one for a signature that doe; passed: suite/errors.test.ts every kind is a problem of AnyFS: a missing object is missing, a failed condition is a con

FR-005checkpassing

The keys of the service MUST come from a source (a pair given in code, the environment through a function, or a provider that is asked again only after its time, which includes a secret asked of a person through the credential port of Headless), and no sentence, URL or recorded request MUST hold a key.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/credentials.test.ts: the sources of keys

Proved by: anyfs-s3/suite/credentials.test.ts

Written in: anyfs-s3/src/credentials.ts

Tests: passed: suite/credentials.test.ts a provider is asked again only after its time, a static pair never, and a provider that fa; passed: suite/credentials.test.ts a secret is asked of a person through the credential port once, never written anywhere, an; passed: suite/credentials.test.ts a static pair is refused where the id or the secret could not be a key, and is given as it; passed: suite/credentials.test.ts the environment is read only through the function it is given, and a pair that is not both

FR-006checkpassing

A request MUST be sent through a transport that gives the status, the headers in lower case and the bytes, MUST NOT follow a redirect, and a connection that fails MUST be a value of the call and never thrown.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/sigv4.test.ts: failures of the transport are values

Proved by: anyfs-s3/suite/sigv4.test.ts

Written in: anyfs-s3/src/transport.ts

Tests: passed: suite/sigv4.test.ts a connection that fails and a key that cannot be had are values of the call and never thro

FR-007checkpassing

A request MUST be signed by Signature Version 4 for the service s3 in the path style or the virtual-hosted style asked for, with the SHA-256 of the payload signed on a connection that is not encrypted and UNSIGNED-PAYLOAD on one that is unless told otherwise, and the signatures MUST be those AWS publishes for its own examples of S3.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/sigv4.test.ts: the examples AWS publishes

Proved by: anyfs-s3/suite/sigv4.test.ts

Written in: anyfs-s3/src/client.ts

Tests: passed: suite/sigv4.test.ts a connection that fails and a key that cannot be had are values of the call and never thro; passed: suite/sigv4.test.ts a GET of an object with a Range is signed as the AWS example of S3 is (signature f0e8bdb8…; passed: suite/sigv4.test.ts a PUT of an object with a body, a date and a storage class is signed as the AWS example is; passed: suite/sigv4.test.ts a request of the bucket with a bare query (`?lifecycle`) and one with sorted values (`?max; passed: suite/sigv4.test.ts path style puts the bucket in the path and virtual-hosted style in the host, each key enco and 1 more

FR-008checkpassing

Opening a store MUST probe the endpoint below the prefix with an object of its own that it removes whatever the answers were, MUST find whether the keys may write, whether If-None-Match: * and If-Match on a write and If-Match on a delete are honoured and whether the checksum is kept, and MUST make a store with cas only where every condition is honoured, a store without cas where one is not, and a store that can only read where the keys may not write; the type of the store MUST say which.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/store.test.ts: the probe and the capabilities it decides

Proved by: anyfs-s3/suite/store.test.ts

Written in: anyfs-s3/src/probe.ts

Tests: passed: suite/store.test.ts a store is not made where the bucket is not there, the keys are not accepted, the signatur; passed: suite/store.test.ts a store of an endpoint that keeps every condition has cas, and the capabilities are exactl; passed: suite/store.test.ts an endpoint that ignores If-Match, or ignores it on a delete, is made without cas in its t; passed: suite/store.test.ts an endpoint that ignores If-None-Match is refused the write that expects no object, with t; passed: suite/store.test.ts keys that may only read give a store that can only read, with the capabilities of read, li

FR-009checkpassing

The revision of an object MUST be sha256: and the digest of its bytes, kept in x-amz-meta-anyfs-sha256 and checked on every read (a read whose bytes are not that digest is tampered) and sent as x-amz-checksum-sha256 where the service keeps it; an object written by another tool MUST be at the digest of its bytes; and with revisions: "etag" the revision MUST be etag: and the entity tag, the other scheme MUST be refused as an expectation, and each MUST be told so by facts.digest.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/store.test.ts: revisions, foreign objects, tampering and the two schemes

Proved by: anyfs-s3/suite/store.test.ts

Written in: anyfs-s3/src/store.ts

Tests: passed: suite/store.test.ts a read whose bytes are not the digest the metadata names is tampered, and a foreign object; passed: suite/store.test.ts an object written by another tool has no digest of ours and is at the digest of its bytes,; passed: suite/store.test.ts in the mode of the entity tag a revision is etag: and the tag, the digest revision is not ; passed: suite/store.test.ts the revision of an object is sha256: and the digest of its bytes, the same as a store in m

FR-010checkpassing

A write that expects no object MUST be If-None-Match: * and one that expects a revision MUST be If-Match on the entity tag, so that of writers that race exactly one wins and the others conflict with what the object is at; where the endpoint ignores If-None-Match a write that expects no object MUST be refused, with the reason, and never made.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/store.test.ts: races of writers on both servers

Proved by: anyfs-s3/suite/store.test.ts

Written in: anyfs-s3/src/store.ts

Tests: passed: suite/store.test.ts of writers that race to make one object, or to replace one revision, exactly one wins and

FR-011checkpassing

A read, a stat and a delete MUST end as the operation of AnyFS says: a missing object is missing, a delete of a missing object succeeds, a delete that expects a revision removes the object only at it and a delete that expects no object conflicts where there is one.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/store.test.ts: delete at a revision and delete of nothing

Proved by: anyfs-s3/suite/store.test.ts

Written in: anyfs-s3/src/store.ts

Tests: passed: suite/store.test.ts a delete that expects a revision removes the object only at it, and a delete of a missing

FR-012checkpassing

A list MUST be in the order of the paths, below its prefix only, in pages of any size up to a thousand with a cursor until the last page and a full page where there is a cursor, for a bucket of more than a thousand objects.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/store.test.ts: names, pages of more than a thousand and keys that are no paths

Proved by: anyfs-s3/suite/store.test.ts

Written in: anyfs-s3/src/store.ts

Tests: passed: suite/store.test.ts a list of more than a thousand objects is given in pages of any size in the order of the p; passed: suite/store.test.ts keys that are no path of the store are not listed and not readable: another prefix that be; passed: suite/store.test.ts names with spaces, plus signs, percent signs, ampersands, equals signs, hashes, question m

FR-013checkpassing

An object of more than maxBytes (64 MiB where it is left out) MUST be refused before it is sent, and the store MUST NOT have batch, rename, history, a range, append or search, MUST say capability-absent where it is asked for one, and MUST NOT retry a call, which the layer retrying does for the problems that say they are worth one.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/store.test.ts: connection loss, throttling and the limits

Proved by: anyfs-s3/suite/store.test.ts

Written in: anyfs-s3/src/store.ts

Tests: passed: suite/store.test.ts a connection that fails during an operation is a problem worth a retry, and an object larg; passed: suite/store.test.ts a request the service throttles ends as a problem worth a retry, and the layer of retries

FR-014checkpassing

The store MUST pass every case of the conformance suite of AnyFS in both modes of revision, and every law of the conformance suite of the bridge between a bucket and memory and between buckets, on the server made in process and on RustFS where it can be run.

Enforced (check): .github: cd commons/anyfs-s3 && bun run verify — suite/store.test.ts and suite/bridge.test.ts: the conformance suites on both servers

Proved by: anyfs-s3/suite/bridge.test.ts, anyfs-s3/suite/store.test.ts

Written in: anyfs-s3/src/store.ts

Tests: passed: suite/bridge.test.ts a tree copied to a bucket and synced back is the same tree, and a second sync of either wa; passed: suite/bridge.test.ts every law of a transfer holds between memory and a bucket, a bucket and memory, and a buck; passed: suite/store.test.ts the store passes every case of the conformance suite of AnyFS, with revisions of the diges

Acceptance scenarios

  1. Given an endpoint that ignores If-Match, When a store is opened, Then it has no cas in its type, per FR-008.
  2. Given twelve writers that race to make one object, When they write, Then one wins and eleven conflict, per FR-010.
  3. Given a bucket of 1105 objects, When it is listed in pages, Then every object is seen once in order, per FR-012.

Out of scope

  • Multipart upload of an object larger than maxBytes, ranges and streams, versioned buckets (history), server-side copy, and the credentials of a cloud's own role: later capabilities.

Edge cases

  • A key written by another tool in decomposed Unicode: not listed, per FR-002.
  • A bucket in another region than the endpoint: a redirect, said as such, per FR-006.
  • A clock that is far from the service's: said, per FR-004.

Assumptions

  • The endpoint is strongly consistent for objects, as Amazon S3 has been since 2020.

Open questions

  • None.

Key entities

  • The probe - the writes and deletes the store makes of an object of its own to find what an endpoint keeps.
  • The revision - sha256: and the digest of the bytes, or etag: and the entity tag.

Success criteria

SC-001success criterion

Both servers pass the same contract.

Review & acceptance checklist

  • [x] Every requirement is testable (MUST / MUST NOT), not aspirational
  • [x] No company fact is asserted here
  • [x] Every open item is marked, not silently decided

Titles, headings, exports, tables, columns and requirements. Esc closes.