Storage And Uploads - developerz-ai/ultimate GitHub Wiki

Storage and uploads

Call sites name a disk, never a driver. Package @ultimat3/storage (tier 1) โ€” the full reference is packages/storage/README.md, and the upload design is docs/architecture/17-uploads.md.

import { defineStorage, disk, localDriver, s3Driver, scopedKey } from '@ultimat3/storage';

defineStorage({
  disks: {
    uploads: localDriver({ root: '.storage/uploads' }),
    media: s3Driver({ bucket: 'media', endpoint: process.env.S3_ENDPOINT, forcePathStyle: true }),
  },
  default: 'uploads',
});

await disk('media').put(scopedKey(orgId, 'avatars', 'a.png'), bytes, { contentType: 'image/png' });

Swapping local for s3 changes no call site, and x dev needs no S3 server.

Driver Backing For
localDriver Bun.file under one root; writes are staged and renamed in an order whose every crash point reads as absent, whole or untyped โ€” never a wrong content type dev, tests, a single node. A key cannot be a path prefix of another (a and a/b): the second is X_STORAGE_KEY_CONFLICT. s3Driver and memoryStorageDriver hold both
s3Driver Bun.s3 any S3-compatible endpoint โ€” AWS, R2, a self-hosted gateway. The difference is endpoint, region and forcePathStyle. Credentials are env var names, never literals
memoryStorageDriver a Map in this process a test's disk: defineStorage({ disks: { uploads: memoryStorageDriver() } }). Every method localDriver answers, the same refusals, the same signing rule; objects() is a copy of the stored bytes by key. Never a deployment's disk โ€” a restart is every object gone

A declaration holds a disk by THUNK, never by value. disk('sessions') resolves through defineStorage(), which boot runs after an app's modules were evaluated โ€” so a scrape() or any other declaration built at import time takes () => disk('sessions') and reads it when it is used (Scraping). Called at module scope, disk() is X_CONFIG_INVALID: storage was not defined yet.

A real S3 on a laptop is docker/docker-compose.dev.yml's s3 service โ€” Versity S3 Gateway over a volume, with the bucket already there (a directory under its root is a bucket, and the volume is mounted at one). Point the app at it with S3_ENDPOINT=http://127.0.0.1:9000, S3_BUCKET=<app>, S3_FORCE_PATH_STYLE=1 and the key pair the file ships. That file chooses a server for development only โ€” production is whichever S3-compatible endpoint you run.

S3_REGION is the region requests are signed for. Unset, they are signed for auto: what R2 wants and what the dev gateway is started with. An endpoint that expects another region refuses the first write, list or delete with X_CONFIG_INVALID, and the fix line names the value to set.

A listing is the framework's one page

disk(name).list({ prefix, cursor, limit }) answers Page<StorageListEntry> โ€” { rows, nextCursor, hasMore }, the shape findMany and a query's .page() answer. nextCursor is a string exactly when another object exists past the page, on every driver: a full last page is hasMore: false, never a cursor to an empty one. cursor: null (or absent) is the first page, so cursor = page.nextCursor until it is null walks the prefix. limit is a positive integer or X_INVARIANT. Until 25.0.0 it was { objects, truncated, cursor? } (Upgrading).

Keys are refused, never sanitised

assertSafeKey() runs on every key: .., absolute keys, backslashes, control bytes, encoded separators, empty segments and over-long keys are X_STORAGE_PATH_UNSAFE. scopedKey(org, โ€ฆ) is org/<org>/โ€ฆ; a surface guards a client key with isWithinOrg(key, ctx.actor.orgId) and isTenantScoped(key) (X_STORAGE_ORG_MISMATCH).

The direct-upload round trip

User uploads never go through put() (which buffers, and stops at maxPutBytes, 10 MB by default). They go straight to the disk, in three calls โ€” and the client never names the key:

Hop Call What it guarantees
1. server, in an action grantUpload({ disk, orgId: ctx.actor.orgId, request, policy }) a signed grant for a key the server chose, under the actor's org
2. browser uploadFile({ file, grant, onProgress }) a PUT with real progress; storage/upload-client.ts is the one XHR seam in the browser
3. server, at PUT /_storage/:disk/*key mounted by the framework in x dev and runRole, around acceptSignedUpload. Answers 201 { key } refuses a bad or expired signature, a key outside the actor's org, more bytes or another type than was granted, or magic bytes that contradict the type

The mounted PUT (As of 2026-09-24, #523) requires a signed-in actor, like the GET. It needs no permission: the grant is the authorization, and the action that minted it ran its own policy. The route checks the signature and the expiry, and it checks that the key is inside the actor's org. It validates the upload against what the grant signed (the content type and maxBytes), so a PDF granted under uploadPolicy({ allowedContentTypes: ['application/pdf'] }) is accepted, and it does not use uploadPolicy()'s image-only default. The signed maxBytes caps how much of the body is read. bodyLimitBytes does not, and neither does an unverified query value.

The signature covers the constraints (the disk's base path, method, key, expiry, maxBytes, content type), so editing ?x-max= invalidates it โ€” and so does moving the URL to another disk: every local disk signs with the one STORAGE_SIGNING_SECRET, and the base path (/_storage/<disk>) is what keeps a grant for uploads from verifying on evidence. Uploads are sniffed: validateUpload() reads magic bytes and refuses a .png that is really HTML (X_STORAGE_TYPE_REJECTED). ISO base media files are told apart by their major brand, so image/avif, image/heic, video/quicktime, audio/mp4 and video/mp4 each pass when the policy allows them; audio/x-m4a and video/x-m4v are read as audio/mp4 and video/mp4.

On an s3 disk the grant's maxBytes is measured later, not at the PUT. A provider presign carries no size, so the browser's PUT is unbounded there. promoteAttachment is where it is measured, and get() refuses an object over the disk's maxGetBytes (default: its maxPutBytes, 10 MB) with X_STORAGE_TOO_LARGE โ€” read anything larger with stream().

The local disk signs with STORAGE_SIGNING_SECRET. The shipped development secret is accepted only in development or test; a process that names no environment is treated as production and refused at construction (X_ENV_MISSING).

Which disks /_storage and /media serve

As of 2026-09-24 (#524): the app's own. Both routes read the process's one registry per request. That registry is the last defineStorage() call, which is the app's when an app module declares its disks. The boot's env-selected disk (object on S3_ENDPOINT, else local) is served only when nothing declared one. A defineStorage at module scope in any file under apps/*/ is enough:

// apps/web/shared/storage.ts
export const storage = defineStorage({
  disks: { uploads: localDriver({ root: '.storage/uploads' }), evidence: s3Driver({ bucket }) },
  default: 'uploads',
});
storage:read the GET requires it. An app that serves nothing through these routes leaves it out of its permission set: every read is then 401 for nobody and 403 X_FORBIDDEN for everyone else (27.0.0). Until then an undeclared storage:read was a runtime 500 X_PERMISSION_UNKNOWN, which x verify's policy step reported
the boot's own disk still built. In production with no S3_ENDPOINT, that is a local disk and still needs STORAGE_SIGNING_SECRET. To skip it, export runtime = { storage } from apps/<app>/runtime.ts. runRole reads that file, and it replaces the env-selected disk entirely
definedStorage() the registry or undefined, without a throw. It is the question the routes ask

Who may read a stored object

As of 2026-10-08 (27.0.0): GET /_storage/:disk/*key and GET /media/*key decide on the session, the policy and the key โ€” in that order โ€” and a key no tenant owns is refused unless the app shared it.

Key Served to Otherwise
org/<id>/โ€ฆ a signed-in actor holding storage:read whose orgId is <id> X_STORAGE_ORG_MISMATCH (404)
under a prefix defineStorage({ shared }) lists for that disk a signed-in actor holding storage:read โ€”
anything else โ€” db-export/โ€ฆ, raw/โ€ฆ, dsr/โ€ฆ nobody X_STORAGE_KEY_UNSHARED (404)

Both refusals are decided from the key alone, before a disk is read, so neither says whether an object exists. Until 26.x the third row was served to every holder of storage:read: "not another tenant's" was read as "everybody's".

import { defineStorage, localDriver, s3Driver } from '@ultimat3/storage';

defineStorage({
  disks: {
    uploads: localDriver({ root: '.storage/uploads' }),
    evidence: s3Driver({ bucket: 'evidence' }),
  },
  shared: { uploads: ['brand/'] }, // whole segments, ending in '/'; never 'org/'
});

A prefix is refused at boot (X_CONFIG_INVALID) when it names an undeclared disk, lacks its trailing /, would share the whole disk, or sits inside org/. storage.isShared(disk, key) is the question both routes ask. Objects a stranger may fetch belong in apps/web/site/assets/ (content-hashed, no disk at all), not in a shared prefix.

The GET does not read x-sig or x-exp. A signed GET URL localDriver mints points at this route, and is served on the terms above whatever its signature says: it neither opens an unshared key nor outlives the session. To serve an object by signature alone (no session), mount your own route around readSignedObject({ url, disk, orgId }), or return the bytes from an action.

Test the real route. storageRoutes({ storage }) from @ultimat3/cli returns the mounted GET and PUT (servedStorage(storage) is the registry they read), as assetRoutes returns /media, so an app's contract test drives the route its deploy serves rather than a copy.

The body is streamed (stat + stream), never buffered: an object past the disk's maxGetBytes is served, and a Range reads and drops the bytes before its start rather than holding the object. /media still buffers โ€” it decodes images.

Attachments, quarantine, orphans

An upload lands under org/<org>/pending/โ€ฆ before its row exists, and is promoted once it has one:

Call Does
promoteAttachment({ disk, key, orgId, target, policy }) measure, copy, then delete, onto org/<org>/<entity>/<id>/<field>/โ€ฆ. policy is required โ€” pass the one the upload was granted under: the object's size is read with stat() and one over policy.maxBytes is X_STORAGE_TOO_LARGE and stays pending. Only a pending key: another row's attached key is X_STORAGE_NOT_PENDING. A second call for a key an earlier call already moved is X_STORAGE_ALREADY_PROMOTED (409, meta.attachedKey) and touches nothing: answer the row that holds that key, and never delete it in your own cleanup
grantUpload({ โ€ฆ, quarantine: true }) โ†’ releaseQuarantine({ disk, key, orgId }) the place for your scanner: a quarantined key cannot be promoted (X_STORAGE_QUARANTINED) until your scan job releases it. The scanner is your app's (axiom 8)
sweepOrphans({ disk, orgId, olderThanMs }) deletes stale pending/ keys of one org and answers { deleted, failed } โ€” a refused delete is reported, never counted as done

Images

A variant's key is the whole source key plus the transform โ€” photos/hero.png at width 640 is photos/[email protected] โ€” so two sources that differ only by extension never share a cached variant. width and height are whole numbers of at least 1 (X_INVARIANT otherwise).

transformImage() and blurPlaceholder() run on core's pipeline (Bun.Image, no sharp) and encode png, jpeg and webp; avif is key and srcset math only. variantKey(), srcsetDescriptors() and fitDimensions() are pure, which is how SEO builds srcset without decoding a byte.

Every X_STORAGE_* code is in Error codes. Encryption at rest, lifecycle rules and storage classes are bucket configuration, not the framework's (axiom 7).

โš ๏ธ **GitHub.com Fallback** โš ๏ธ