Sending Records Back - SmithSoft-Studios/JAZMIN GitHub Wiki
People in the field capture records (visits, inspections, readings, receipts), often offline, and the owner files them into the shared file. Only the owner key can change a shared file, so each person sends their records back in a small file of their own, locked with their submission key.
sequenceDiagram
participant P as Field app (access key only)
participant S as Filing service (owner key)
participant F as shared.jzm
P->>F: open with access key
F-->>P: own rows + submission key
P->>P: write records.jzm locked with the submission key
P->>S: upload records.jzm + access key id
S->>F: accessKeyOf(id): does the grant still exist?
S->>S: open the upload with owner.submissionKey(id)
S->>F: append rows into the sender's own partition
-
Each access key has one. It's sealed in that key's slot of the shared file, so a person gets it only by opening
the shared file with their access key (plus an unlock token for an online grant):
reader.submissionKey. -
The owner derives the same key from the owner key and the access key's id:
owner.submissionKey(keyId). - What a submitted file proves: it was made by someone who opened the shared file with that access key. An access key that leaked, without the shared file, isn't enough to send anything.
- Safety: the submission key opens nothing in the shared file, and neither the owner key nor the access key can be worked out from it.
import { JazminKey, accessKeyOf, append, open, write } from '@smithsoft-studios/jazmin';
// The owner: a shared file of site visits, one partition per field worker
const owner = JazminKey.generate();
const bob = owner.createAccessKey();
const columns = [
{ name: 'id', type: 'string', nullable: false },
{ name: 'worker', type: 'string', nullable: false },
{ name: 'site', type: 'string' },
{ name: 'note', type: 'string' },
];
write('shared.jzm', [
{ id: 'v-1', worker: 'bob', site: 'Depot A', note: 'scheduled' },
{ id: 'v-2', worker: 'ann', site: 'Depot B', note: 'scheduled' },
], { columns, key: owner, access: { partitionBy: 'worker', grants: [{ key: bob, rows: ['bob'], label: 'Bob' }] } });
// Bob's phone: holds only Bob's access key. Opening the shared file gives the submission key.
const shared = open('shared.jzm', { key: bob.export() });
const submissionKey = shared.submissionKey;
shared.close();
const upload = write(null, [
{ id: 'v-1', worker: 'bob', site: 'Depot A', note: 'done, gate repaired' }, // a correction to a filed record
{ id: 'v-3', worker: 'ann', site: 'Depot C', note: 'new visit' }, // claims to be Ann's: filed as Bob's
], { columns, key: submissionKey });
// The filing service: holds the owner key; told the sender's key id with the upload
function fileUpload(bytes, keyId) {
accessKeyOf('shared.jzm', owner, keyId); // throws if the key has no grant (revoked, expired, unknown)
const sent = open(bytes, { key: owner.submissionKey(keyId) }); // throws if not made with this sender's key
const partition = 'bob'; // in a real service: the partition(s) of the sender's grant
const rows = [...sent.rows()].map((r) => ({ ...r, worker: partition })); // own rows only
sent.close();
return append('shared.jzm', { key: owner, upsert: rows, keyColumns: ['id'] });
}
const { inserted, updated } = fileUpload(upload, bob.id);
console.log(`filed: ${inserted} new, ${updated} updated`);
// A file not made with Bob's submission key is refused
try {
fileUpload(write(null, [{ id: 'v-9', worker: 'bob', site: 'X', note: 'forged' }], { columns, key: JazminKey.generate() }), bob.id);
} catch (e) {
console.log('forged upload refused:', e.message);
}
// Bob opens the shared file again and sees his filed records
for (const row of open('shared.jzm', { key: bob.export() }).rows()) console.log(row);filed: 1 new, 1 updated
forged upload refused: Decryption failed - wrong key/password or the data was tampered with
{
id: 'v-1',
worker: 'bob',
site: 'Depot A',
note: 'done, gate repaired'
}
{ id: 'v-3', worker: 'bob', site: 'Depot C', note: 'new visit' }
In .NET the same keys are reader.SubmissionKey, owner.SubmissionKey(keyId) and
JazminFile.AccessKeyOf(path, owner, keyId):
var key = JazminReader.Open("shared.jzm", new JazminReadOptions { AccessKey = bob }).SubmissionKey; // field app
JazminFile.AccessKeyOf("shared.jzm", owner, keyId); // filing service
using var sent = JazminReader.Open(upload, new JazminReadOptions { Key = owner.SubmissionKey(keyId) });- Own rows only: write each upload's rows into the sender's own partition, whatever the file says, so nobody can file rows as someone else.
-
Give every record an id. A record whose id is already filed replaces the filed one (
upsert), so corrections made on a phone reach the shared file. A file sent twice changes nothing.- Last arrival wins when two people change the same record.
- Who may change a record: anyone whose grant covers its partition. A change can't move a record to another partition.
-
Send only what changed: in the ready-made service, an empty value (
null, missing or'') leaves the field as it is, so a phone can't clear a field. -
Only what the sender can see: columns the sender's grant doesn't cover are ignored, in new records too. So are
columns the shared file doesn't have (reported as
ignoredColumns).
-
Revoked keys: once a key is revoked,
accessKeyOfno longer finds it, and its uploads are refused. -
Keys that expire: file an upload only if it was both written and received before the expiry. The phone's clock
(
reader.writtenAt) can be set to anything; your server's clock can't. File each upload as it arrives: the first write to the shared file after a key expires removes its grant. -
Older shared files written before submission keys existed have none (
submissionKeyisnull) until the owner's next rewrite orcompact().
A record can carry photos or PDFs. The phone embeds them in the file it sends and lists each record's files in a
json column, attachments:
await JazminBrowser.write([
{ id: 'v-17', note: 'site visit', attachments: ['v-17/receipt.pdf', 'v-17/photo.jpg'] },
], { columns, key: submissionKey, files: [
{ path: 'v-17/receipt.pdf', content: receiptFile },
{ path: 'v-17/photo.jpg', content: photoFile },
] });The ready-made filing service then checks that every listed file is in the batch and every file in the batch is
listed; checks each file's real type from its first bytes (PDF, JPEG, PNG and WebP by default, whatever name the phone
gives); limits sizes (10 MB per file, 50 MB per batch by default); stores each file once at
attachments/<key id>/<sha256>.<ext>, visible only to keys that see the record's partition; and rewrites the record's
attachments to [{ path, name, type, size }], so a reader opens a file with readFile(path).
fileSubmission() (.NET JazminFile.FileSubmission) applies every rule above in one call: one read of the shared
file and one append. It ships in 1.6.0 of both packages.
// not run: inside your filing service, which gives upload, owner and keyId
import { JazminSubmissionRejectedError, fileSubmission } from '@smithsoft-studios/jazmin';
try {
const result = fileSubmission('shared.jzm', upload, { key: owner, keyId, receivedAt: Date.now() });
// { sender, filed, updated, duplicates, files, ignoredColumns }
} catch (error) {
if (error instanceof JazminSubmissionRejectedError) { /* tell the sender error.message; keep the upload aside */ }
else throw error; // the shared file busy, a full disk: try again later
}
fileSubmission('shared.jzm', upload, { key: owner, keyId, dryRun: true }); // what would be filed, writing nothing// fragment
try
{
var result = JazminFile.FileSubmission("shared.jzm", upload, new JazminFileSubmissionOptions { Key = owner, KeyId = keyId, ReceivedAt = DateTimeOffset.UtcNow });
}
catch (JazminSubmissionRejectedException rejected) { /* tell the sender rejected.Message */ }Options: idColumn (id), onDuplicate (replace, or skip to keep the filed version), filesColumn
(attachments), fileTypes (PDF, JPEG, PNG, WebP), maxFileBytes (10 MB), maxBatchFileBytes (50 MB), dryRun.
js/examples/filing-service in the
repository applies all of these rules (from 1.6.0, by calling fileSubmission()):
-
fileBatch(sharedPath, ownerKey, { keyId, batch })files one upload and returns{ filed, updated, duplicates, files, ignoredColumns }, or throwsRejectedBatchwith a reason to show the user. -
inbox.mjsfiles every batch waiting in a folder (oldest first), moves them tofiled/orrejected/, and compacts the shared file after 50 appends. Run it on a schedule where the owner key is kept.
Speed: filing a batch takes about 20–25 ms whether it holds 1 record or 50, whether the shared file holds 10,000 records or 250,000, and with 1,000 people about as with 100.
Writing these files in a browser, including offline, is covered in Viewer and Browser.