Feature CTP Intake - Gadreel/divconq GitHub Wiki

CTP Intake

CTP (Common Transfer Protocol) Intakes are when a client uploads files to the dcFileServer.

Intake Concept

One or more files that constitute a contribution to the File Store. For a simple file server the Intake is just an upload followed by acceptance of evidence. For more advanced cases the acceptance of evidence is followed by Intake Processing. Intake Processing is minimal processing required before the upload(s) are considered complete. Advanced or complex operations should be done in a further step by the server, after acknowledging the files and intact and complete.

CTP Intake Handling

Before doing an upload request a IntakeId. Upload all files to that IntakeId. Conclude the Intake. Additional Intake processing may occur, after which the Intake is considered complete. When an IntakeId is used the files uploaded may be stored in a temporary file for processing before being moved to the final destination visible to the user.

Legacy Intake Handling

Any Intake can be handled using the legacy approach, even those over HTTP. The Legacy approach is most valuable for compatibility with SFTP, FTPS, WebDAV or other such protocols as a substitute for CTP.

Trigger and Sweep Approach

When a trigger file appears a Intake Process is attached to the folder with the folder and trigger as arguments. The Path or Token from the Upload request may be present in the trigger file.

CTP Uploads

Upload Request

Contents:

• Path (optional if Token is present) where the file should reside in the file store presented to this user
• Token (optional if Path is present) indirectly represent the file path
• IntakeId (optional) the Intake this file is a part of
• IntakePath (optional) the (additional) intake processing to do on this file

TODO review the rest of this page

Evidence

Evidence may be

• Size where Value equals size in bytes
• Hash (MD5, SHA1, etc) where value is the hashed value
• PGPSigned (for self signed file) where value is the PGP public key name or id (their key)

Intake Monitoring

Poll the IntakeId until Intake is accepted or rejected.

Intake Processing (optional)

During an upload the server accepts the file content from the client and then accepts evidence that the upload is intact. This can be the end of it, the file may be stored and ready for use or archiving. However, the server may also do a final step of processing called an Intake - after evidence is presented and before the file is considered to be truly ready.

By default no Intake Path is defined in dcFileServer. You may configured a default Intake Path, or simply make Intake Paths available but optional. When an IntakePath is used then the uploaded file is stored in a local temp folder until evidence is given. Once acceptable evidence is provided then a Intake Script is run as defined for the Path. That script will be provided with source and intended destination values.

Good for:

• Bulk Transfer (see below)
• Virus check
• Integrity check (PGPSigned files must stripped of signature before
• File validation
• TODO more