Feature dcBus - Gadreel/divconq GitHub Wiki

Intro to dcBus

TODO add intro, this whole page applies to v0.9.5 on, not v0.9.3 or v0.9.4.

Connection Management

DivConq File Servers connect to DivConq File Gateways so that Gateways, often placed in a DMZ, do not have to have firewall ports open back to the File Server. So the File Servers initiate the connection but once connected the communication channel operates full duplex so either party may freely send messages to the other.

A server configuration may look like:

<Bus>
	<Trust Thumbprint="F7:15:BB:A0:68:01:B3:4A:C0:ED:19:58:26:77:1D:E8:98:13:27:AF" />
	<SslContext Password="A1s2d3f4" File="./packages/dcTest/keys/backend.jks" /> 
	<Connector Address="192.168.0.15" HubId="00110" Port="7443" Ssl="True" Number="100" />
</Bus>

Trust only trust certificates (on gateway) presenting this thumbprint. Other options coming (TODO).

SslContext where to load my ssl certificate from. dcBus requires that both parties present an SSL Certificate, in this case we are a "client" because we initiate the connection to the gateway server. So we are loading and presenting the certificate in "client mode".

Connector port and address to connect to, the id of the hub we expect to connect to (verified on connection). SLL enabled, default of course. And finally the Number of connections to make, defaults to 25.

A client configuration may look like:

<Bus>
	<Trust Thumbprint="F7:15:BB:A0:68:01:B3:4A:C0:ED:19:58:26:77:1D:E8:98:13:27:AF" />
	<SslContext Password="A1s2d3f4" File="./packages/dcTest/keys/backend.jks" /> 
	<Listener Port="7443" Ssl="True" />
</Bus>

Trust and SslContext are the same as the backend server config. In this case we even use the same certificate file, though I advice against this in production.

Since the gateway server is technically the "server" end of the dcBus connection it setsup a Listener and passively awaits a connection from the backend server. A few of the settings are optional. To default to port 7443, secure, default connection number and no hub check use:

<Connector Address="192.168.0.15" />

And:

<Listener />

Connection Pool

dcBus

CTP protocol
CTP-B protocol and Bus Pool
CTP Dedicated Pool

When dealing with file transfer (large file chunks) then "streaming" is an unavoidable topic due to network performance. If you want speed then you need to "stream". In this context means sending the file over a TCP/IP or UDT connection without sending confirmation messages - or any unrelated messages - over the connection during the streaming. TCP/IP and UDT have built-in packet confirmation schemes and TLS has packet verification schemes, so we just have to trust those mechanisms and pump the file through the connection as fast as the channel can write.

This is why there is a Dedicated Pool, because for streaming to work the connection must be dedicated to a specific task such as a file transfer. Although we have this pool of dcBus (CTP-B) connections - say 3 - that can be used by any thread on either end of the connection to send a message, we cannot just stream over that connection while also sending general messages.

dcClient API (CTP)

A CTP client may connect via HTTP/2 (or WS or HTTP/1.1), or via direct protocol using many of the same messages as dcBus does (above). Only issue is that all Session control messages must go through HTTP/2. A client must login and start a session via HTTP/2 before they are allowed to connect on direct.

Messages supported via direct protocol (which is effectively a hotline straight back to the backend, to a specific service).

** terminating the direct protocol is done either by closing connection or via session control (HTTP/2)

STREM 4 - a stream message (dedicated mode)
CONTL 5 - control messages
CHELO 6 - HELLO used to acknowledge the direct pipe is ready (for client) and used for keep alive

Keep in mind none of these messages are seen by the Gateway, they just pass straight to the Backend server. The Gateway releases the dedicated stream only when:

The session times out
The pipe is inactive too long
Client closes its end of the pipe
Backend closes its end of the pipe (backend may ask gateway to kill session).

One, two or three allows the dcBus to be used again.

TODO cleanup above

TODO Dedicated may be TCP or UDT

CTP Commands

Command Codes are 2 bytes

CTP has a few Sub-Protocols

Base
Bus
Client Startup
Message Xchange

Base Commands

Common to All Sub-Protocols

CTP_CMD_EXIT

command code: two bytes

w/o sign out, but the session will timeout soon if no connection is present

CTP_CMD_EXIT_SIGN_OUT

command code: two bytes

CTP_CMD_ALIVE

command code: two bytes

CTP_CMD_ENGAGE

command code: two bytes
dest code: one byte
- 0 = join message bus - CTP_B
- 1 = join dedicated operation
claim id length: one byte
- 200 max length supported
claim id: N bytes

Used only with Bus connections, tell the idle connection to engage in some duty. Respond with CTP_CMD_ALIVE, because with BUS it is the back end that sends Alive and sends Engage - so CTP_CMD_ALIVE from gateway is meaningfully different than normal operations.

Bus Commands

Bus defaults to compact JSON mode for message bodies.

CTP_B_CMD_STATUS

Routinely sent by both sides of the connection in dcBus. Used in place of CTP_CMD_ALIVE for bus.

command code: two bytes
JSON-like structure: unknown bytes

CTP_B_CMD_SEND

command code: two bytes
JSON-like structure: unknown bytes

CTP_B_CMD_SEND_FORGET

command code: two bytes
JSON-like structure: unknown bytes

Startup Commands

External client connections default into Startup Protocol. Startup defaults to full JSON mode for message bodies.

CTP_S_CMD_ENGAGE

command code: two bytes
JSON-like structure: unknown bytes

Present Schema

<Record Id="CtpSEngageMessage">
	<Field Name="ClientVersion" Type="dcTinyString" />
	<Field Name="ClientName" Type="dcTinyString" />
	<Field Name="Credentials" Type="UserCredentials" />
	<Field Name="Service" Type="dcSmallString" Required="True" />
	<Field Name="Protocol" Type="dcTinyString" Required="True" />
	<Field Name="Params" Type="Any" />
</Record>

<Record Id="UserCredentials">
	<Field Name="Username" Type="dcUser:dcUsername" />
	<Field Name="Password" Type="dcUser:dcPassword" />
	<Field Name="ConfirmationCode" Type="dcUser:dcConfirmCode" />
	<Field Name="AccessCode" Type="dcTinyString" />
	<Field Name="SessionId" Type="dcTinyString" />
</Record>

Username + Password for login
Username + ConfirmationCode for login with password recovery
AccessCode for auth via a generic code system, not enabled by default
SessionId + AccessCode for auth via existing session

Response is CTP_S_CMD_RESPONSE.

CTP_S_CMD_RESPONSE

command code: two bytes
JSON-like structure: unknown bytes

Response Schema

<Record Id="CtpResponseMessage">
	<Field Name="Messages">
		<List Type="ResultMessage" />
	</Field>
	<Field Name="Body" Type="Any" />
</Record>

Any Error message = Failure
No Error message = Success

Message Xchange Commands

May issue INIT and multiple SETTING before START - once START in SEND_* mode only.

CTP_M_CMD_INIT

command code: two bytes
JSON-like structure: unknown bytes

Response is CTP_M_CMD_RESPONSE_*

CTP_M_CMD_SETTING

command code: two bytes
JSON-like structure: unknown bytes

Response is CTP_M_CMD_RESPONSE_*

CTP_M_CMD_START

command code: two bytes

Indicates that we are now in full duplex mode, server or client may send messages. Must have successfully called INIT first. Wait for response before SENDing. Response is CTP_M_CMD_RESPONSE_*.

CTP_M_CMD_SEND

command code: two bytes
JSON-like structure: unknown bytes

No immediate message response excepted, response handled via message tag.

CTP_M_CMD_SEND_FORGET

command code: two bytes
JSON-like structure: unknown bytes

No message response at all.

CTP_M_CMD_RESPONSE

command code: two bytes
JSON-like structure: unknown bytes

Response Schema

<Record Id="CtpResponseMessage">
	<Field Name="Messages">
		<List Type="ResultMessage" />
	</Field>
	<Field Name="Body" Type="Any" />
</Record>

A non-zero Result code = Failure
A zero Result code = Success

File Transfer Commands

CTP_F_CMD_INIT

command code: two bytes
JSON-like structure: unknown bytes

Response is CTP_F_CMD_RESPONSE_*. Response JSON contains things like:

CTP_F Versions supported
Want Progress Flag (server wants progress for writes)
List of "public" Batch Plans
List of Predefined Select Plans
List of Extra Attributes (e.g. MIME or HASH) readily available (no extra work on server)
- see CTP_F_SELECT_PREFERED, this is flag to use for these. Preferred = Basic plus Extra attributes
Pipes In List and Pipes Out List
- Pipe Name
- Evidence Method BULK: basic/[strict]/no
- Evidence Method MANUAL: yes/[no]
- Preferred Evidence Level: N [SHA1]
- Minimal Evidence Level: N [SIZE]

CTP_F_CMD_SETTING

command code: two bytes
JSON-like structure: unknown bytes

Set server settings. Response is CTP_F_CMD_RESPONSE_*.

progress during read to on or off
attributes to return in READ, defaults to CTP_F_SELECT_PREFERED
send log error messages untranslated when possible: yes/no

Stream Notes

By convention a single batch or stream operation should not pass more than 10,000,000 files through it. However, it is recommended to pass no more than 10,000 files in a single batch or stream.

CTP_F_CMD_STREAM_READ

Command Code: two bytes
Evidence Flag: one byte
- 0 = Use BASIC
- 1 = Use STRICT
- 2 = Use MANUAL
- 3 = Use BASIC and MANUAL
Out Pipe Name length: two bytes
- 32,000 max length supported
- length 0 = "default" pipe
Out Pipe Name: up to 32000 bytes

After sending this command enter Stream Read mode. Response is CTP_F_CMD_STREAM_*, typically BLOCK.

CTP_F_CMD_STREAM_WRITE

Command Code: two bytes
Evidence Flag: one byte
- 0 = Use BASIC
- 1 = Use STRICT
- 2 = Use MANUAL
- 3 = Use BASIC and MANUAL
In Pipe Name length: two bytes
- 32,000 max length supported
- length 0 = "default" pipe
In Pipe Name: up to 32000 bytes

Auto creates folders as needed. Response is CTP_F_CMD_RESPONSE_*. Begin writing after SUCCESS

CTP_F_CMD_STREAM_BLOCK

BASE COMMAND

Command Code: two bytes
Block Flag: one byte
- see CTP_F_BLOCK_TYPE_*

HEADER

If HEADER is in Block Flag, then look for fields below, else skip to CONTENT.

repeat following until attribute 0 (CTP_F_ATTR_END) found
- attribute type: 2 bytes (see CTP_F_ATTR_*)
- value length: two bytes
  - 32,000 max length supported
- value: N bytes (treat as UTF8 characters)

CONTENT

If CONTENT is in Block Flag, then look for fields below, else skip to EOF.

Payload Offset - 8 bytes
Payload Size - 3+ bytes
- binary
- never send more than 16MB in a single message [at least 32KB or 48KB to reduce impact on memory/cpu]

If EOF is in Block Flag, then done with that file.

CTP_F_CMD_STREAM_PROGRESS

Command Code: two bytes
Total Files Code: three bytes
- 10,000,001 = unknown
- <= 10,000,000 = literal
Current File Code: three bytes
- 10,000,001 = maxed
- <= 10,000,000 = correct
Amount Complete: 1 byte

No response

CTP_F_CMD_STREAM_FINAL

Command Code: two bytes

Response is CTP_F_CMD_RESPONSE_* no matter who sent FINAL.

CTP_F_CMD_STREAM_ABORT

Command Code: two bytes

If ABORT is sent by the recipient of BLOCKs then other side sets a flag and responds with ABORT, no immediate response. If ABORT is sent by the sender of BLOCKs then response is CTP_F_CMD_RESPONSE_* and STREAM is complete.

CTP_F_CMD_SEND_WAIT

command code: two bytes
JSON-like structure: unknown bytes, parse till end

Expect CTP_F_CMD_RESPONSE_* with JSON body for response details.

CTP_F_CMD_SEND_FORGET

command code: two bytes
JSON-like structure: unknown bytes, parse till end

Expect immediate CTP_F_CMD_RESPONSE_* indicating send or rejected.

CTP_F_CMD_OP_SELECT

command code: two bytes
JSON-like structure: unknown bytes, parse till end

Expect CTP_F_CMD_RESPONSE_* response.

Servers may elect to support Basic SELECT or Advanced SELECT.

Request example format for Basic SELECT:

{
	Select: {
		Mode: "[Detail]|Listing|Expanded",
		RelativeTo: "the base path for all Files Sets, defaults to /",
		// Select contains either File (fields below) or File Sets, not both - File checked first
		Path: "path relative to RelativeTo",
		Rename: "optional, new name, used during MOVE, name only no path",
		Offset: "optional, bytes from start, used during READ for resume download",
		Attributes: [
			N, N, N
		]
	}
}

Request example format for Advanced SELECT:

{
	Select: {
		Mode: "[Detail]|Listing|Expanded",
		RelativeTo: "the base path for all Files Sets, defaults to /",
		// Select contains either File (fields below) or File Sets, not both - File checked first
		Path: "path relative to RelativeTo",
		Recursion: N [1] - where 0 means none, 1 means 1 level, etc,     
		Rename: "optional, new name, used during MOVE, name only no path",
		Offset: "optional, bytes from start, used during READ for resume download",
		// if no file sets listed then match every thing under RelativeTo
		FileSets: [		// each file set is independent and cumulative - list within a list
			[
				{
					Type: "File",			// select specific files/folders
					Path: "path relative to RelativeTo",
					Recursion: N [1] - where 0 means none, 1 means 1 level, etc,     
					Rename: "optional, new name, used during MOVE, name only no path",
					Offset: "optional, bytes from start, used during READ for resume download",
					Not: t/[f]
				},
				{
					Type: "NameFilter",
					Pattern: "reg ex for name",
					Not: t/[f]
				},
				{
					Type: "PathFilter",
					Pattern: "reg ex for path relative to RelativeTo",
					Not: t/[f]
				},
				{
					Type: "SizeFilter",							// conditions may be combined
					Equal: N, 
					LessThan: N, 
					GreaterThan: N, 
					LessThanOrEqual: N, 
					GreaterThanOrEqual: N,
					Not: t/[f]
				},
				{
					Type: "ModifiedFilter",						// if time is left off then matches on DATE only
					Equal: "ISO UTC DATE/TIME", 
					LessThan: "ISO UTC DATE", 
					GreaterThan: "ISO UTC DATE", 
					LessThanOrEqual: "ISO UTC DATE", 
					GreaterThanOrEqual: "ISO UTC DATE",
					Not: t/[f]
				}
			]
		],
		Sort: {
			// Match gets value from first RegEx pattern in filter
			Type: "[Name]|Path|Modified|Size|Match|Value",		// if any sort is defined but not Type then Name is default
			Direction: "Asc|Desc",
			SortAs: "Number|[String]",							// used with Match
			Value: "mix in %attrib% with text"					// used with Value
		},
		Attributes: [
			N, N, N
		]
	}
}

In Basic SELECT Recursion defaults to 0 for Detail mode, 1 for Listing mode and 999 for Download and Expanded mode. This means Detail mode only gets details about the 1 selected file. Listing only gets the files listed immediately under the selected file (folder) and Downloading gets all the files under the Folder if selected file is a folder.

In Advanced SELECT you can override the Recursion setting, but it still defaults the same as Basic.

Folder Listing Request example format (Basic or Advanced):

Assumes path points to a folder, file details returned for files directly under folder.

{
	Select: {
		Mode: "Listing",
		Path: "/",
		Attributes: [
			10
		]
	}
}

File Details Request example format (Basic or Advanced):

Assumes path points to a folder or file, only 1 item returned.

{
	Select: {
		Mode: "Detail",
		Path: "/",
		Attributes: [
			10
		]
	}
}

File Download Request example format (Basic or Advanced):

Assumes path points to a file, download only that one file.

{
	Select: {
		Mode: "Download",
		Path: "/charity/hello.txt",
		Attributes: [
			10, 20
		]
	}
}

Folder Download Request example format (Basic or Advanced):

Assumes path points to a folder. Download all files in it, and in sub folders, recursively until 999 levels deep (999 = convention for all).

{
	Select: {
		Mode: "Download",
		Path: "/charity",
		Attributes: [
			10, 20
		]
	}
}

Folder Download Request example format (Advanced):

Assumes path points to a folder. Download all files in that folder, but not lower.

{
	Select: {
		Mode: "Download",
		Path: "/charity",
		Recursion: 1,
		Attributes: [
			10, 20
		]
	}
}

CTP_F_CMD_OP_DELETE

command code: two bytes

Use the last SELECT and apply delete. Expect CTP_F_CMD_RESPONSE_*, including PROGRESS if enabled.

CTP_F_CMD_OP_MOVE

command code: two bytes
dest relative to path length: two bytes
- 32,000 max length supported
- length 0 = this is a Rename operation only, no actual moves
dest relative to path: N bytes

Use the last SELECT (which supplies a src relative path) and apply move. Select supplies new names if renaming is to occur. Expect CTP_F_CMD_RESPONSE_*, including PROGRESS if enabled.

CTP_F_CMD_OP_UPDATE

command code: two bytes
repeat following until attribute 0 (CTP_F_ATTR_END) found
- attribute type: 2 bytes (see CTP_F_ATTR_*, especially CTP_F_ATTR_MODTIME)
- value length: two bytes
  - 32,000 max length supported
- value: N bytes (treat as UTF8 characters)

Use the last SELECT and apply attribute update. Expect CTP_F_CMD_RESPONSE_*, including PROGRESS if enabled.

CTP_F_CMD_BATCH_UPLOAD_START

command code: two bytes
name length: two bytes
- 32,000 max length supported
- length 0 = "default" name
name: N bytes
JSON-like structure: unknown bytes

Resets Evidence collection. Expect CTP_F_CMD_RESPONSE_* indicating send or rejected. If SUCCESS then body contains a unique contrib id.

CTP_F_CMD_BATCH_UPLOAD_RESUME

command code: two bytes
contrib id length: two bytes
- 32,000 max length supported
contrib id: N bytes
JSON-like structure: unknown bytes

Resets Evidence collection. Expect CTP_F_CMD_RESPONSE_* indicating send or rejected.

CTP_F_CMD_BATCH_UPLOAD_FINISH

command code: two bytes
Type Code: one byte
- see CTP_F_EVIDENCE_*
Total Files Code: three bytes
- 10,000,001 = unknown
- <= 10,000,000 = literal
JSON-like structure: unknown bytes

Upload mode reminds server how many files were passed. Server reviews and completes Evidence collection. Expect CTP_F_CMD_RESPONSE_* indicating send or rejected. Expect PROGRESS messages.

CTP_F_CMD_BATCH_UPLOAD_STATUS

command code: two bytes
contrib id length: two bytes
- 32,000 max length supported
contrib id: N bytes
JSON-like structure: unknown bytes

Expect CTP_F_CMD_RESPONSE_* indicating send or rejected.

CTP_F_CMD_BATCH_UPLOAD_SAVE

command code: two bytes

Expect CTP_F_CMD_RESPONSE_* response.

CTP_F_CMD_BATCH_UPLOAD_ABORT

command code: two bytes

Expect CTP_F_CMD_RESPONSE_* response.

CTP_F_CMD_BATCH_DOWNLOAD_START

command code: two bytes
name length: two bytes
- 32,000 max length supported
- length 0 = "default" name
name: N bytes
JSON-like structure: unknown bytes

Resets Evidence collection. Expect CTP_F_CMD_RESPONSE_* indicating send or rejected. If SUCCESS then body contains a unique obtain id.

CTP_F_CMD_BATCH_DOWNLOAD_RESUME

command code: two bytes
obtain id length: two bytes
- 32,000 max length supported
obtain id: N bytes
JSON-like structure: unknown bytes

Resets Evidence collection. Expect CTP_F_CMD_RESPONSE_* indicating send or rejected.

CTP_F_CMD_BATCH_DOWNLOAD_FINISH

command code: two bytes
Type Code: one byte
- see CTP_F_EVIDENCE_*
Successful Files Code: three bytes
- 10,000,001 = unknown
- <= 10,000,000 = literal
Failed Files Code: three bytes
- 10,000,001 = unknown
- <= 10,000,000 = literal
JSON-like structure: unknown bytes

Server reviews and completes Evidence collection. Expect CTP_F_CMD_RESPONSE_* indicating send or rejected. Expect PROGRESS messages.

CTP_F_CMD_BATCH_DOWNLOAD_STATUS

command code: two bytes
obtain id length: two bytes
- 32,000 max length supported
obtain id: N bytes
JSON-like structure: unknown bytes

Expect CTP_F_CMD_RESPONSE_* indicating send or rejected.

CTP_F_CMD_BATCH_DOWNLOAD_SAVE

command code: two bytes

Expect CTP_F_CMD_RESPONSE_* response.

CTP_F_CMD_BATCH_DOWNLOAD_ABORT

command code: two bytes

Expect CTP_F_CMD_RESPONSE_* response.

Evidence Notes

BASIC evidence - transport mode ensures that:

packets are in the correct order TCP/IP
packet integrity TLS (or SSH Style or QUIC style)
block offsets for each body block
footer block indicates the end
try to compare size if size is in header/footer

STRICT evidence - in addition to BASIC, look at hash in the file header/footer and compare that with written file.

MANUAL evidence - don't run evidence at all during transport, expect a manual evidence later.

CTP_F_CMD_EVIDENCE_MANUAL

command code: two bytes
type code: one byte
- see CTP_F_EVIDENCE_*
paths enclosed: one byte
- 250 max supported at once
repeat following until all enclosed processed
- path length: two bytes
  - 32,000 max length supported
- path name: N bytes
- value length: two bytes
  - 32,000 max length supported
- value: N bytes (treat as UTF8 characters)

Expect CTP_F_CMD_RESPONSE_* response. Resets evidence counter/collection.

CTP_F_CMD_RESPONSE

command code: two bytes
JSON-like structure: unknown bytes

Response Schema

<Record Id="CtpResponseMessage">
	<Field Name="Messages">
		<List Type="ResultMessage" />
	</Field>
	<Field Name="Body" Type="Any" />
</Record>

A non-zero Result code = Failure
A zero Result code = Success

Message should be basic - FAILURE, DENIED or UNSUPPORTED approach

No response (to Response) expected.

CTP_F_CMD_PROGRESS

command code: two bytes
progress percent: one byte

No response expected.

TODO

Get the following sorted out...raw notes for now.

Visualize Diagrams with: http://bramp.github.io/js-sequence-diagrams/

     * 
     * Client Message -> raw bytes (with hashing) -> server -> session channel (gateway) -> bus (dedicated conn)
     * 		-> session channel (server) -> raw bytes (with hashing) -> 
     *

CTP_F Rules

Client Regular Mode

Applies to all commands except for Streams and Responses

client and server always in read mode
client or server may accept EXIT or ALIVE at any time - EXIT no reply, just stop, ALIVE no reply and return to read mode
technically client is responsible for ALIVE, but server may issue if it is curious
client may issue non-stream commands (SEND_, EVIDENCE_, OP_, BATCH_), assumes the next read (except for EXIT or ALIVE) is a response to that command
- response may be any RESPONSE_* - typically SUCCESS or PROGRESS
- keep reading until you get a non-PROGRESS type of RESPONSE_*
server may not issue any commands beyond EXIT or ALIVE (except in stream mode)

Title: Regular Mode Commands
participant Client
participant Server
Note right of Server: in read mode\nmay issue\nEXIT or ALIVE
Note left of Client: in read mode\nmay issue \nEXIT or ALIVE or\nF_CMD_*
Client->Server: F_CMD_ except F_CMD_STREAM\nor F_CMD_RESPONSE
Note left of Client: flush write\nno wait on write\nwait for RESPONSE
Note right of Server: take as long as\nneeded (async)\nsend progress\nsend response\nflush writes\nno wait on write\nwait on next cmd
Server->Client: F_CMD_RESPONSE_* 
Note left of Client: keep reading until\nnot-PROGRESS and\nnot EXIT\nafter response\ndo next cmd

Client Read Mode

With STREAM_READ Stream Command

client or server may accept EXIT or ALIVE or STREAM_ABORT at any time
- if client issues ABORT then it must wait for FINAL or ABORT from server
- ABORT from server returns everyone to regular mode
server always in read mode - looking mostly for STREAM_ABORT or EXIT (client need not issue ALIVE during read)
server writes STREAM_BLOCK and STREAM_PROGRESS
- client reads block but does not read again until a block has been dealt with
- client reads progress and updates transfer's progress, reads again immediately
server writes STREAM_FINAL or STREAM_ABORT to end
- client send SUCCESS without looking for response
- return to regular mode

Title: Read Mode Commands
participant Client
participant Server
Note right of Server: in read mode\nmay issue\nEXIT or ALIVE
Note left of Client: in read mode\nmay issue \nEXIT or ALIVE\nor F_CMD_*
Client->Server: F_CMD_STREAM_READ
Note right of Server: in read/write mode\nmay issue\nEXIT or ALIVE\nor STREAM_*
Note left of Client: in read mode\nmay issue \nEXIT or ALIVE\nor STREAM_ABORT
Server->Client: F_CMD_STREAM_BLOCK,\nor STREAM_PROGRESS 
Note right of Server: buffer till full\nflush buffer write\nwait on write\nthen send more
Note right of Server: logic to decide\nwhen to interleave\nprogress
Note right of Server: check abort flag\nbefore next write
Note left of Client: take as long as\nneeded (async)\nissue read after\nblock is handled
Client-->Server: F_CMD_STREAM_ABORT\noption
Server->Client: F_CMD_STREAM_BLOCKS\nand progress
Server->Client: F_CMD_STREAM_FINAL\nor STREAM_ABORT 
Note right of Server: flush buffer write\nreturn to read\nmode
Client->Server: F_CMD_RESPONSE_*\ntypically success
Note right of Server: in read mode\nmay issue\nEXIT or ALIVE
Note left of Client: in read mode\nmay issue \nEXIT or ALIVE\nor F_CMD_*

Client Write Mode

With STREAM_WRITE Stream Command

client or server may accept EXIT or ALIVE or STREAM_ABORT at any time
- if server issues ABORT then it must wait for FINAL or ABORT from client
- ABORT from client returns everyone to regular mode
client always in read mode - looking mostly for STREAM_ABORT or EXIT
client writes STREAM_BLOCK and STREAM_PROGRESS
- server reads block but does not read again until a block has been dealt with
- server reads progress and updates transfer's progress, reads again immediately
client writes STREAM_FINAL or STREAM_ABORT to end and waits for SUCCESS
- server sends SUCCESS
- return to regular mode

Title: Write Mode Commands
participant Client
participant Server
Note right of Server: in read mode\nmay issue\nEXIT or ALIVE
Note left of Client: in read mode\nmay issue \nEXIT or ALIVE\nor F_CMD_*
Client->Server: F_CMD_STREAM_WRITE
Server->Client: F_CMD_RESPONSE_*\ntypically success
Note right of Server: in read mode\nmay issue\nEXIT or ALIVE\nor STREAM_ABORT
Note left of Client: in read/write mode\nmay issue \nEXIT or ALIVE\nor STREAM_*
Client->Server: F_CMD_STREAM_BLOCK,\nor STREAM_PROGRESS 
Note left of Client: buffer till full\nflush buffer write\nwait on write\nthen send more
Note left of Client: logic to decide\nwhen to interleave\nprogress
Note left of Client: check abort flag\nbefore next write
Note right of Server: take as long as\nneeded (async)\nissue read after\nblock is handled
Server-->Client: F_CMD_STREAM_ABORT\noption
Client->Server: F_CMD_STREAM_BLOCKS\nand progress
Client->Server: F_CMD_STREAM_FINAL\nor STREAM_ABORT 
Note left of Client: flush buffer write\nreturn to read\nmode
Server->Client: F_CMD_RESPONSE_*\ntypically success
Note right of Server: in read mode\nmay issue\nEXIT or ALIVE
Note left of Client: in read mode\nmay issue \nEXIT or ALIVE\nor F_CMD_*

CTP-F Example Usage

Title: Init
participant Client
participant Server
Client->Server: INIT
Server->Client: SUCCESS w/ JSON

Title: Basic Folder Listing
Client->Server: SELECT w/ Filters
Server->Client: SUCCESS 
Client->Server: READ w/ Progress On
Server->Client: BLOCK*
Server->Client: PROGRESS*
Note left of Server: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Server->Client: FINAL
Client->Server: SUCCESS

SUCCESS cmd means the stream completed correctly and we return to req/resp model

Title: File Detail
Client->Server: SELECT w/ Path\nand Attributes
Server->Client: SUCCESS 
Client->Server: READ 
Server->Client: BLOCK
Server->Client: FINAL
Client->Server: SUCCESS

Evidence is not tracked on server for basic downloads.

Title: Basic Download
Client->Server: SELECT w/ File
Server->Client: SUCCESS 
Client->Server: READ w/ Progress On
Server->Client: BLOCK*
Server->Client: PROGRESS*
Note left of Server: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Server->Client: FINAL
Client->Server: SUCCESS

Can manually for evidence on server, even with basic downloads.

Title: Basic Download w/ Evidence
Client->Server: SELECT w/ File
Server->Client: SUCCESS 
Client->Server: READ w/ Progress On
Server->Client: BLOCK*
Server->Client: PROGRESS*
Note left of Server: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Server->Client: FINAL
Client->Server: SUCCESS 
Client->Server: EVIDENCE_MANUAL*
Server->Client: RESPONSE*
Note right of Client: 0 or more Evidence\nnmessages, 1 for each\nfile transfered

Title: Basic Download Resume
Client->Server: SELECT w/ File\nand Offset
Server->Client: SUCCESS 
Client->Server: READ w/ Progress On
Server->Client: BLOCK*
Server->Client: PROGRESS*
Note left of Server: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Server->Client: FINAL
Client->Server: SUCCESS

Batch downloads do track Evidence Automatically

Title: Batch Download 
Client->Server: BATCH_DOWNLOAD_START
Server->Client: SUCCESS 
Client->Server: SELECT w/ File
Server->Client: SUCCESS 
Client->Server: READ w/ Progress On
Server->Client: BLOCK*
Server->Client: PROGRESS*
Note left of Server: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Server->Client: FINAL
Client->Server: SUCCESS 
Client->Server: BATCH_DOWNLOAD_FINISH
Server->Client: RESPONSE_*
Note right of Client: 1 FINISH (evidence)\nblock for all\nfiles transferred\nsince last START

Client may issue EVIDENCE_MANUAL before FINISH to:

clarify evidence
complete evidence if MANUAL REQUIRED
evidence is still not applied until FINISH

Title: Batch Download Resume
Client->Server: BATCH_DOWNLOAD_START
Server->Client: SUCCESS 
Client->Server: SELECT w/ File\nand Offset
Server->Client: SUCCESS 
Client->Server: READ w/ Progress On
Server->Client: BLOCK*
Server->Client: PROGRESS*
Note left of Server: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Server->Client: FINAL
Client->Server: SUCCESS 
Client->Server: BATCH_DOWNLOAD_FINISH
Server->Client: RESPONSE_*

Title: Client READ Abort
Client->Server: READ w/ Progress On
Server->Client: BLOCK*
Server->Client: PROGRESS*
Note left of Server: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Client->Server: ABORT
Server->Client: ABORT
Client->Server: SUCCESS

after ABORT the SUCCESS cmd means the stream completed correctly and we return to req/resp model, not that the download was a success

Title: Server READ Failure
Client->Server: READ w/ Progress On
Server->Client: BLOCK*
Server->Client: PROGRESS*
Note left of Server: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Server->Client: FAILURE or DENIED\nor ABORT
Client->Server: SUCCESS

Title: Download w/ Pipe
Client->Server: SELECT w/ Path\nand Attributes
Server->Client: SUCCESS 
Client->Server: READ w/ Progress On,\nPipe name
Server->Client: BLOCK*
Server->Client: PROGRESS*
Note left of Server: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Server->Client: FINAL
Client->Server: SUCCESS

Title: Download STRICT Evidence
participant Client
participant Server
Server->Client: BLOCK [file header]
Server->Client: BLOCK [file body]*
Note left of Server: 0 or more body blocks 
Server->Client: BLOCK [file footer]
Note left of Server: footer, like header,\nbut contains hash\nclient now does hash

Basic Upload does not track evidence

Title: Basic Upload
Client->Server: WRITE w/ Progress On
Server->Client: SUCCESS
Client->Server: BLOCK*
Client->Server: PROGRESS*
Note right of Client: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Client->Server: FINAL
Note right of Client: Client keeps receiving\nmessages until SUCCESS\nthen return to req/resp
Server->Client: SUCCESS

Basic Upload does not track evidence, but evidence handler can be invoked manually

Title: Basic Upload w/ Evidence
Client->Server: WRITE w/ Progress On
Server->Client: SUCCESS
Client->Server: BLOCK*
Client->Server: PROGRESS*
Note right of Client: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Client->Server: FINAL
Note right of Client: Client keeps receiving\nmessages until SUCCESS\nthen return to req/resp
Server->Client: SUCCESS
Client->Server: EVIDENCE_MANUAL*
Server->Client: RESPONSE*
Note right of Client: 0 or more Evidence\nnmessages

Title: Basic Upload Resume
Client->Server: WRITE w/ Progress On
Server->Client: SUCCESS
Client->Server: BLOCK*
Note right of Client: block contains offset for\nresume position
Client->Server: PROGRESS*
Note right of Client: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Client->Server: FINAL
Note right of Client: Client keeps receiving\nmessages until SUCCESS\nthen return to req/resp
Server->Client: SUCCESS

Batch Uploads do track evidence

Title: Batch Upload 
Client->Server: BATCH_UPLOAD_START
Server->Client: SUCCESS
Client->Server: WRITE w/ Progress On\nand with pipe name
Server->Client: SUCCESS
Client->Server: BLOCK*
Client->Server: PROGRESS*
Note right of Client: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Client->Server: FINAL
Server->Client: SUCCESS
Client->Server: BATCH_UPLOAD_FINISH
Server->Client: RESPONSE_*
Note right of Client: 1 FINISH (evidence)\nblock for all\nfiles transferred\nsince last START

Client may issue EVIDENCE_MANUAL before FINISH to:

clarify evidence
complete evidence if MANUAL REQUIRED
evidence is still not applied until FINISH

Title: Batch Upload Resume
Client->Server: BATCH_UPLOAD_START
Server->Client: SUCCESS
Client->Server: WRITE w/ Progress On
Server->Client: SUCCESS
Client->Server: BLOCK*
Note right of Client: block contains offset for\nresume position
Client->Server: PROGRESS*
Note right of Client: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Client->Server: FINAL
Note right of Client: Client keeps receiving\nmessages until SUCCESS\nthen return to req/resp
Server->Client: SUCCESS
Client->Server: BATCH_UPLOAD_FINISH
Server->Client: RESPONSE_*

Pipes may be used by Basic or Batch

Title: Upload w/ Pipe
Client->Server: WRITE w/ Progress On\nand with pipe name
Server->Client: SUCCESS
Client->Server: BLOCK*
Client->Server: PROGRESS*
Note right of Client: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Client->Server: FINAL
Server->Client: SUCCESS

Title: Client WRITE Abort
Client->Server: WRITE 
Server->Client: SUCCESS
Client->Server: BLOCK*
Client->Server: PROGRESS*
Note right of Client: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Client->Server: ABORT
Server->Client: SUCCESS

Title: Server WRITE Failure
Client->Server: WRITE 
Server->Client: SUCCESS
Client->Server: BLOCK*
Client->Server: PROGRESS*
Note right of Client: 0 or more blocks and\n0 or more progress\nmessages, intertwined
Server->Client: FAILURE or DENIED\nor ABORT
Client->Server: ABORT
Server->Client: SUCCESS

Title: Create Folder
Client->Server: WRITE
Server->Client: SUCCESS
Client->Server: BLOCK
Note right of Client: 1 block w/ folder path
Client->Server: FINAL
Server->Client: SUCCESS

Title: File or Folder Delete
Client->Server: SELECT w/ Path
Server->Client: SUCCESS 
Client->Server: DELETE w/ Progress On
Server->Client: PROGRESS*
Note left of Server: 0 or more progress\nmessages
Server->Client: SUCCESS or FAILURE\nor DENIED

Title: File or Folder Move
Client->Server: SELECT w/ Path
Server->Client: SUCCESS 
Client->Server: MOVE w/ Progress On\nDest paths
Server->Client: PROGRESS*
Note left of Server: 0 or more progress\nmessages
Server->Client: SUCCESS or FAILURE\nor DENIED

Title: File Rename Only
Client->Server: SELECT w/ File\nand new name
Server->Client: SUCCESS 
Client->Server: MOVE 
Server->Client: SUCCESS or FAILURE\nor DENIED

Title: Update File or Folder Attribs 
Client->Server: SELECT w/ Path\nand Attributes
Server->Client: SUCCESS 
Client->Server: UPDATE w/ Progress On
Server->Client: PROGRESS*
Note left of Server: 0 or more progress\nmessages
Server->Client: SUCCESS or FAILURE\nor DENIED

Title: Perform Operation
Client->Server: SEND_WAIT w/ JSON Body
Note left of Server: collect a JSON\nresponse and then send\nclient waits
Server->Client: SUCCESS or FAILURE\nor DENIED

Title: Perform Operation Alt
Client->Server: SEND_FORGET w/ JSON Body
Note left of Server: immediate response (sent or refused)
Server->Client: SUCCESS or FAILURE\nor DENIED

Title: Batch Upload w/ Multiple Clients
Participant Client 1
Participant Client 2
Participant Client 3
Participant Server
Note left of Client 1: All clients \njoin a batch
Client 1->Server: BATCH_UPLOAD_START
Server->Client 1: SUCCESS
Client 2->Server: BATCH_UPLOAD_RESUME
Note right of Server: RESUME means JOIN
Server->Client 2: SUCCESS
Client 3->Server: BATCH_UPLOAD_RESUME
Server->Client 3: SUCCESS
Note left of Client 1: All clients \nenter write mode
Client 1->Server: WRITE 
Server->Client 1: SUCCESS
Client 2->Server: WRITE 
Server->Client 2: SUCCESS
Client 3->Server: WRITE 
Server->Client 3: SUCCESS
Note left of Client 1: All clients \nwrite blocks
Client 1->Server: BLOCK*
Client 2->Server: BLOCK*
Client 3->Server: BLOCK*
Note left of Client 1: All clients \nexit WRITE mode
Client 1->Server: FINAL
Server->Client 1: SUCCESS
Client 2->Server: FINAL
Server->Client 2: SUCCESS
Client 3->Server: FINAL
Server->Client 3: SUCCESS
Note left of Client 1: Two clients \nSAVE
Client 2->Server: BATCH_UPLOAD_SAVE
Server->Client 2: RESPONSE_*
Client 3->Server: BATCH_UPLOAD_SAVE
Server->Client 3: RESPONSE_*
Note left of Client 1: One client\nFINISHes
Client 1->Server: BATCH_UPLOAD_FINISH
Server->Client 1: RESPONSE_*

Above could be multiple clients or multiple connections from a client. One client might take all files in a SELECT under 100MB and handle those while other client might handle the 100MB and up files. A good server model can handle batch uploads across servers - but not across zones.

BATCH_UPLOADs may count as Contributions to a Deposit. Deposit handling can be across zones, so break up Deposit into multiple contributions to handle across zones. Keep in mind that BATCH_UPLOAD has to process quickly with RESPONSE_SUCCESS meaning the batch is approved.

Feature dcBus - Gadreel/divconq GitHub Wiki

Intro to dcBus

Connection Management

Connection Pool

dcClient API (CTP)

CTP Commands

Base Commands

CTP_CMD_EXIT

CTP_CMD_EXIT_SIGN_OUT

CTP_CMD_ALIVE

CTP_CMD_ENGAGE

Bus Commands

CTP_B_CMD_STATUS

CTP_B_CMD_SEND

CTP_B_CMD_SEND_FORGET

Startup Commands

CTP_S_CMD_ENGAGE

CTP_S_CMD_RESPONSE

Message Xchange Commands

CTP_M_CMD_INIT

CTP_M_CMD_SETTING

CTP_M_CMD_START

CTP_M_CMD_SEND

CTP_M_CMD_SEND_FORGET

CTP_M_CMD_RESPONSE

File Transfer Commands

CTP_F_CMD_INIT

CTP_F_CMD_SETTING

Stream Notes

CTP_F_CMD_STREAM_READ

CTP_F_CMD_STREAM_WRITE

CTP_F_CMD_STREAM_BLOCK

CTP_F_CMD_STREAM_PROGRESS

CTP_F_CMD_STREAM_FINAL

CTP_F_CMD_STREAM_ABORT

CTP_F_CMD_SEND_WAIT

CTP_F_CMD_SEND_FORGET

CTP_F_CMD_OP_SELECT

CTP_F_CMD_OP_DELETE

CTP_F_CMD_OP_MOVE

CTP_F_CMD_OP_UPDATE

CTP_F_CMD_BATCH_UPLOAD_START

CTP_F_CMD_BATCH_UPLOAD_RESUME

CTP_F_CMD_BATCH_UPLOAD_FINISH

CTP_F_CMD_BATCH_UPLOAD_STATUS

CTP_F_CMD_BATCH_UPLOAD_SAVE

CTP_F_CMD_BATCH_UPLOAD_ABORT

CTP_F_CMD_BATCH_DOWNLOAD_START

CTP_F_CMD_BATCH_DOWNLOAD_RESUME

CTP_F_CMD_BATCH_DOWNLOAD_FINISH

CTP_F_CMD_BATCH_DOWNLOAD_STATUS

CTP_F_CMD_BATCH_DOWNLOAD_SAVE

CTP_F_CMD_BATCH_DOWNLOAD_ABORT

Evidence Notes

CTP_F_CMD_EVIDENCE_MANUAL

CTP_F_CMD_RESPONSE

CTP_F_CMD_PROGRESS

TODO

CTP_F Rules

Client Regular Mode

Client Read Mode

Client Write Mode

CTP-F Example Usage

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️