Feature FileOps Remote - Gadreel/divconq GitHub Wiki

Remote to DivConq (FUTURE)

There are a lot of additional options when both client and server are based on DivConq. Using the DivConq library to build a client solution can streamline many operations.

TODO expand on features

Intake Paths

During an upload the server accepts the file content from the client and then accepts evidence that the upload is intact. This can be the end of it, the file may be stored and ready for use or archiving. However, the server may also do a final step of processing called an Intake - after evidence is presented and before the file is considered to be truly ready.

By default no Intake Path is defined in dcFileServer. You may configured a default Intake Path, or simply make Intake Paths available but optional. When an IntakePath is used then the uploaded file is stored in a local temp folder until evidence is given. Once acceptable evidence is provided then a Intake Script is run as defined for the Path. That script will be provided with source and intended destination values.

Good for:

  • Bulk Transfer (see below)
  • Virus check
  • File validation
  • TODO more

Bulk Transfer

When transferring many smallish files it makes more sense to transfer them in bulk as if they had been tarred and gzipped. This approach uses only one connection (e.g. HTTP) and one channel to send many files. Because many clients may have access to tar and gzip the standard for DivConq Bulk Transfer is based on these transformers.

Client Side Script Example I

This is a literal example of how the client can initiate a Bulk Transfer using streams. Following example assumes we already have a remote connection defined in $conn1.

<Var Name="Date" SetTo="2014-09-06" />
	
<LocalFolder Name="DepositSrc" Path="/files/{$Date}" />
<CtpFileStore Name="DepositDest" RootPath="/deposit/{$Date}" Connection="$conn1" />

<FileOps>
	<Tar Name="CombinedDeposit" Source="$DepositSrc" />
	<Gzip Name="DeflatedDeposit" Source="$CombinedDeposit" />
	<Hash Name="ClientHash" Method="SHA256" Source="$DeflatedDeposit" StreamTo="HashedDeposit" />
	<Copy Source="$HashedDeposit" Dest="$DepositDest" IntakePath="BulkTransfer"
		EvidenceName="SHA256" EvidenceValue="$ClientHash" />
</FileOps>

$ClientHash is calculated on demand so it is not available until Copy is done, but we handle that automatically inside Copy.

Server Side Script Example I

When processing a BulkTransfer flagged file the file is automatically placed in a Temporary folder on the server and then an expand script is run. The expand script will automatically have:

  • $Args.TempFile - the ".tar.gz" file uploaded (raw bytes uploaded)
  • $Args.DestFile - the destination file (folder), for example /deposit/2014-09-06 from client above
  • $Args.EvidenceName - the evidence type presented
  • $Args.EvidenceValue - the evidence value presented

Evidence will typically already by processed. TODO review when this is not the case?

Following is a simplification of the server's expand script:

<FileOps>
	<Ungzip Name="InflatedDeposit" Source="$Args.TempFile" />
	<Untar Source="$InflatedDeposit" Dest="$Args.DestFile" />
</FileOps>

Client Side Script Example II

Example I can be condensed as follows, with the added advantage of Evidence auto-discovery.

<Var Name="Date" SetTo="2014-09-06" />
	
<LocalFolder Name="DepositSrc" Path="/files/{$Date}" />
<RemoteFileStore Name="DepositDest" RootPath="/deposit/{$Date}" Connection="$conn1" />

<FileOps>
	<BulkTransfer Source="$DepositSrc" Dest="$DepositDest" />
</FileOps>

Client Side Script Example III

BulkTransfer can be done on a regular .tar.gz file. The operation automatically adopt to the fact that the file is already zipped.

<Var Name="Date" SetTo="2014-09-06" />
	
<LocalFile Name="DepositSrc" Path="/files/{$Date}.tar.gz" />
<CtpFileStore Name="DepositDest" RootPath="/deposit/{$Date}" Connection="$conn1" />

<FileOps>
	<BulkTransfer Source="$DepositSrc" Dest="$DepositDest" />
</FileOps>

Client Side Script Example IV

BulkTransfer is best with small files as it cannot do a resume on individual files, bulk is an all or nothing operation. The following shows how to send all small files in bulk while sending larger files (2MB and up) individually for more effecient retries.

<Var Name="Date" SetTo="2014-09-06" />
	
<LocalFolder Name="DepositSrc" Path="/files/{$Date}" />

<SelectFiles Name="SmallSrc" In="$DepositSrc">
	<SizeFilter LessThan="2097152" />
</SelectFiles>

<SelectFiles Name="RemainingSrc" In="$DepositSrc">
	<SizeFilter GreaterThanOrEqual="2097152" />
</SelectFiles>

<CtpFileStore Name="DepositDest" RootPath="/deposit/{$Date}" Connection="$conn1" />

<FileOps>
	<BulkTransfer Source="$SmallSrc" Dest="$DepositDest" />
	
	<XCopy Source="$RemainingSrc" Dest="$DepositDest" />
</FileOps>
⚠️ **GitHub.com Fallback** ⚠️