Feature FileOps Remote - Gadreel/divconq GitHub Wiki
There are a lot of additional options when both client and server are based on DivConq. Using the DivConq library to build a client solution can streamline many operations.
TODO expand on features
During an upload the server accepts the file content from the client and then accepts evidence that the upload is intact. This can be the end of it, the file may be stored and ready for use or archiving. However, the server may also do a final step of processing called an Intake - after evidence is presented and before the file is considered to be truly ready.
By default no Intake Path is defined in dcFileServer. You may configured a default Intake Path, or simply make Intake Paths available but optional. When an IntakePath is used then the uploaded file is stored in a local temp folder until evidence is given. Once acceptable evidence is provided then a Intake Script is run as defined for the Path. That script will be provided with source and intended destination values.
Good for:
- Bulk Transfer (see below)
- Virus check
- File validation
- TODO more
When transferring many smallish files it makes more sense to transfer them in bulk as if they had been tarred and gzipped. This approach uses only one connection (e.g. HTTP) and one channel to send many files. Because many clients may have access to tar and gzip the standard for DivConq Bulk Transfer is based on these transformers.
This is a literal example of how the client can initiate a Bulk Transfer using streams. Following example assumes we already have
a remote connection defined in $conn1
.
<Var Name="Date" SetTo="2014-09-06" />
<LocalFolder Name="DepositSrc" Path="/files/{$Date}" />
<CtpFileStore Name="DepositDest" RootPath="/deposit/{$Date}" Connection="$conn1" />
<FileOps>
<Tar Name="CombinedDeposit" Source="$DepositSrc" />
<Gzip Name="DeflatedDeposit" Source="$CombinedDeposit" />
<Hash Name="ClientHash" Method="SHA256" Source="$DeflatedDeposit" StreamTo="HashedDeposit" />
<Copy Source="$HashedDeposit" Dest="$DepositDest" IntakePath="BulkTransfer"
EvidenceName="SHA256" EvidenceValue="$ClientHash" />
</FileOps>
$ClientHash
is calculated on demand so it is not available until Copy
is done, but we handle that automatically inside Copy.
When processing a BulkTransfer
flagged file the file is automatically placed in a Temporary folder on the server and then an expand script is run. The expand script will automatically have:
-
$Args.TempFile
- the ".tar.gz" file uploaded (raw bytes uploaded) -
$Args.DestFile
- the destination file (folder), for example/deposit/2014-09-06
from client above -
$Args.EvidenceName
- the evidence type presented -
$Args.EvidenceValue
- the evidence value presented
Evidence will typically already by processed. TODO review when this is not the case?
Following is a simplification of the server's expand script:
<FileOps>
<Ungzip Name="InflatedDeposit" Source="$Args.TempFile" />
<Untar Source="$InflatedDeposit" Dest="$Args.DestFile" />
</FileOps>
Example I can be condensed as follows, with the added advantage of Evidence auto-discovery.
<Var Name="Date" SetTo="2014-09-06" />
<LocalFolder Name="DepositSrc" Path="/files/{$Date}" />
<RemoteFileStore Name="DepositDest" RootPath="/deposit/{$Date}" Connection="$conn1" />
<FileOps>
<BulkTransfer Source="$DepositSrc" Dest="$DepositDest" />
</FileOps>
BulkTransfer
can be done on a regular .tar.gz
file. The operation automatically adopt to
the fact that the file is already zipped.
<Var Name="Date" SetTo="2014-09-06" />
<LocalFile Name="DepositSrc" Path="/files/{$Date}.tar.gz" />
<CtpFileStore Name="DepositDest" RootPath="/deposit/{$Date}" Connection="$conn1" />
<FileOps>
<BulkTransfer Source="$DepositSrc" Dest="$DepositDest" />
</FileOps>
BulkTransfer
is best with small files as it cannot do a resume on individual files, bulk is an all or nothing
operation. The following shows how to send all small files in bulk while sending larger files (2MB and up)
individually for more effecient retries.
<Var Name="Date" SetTo="2014-09-06" />
<LocalFolder Name="DepositSrc" Path="/files/{$Date}" />
<SelectFiles Name="SmallSrc" In="$DepositSrc">
<SizeFilter LessThan="2097152" />
</SelectFiles>
<SelectFiles Name="RemainingSrc" In="$DepositSrc">
<SizeFilter GreaterThanOrEqual="2097152" />
</SelectFiles>
<CtpFileStore Name="DepositDest" RootPath="/deposit/{$Date}" Connection="$conn1" />
<FileOps>
<BulkTransfer Source="$SmallSrc" Dest="$DepositDest" />
<XCopy Source="$RemainingSrc" Dest="$DepositDest" />
</FileOps>