Feature FileOps - Gadreel/divconq GitHub Wiki
Within [dcScript] (Feature dcScript) there is a selection of file processing operations known as File Ops.
In DivConq a FileStore represents a local or remote collection of files that may be accessed via a path.
For a local file store:
<LocalFileStore Name="LocalDeposits" RootFolder="[local folder]" RootPath="/deposits" />
Remote DivConq File System (via API) (FUTURE)
<CtpFileStore Name="RemoteDeposits" RootPath="/" Connection="..." />
SFTP (FUTURE)
<SftpFileStore Name="SftpDeposits" RootPath="/" Connection="..." />
S3 (FUTURE)
<S3FileStore Name="S3Deposits" RootPath="/uploads" Connection="..." />
Zip file exposed as a FileStore (future)
<ZipFileStore Name="ZipDeposits" RootPath="/deposits.zip" />
WebDAV (future)
<WebDAVFileStore Name="WDDeposits" RootPath="/files" Connection="..." />
Hadoop Distributed File System (HDFS) (future)
<HdfsFileStore Name="HadoopDeposits" RootPath="/" Connection="..." />
Create a file reference using File or Folder.
<File Name="DepositA" Path="/karen/deposita.tar.enc" In="$LocalDeposits" />
This indicates that this is a file inside the LocalDeposits file store. So the full local path for DepositA is /deposits/karen/deposita.tar.enc.
<Folder Name="FolderB" Path="/b" In="$LocalDeposits" />
This is a folder inside the LocalDeposits file store, full path is /deposits/b.
It is just as easy to refer to a file in a remote file store, for example:
<File Name="DepositA" Path="/karen/deposita.tar.enc" In="$S3Deposits" />
This is the file /uploads/karen/deposita.tar.enc inside a bucket in the S3 file store.
<Folder Name="FolderB" Path="/b" In="$ZipDeposits" />
This is folder /b inside the zip file.
Once you have a folder defined like this:
<Folder Name="FolderB2c" Path="/bee/two/see" In="$ZipDeposits" />
Then you can create a file or folder under that folder by referencing the folder instead of the store:
<File Name="Checklist" Path="/steve/list.txt" In="$FolderB2c" />
This file is /bee/two/see/steve/list.txt inside the Zip file.
Or
<Folder Name="Pictures" Path="/pictures" In="$FolderB2c" />
This folder is /bee/two/see/pictures inside the Zip file.
A path independent of any file store.
<Path Name="PathA" Value="/pictures" />
For ease of use there are LocalFiles and LocalFolders which take a complete path:
<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />
The full local path for DepositA is /karen/deposita.tar.enc.
<LocalFolder Name="FolderB" Path="/b" />
The full local path is /b.
If you want a temporary (disk) file or folder that will be automatically removed at the end of the script:
<TempFile Name="FileA" />
<TempFolder Name="FolderB" />
If you want a temporary (heap) file or folder that will be automatically removed at the end of the script:
<HeapFile Name="FileA" />
<HeapFolder Name="FolderB" />
A collection of filestore file references:
<FileCollection Name="SomeFiles">
<Add File="$DepositA" />
<Add File="$CheckList" />
</FileCollection>
Most standard file operations are available, provided by the FileOps feature:
<FileOps>
...copy, delete, etc...
</FileOps>
<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />
<LocalFolder Name="FolderB" Path="/b" />
<FileOps>
<Copy Source="$DepositA" Dest="$FolderB" />
</FileOps>
Result is /b/deposita.tar.enc
<FileOps>
<Copy Source="$DepositA" Dest="$FolderB" Relative="true" />
</FileOps>
Result is /b/karen/deposita.tar.enc
<LocalFolder Name="FolderA" Path="/a" />
<File Name="DepositA" Path="/karen/deposita.tar.enc" In="$FolderA" />
<LocalFolder Name="FolderB" Path="/b" />
<FileOps>
<Copy Source="$DepositA" Dest="$FolderB" RelativeTo="$FolderA" />
</FileOps>
Results in /a/karen/deposita.tar.enc
being copied to /b/karen/deposita.tar.enc
.
<LocalFolder Name="FolderA" Path="/a" />
<Folder Name="FolderB2c" Path="/bee/two/see" In="$FolderA" />
<LocalFolder Name="FolderX" Path="/x" />
<FileOps>
<Copy Source="$FolderB2c" Dest="$FolderX" RelativeTo="$FolderA" />
</FileOps>
Results in files /a/bee/two/see/*
being copied to /x/bee/two/see/*
. Where as:
<LocalFolder Name="FolderA" Path="/a" />
<Folder Name="FolderB2c" Path="/bee/two/see" In="$FolderA" />
<LocalFolder Name="FolderX" Path="/x" />
<FileOps>
<XCopy Source="$FolderB2c" Dest="$FolderX" />
</FileOps>
Results in files /a/bee/two/see/* being copied to /x/* with relative paths intact.
Attributes:
-
Source
stream or file(s) to copy -
Dest
file/folder to copy to -
Relative
true or false, should we use source relative path -
RelativeTo
resolve to folder path or a literal path (e.g. RelativeTo="/dest1"), automatically enables RelativeTo as well
Same as Copy but always relative, no need for Relative="true".
Move works like Copy, for example:
<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />
<LocalFolder Name="FolderB" Path="/b" />
<FileOps>
<Move Source="$DepositA" Dest="$FolderB" />
</FileOps>
Result is /karen/deposita.tar.enc is moved to /b/deposita.tar.enc.
Attributes:
-
Relative
- true or false, should we use source relative path -
RelativeTo
- resolve to folder path or a literal path (e.g. RelativeTo="/dest1"), automatically enables RelativeTo as well
Same as Move but always relative, no need for Relative="true".
Creating a file reference does not automatically create the file or folder. Use Create to make a new empty file or folder from a reference. For a file:
<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />
<FileOps>
<Create Dest="$DepositA" />
</FileOps>
For a folder (creates parent folders if necessary):
<LocalFolder Name="FolderB" Path="/b" />
<FileOps>
<Create Dest="$FolderB" />
</FileOps>
For a file:
<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />
<FileOps>
<Delete Dest="$DepositA" />
</FileOps>
For a folder (delete folder contents if necessary):
<LocalFolder Name="FolderB" Path="/b" />
<FileOps>
<Delete Dest="$FolderB" />
</FileOps>
Take a file and create a variable holding the hash value:
<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />
<FileOps>
<Hash Name="DepositAHash" Method="MD5" Source="$DepositA" />
</FileOps>
<Console>Hash value for {$DepositA.Path} is {$DepositAHash}</Console>
Hash methods:
- CRC32
- CRC64
- MD5
- SHA128
- SHA256
- SHA384
- SHA512
Hash Attributes:
- Name
- Method
- Source
- StreamTo (see streaming)
Files and folders have the following properties:
- Name The file or folder name
- Size File size (only if exists)
- Modified File or folder last modified date time (only if exists)
- IsFolder True if is marked as a folder
- Exists True if the file or folder is present in the file store
- Path Is relative to the file store
- FullPath For local file stores only, is the complete file path ignoring the file store layer
Examples:
<LocalFolder Name="FolderB" Path="/b" />
<If Target="$FolderB.IsFolder">
...
</If>
Or:
<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />
<If Target="$DepositA.Exists">
...
</If>
Encryption, compression, splitting and other operations fall under [Advanced Operations] (Feature FileOps Advanced).
Sometimes file operations are more efficient if they are strung together, see issue #1 for an example.
dcScript provides a convenient method for [streaming operations] (Feature FileOps Streaming).