Feature FileOps - Gadreel/divconq GitHub Wiki

Within [dcScript] (Feature dcScript) there is a selection of file processing operations known as File Ops.

In DivConq a FileStore represents a local or remote collection of files that may be accessed via a path.

For a local file store:

<LocalFileStore Name="LocalDeposits" RootFolder="[local folder]" RootPath="/deposits" />

Remote DivConq File System (via API) (FUTURE)

<CtpFileStore Name="RemoteDeposits" RootPath="/" Connection="..." />

SFTP (FUTURE)

<SftpFileStore Name="SftpDeposits" RootPath="/" Connection="..." />

S3 (FUTURE)

<S3FileStore Name="S3Deposits" RootPath="/uploads" Connection="..." />

Zip file exposed as a FileStore (future)

<ZipFileStore Name="ZipDeposits" RootPath="/deposits.zip" />

WebDAV (future)

<WebDAVFileStore Name="WDDeposits" RootPath="/files" Connection="..." />

Hadoop Distributed File System (HDFS) (future)

<HdfsFileStore Name="HadoopDeposits" RootPath="/" Connection="..." />

Create a file reference using File or Folder.

<File Name="DepositA" Path="/karen/deposita.tar.enc" In="$LocalDeposits" />

This indicates that this is a file inside the LocalDeposits file store. So the full local path for DepositA is /deposits/karen/deposita.tar.enc.

<Folder Name="FolderB" Path="/b" In="$LocalDeposits" />

This is a folder inside the LocalDeposits file store, full path is /deposits/b.

It is just as easy to refer to a file in a remote file store, for example:

<File Name="DepositA" Path="/karen/deposita.tar.enc" In="$S3Deposits" />

This is the file /uploads/karen/deposita.tar.enc inside a bucket in the S3 file store.

<Folder Name="FolderB" Path="/b" In="$ZipDeposits" />

This is folder /b inside the zip file.

Once you have a folder defined like this:

<Folder Name="FolderB2c" Path="/bee/two/see" In="$ZipDeposits" />

Then you can create a file or folder under that folder by referencing the folder instead of the store:

<File Name="Checklist" Path="/steve/list.txt" In="$FolderB2c" />

This file is /bee/two/see/steve/list.txt inside the Zip file.

Or

<Folder Name="Pictures" Path="/pictures" In="$FolderB2c" />

This folder is /bee/two/see/pictures inside the Zip file.

A path independent of any file store.

<Path Name="PathA" Value="/pictures" />

For ease of use there are LocalFiles and LocalFolders which take a complete path:

<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />

The full local path for DepositA is /karen/deposita.tar.enc.

<LocalFolder Name="FolderB" Path="/b" />

The full local path is /b.

If you want a temporary (disk) file or folder that will be automatically removed at the end of the script:

<TempFile Name="FileA" />

<TempFolder Name="FolderB" />

If you want a temporary (heap) file or folder that will be automatically removed at the end of the script:

<HeapFile Name="FileA" />

<HeapFolder Name="FolderB" />

A collection of filestore file references:

<FileCollection Name="SomeFiles">
	<Add File="$DepositA" />
	<Add File="$CheckList" />
</FileCollection>

Most standard file operations are available, provided by the FileOps feature:

<FileOps>
	...copy, delete, etc...
</FileOps>

<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />

<LocalFolder Name="FolderB" Path="/b" />

<FileOps>
	<Copy Source="$DepositA" Dest="$FolderB" />
</FileOps>

Result is /b/deposita.tar.enc

<FileOps>
	<Copy Source="$DepositA" Dest="$FolderB" Relative="true" />
</FileOps>

Result is /b/karen/deposita.tar.enc

<LocalFolder Name="FolderA" Path="/a" />

<File Name="DepositA" Path="/karen/deposita.tar.enc" In="$FolderA" />

<LocalFolder Name="FolderB" Path="/b"  />

<FileOps>
	<Copy Source="$DepositA" Dest="$FolderB" RelativeTo="$FolderA" />
</FileOps>

Results in /a/karen/deposita.tar.enc being copied to /b/karen/deposita.tar.enc.

<LocalFolder Name="FolderA" Path="/a" />

<Folder Name="FolderB2c" Path="/bee/two/see" In="$FolderA" />

<LocalFolder Name="FolderX" Path="/x"  />

<FileOps>
	<Copy Source="$FolderB2c" Dest="$FolderX" RelativeTo="$FolderA" />
</FileOps>

Results in files /a/bee/two/see/* being copied to /x/bee/two/see/*. Where as:

<LocalFolder Name="FolderA" Path="/a" />

<Folder Name="FolderB2c" Path="/bee/two/see" In="$FolderA" />

<LocalFolder Name="FolderX" Path="/x"  />

<FileOps>
	<XCopy Source="$FolderB2c" Dest="$FolderX"  />
</FileOps>

Results in files /a/bee/two/see/* being copied to /x/* with relative paths intact.

Attributes:

• Source stream or file(s) to copy
• Dest file/folder to copy to
• Relative true or false, should we use source relative path
• RelativeTo resolve to folder path or a literal path (e.g. RelativeTo="/dest1"), automatically enables RelativeTo as well

Same as Copy but always relative, no need for Relative="true".

Move works like Copy, for example:

<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />

<LocalFolder Name="FolderB" Path="/b" />

<FileOps>
	<Move Source="$DepositA" Dest="$FolderB" />
</FileOps>

Result is /karen/deposita.tar.enc is moved to /b/deposita.tar.enc.

Attributes:

• Relative - true or false, should we use source relative path
• RelativeTo - resolve to folder path or a literal path (e.g. RelativeTo="/dest1"), automatically enables RelativeTo as well

Same as Move but always relative, no need for Relative="true".

Creating a file reference does not automatically create the file or folder. Use Create to make a new empty file or folder from a reference. For a file:

<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />

<FileOps>
	<Create Dest="$DepositA" />
</FileOps>

For a folder (creates parent folders if necessary):

<LocalFolder Name="FolderB" Path="/b" />

<FileOps>
	<Create Dest="$FolderB" />
</FileOps>

For a file:

<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />

<FileOps>
	<Delete Dest="$DepositA" />
</FileOps>

For a folder (delete folder contents if necessary):

<LocalFolder Name="FolderB" Path="/b" />

<FileOps>
	<Delete Dest="$FolderB" />
</FileOps>

Take a file and create a variable holding the hash value:

<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />

<FileOps>
	<Hash Name="DepositAHash" Method="MD5" Source="$DepositA" />
</FileOps>

<Console>Hash value for {$DepositA.Path} is {$DepositAHash}</Console>

Hash methods:

• CRC32
• CRC64
• MD5
• SHA128
• SHA256
• SHA384
• SHA512

Hash Attributes:

• Name
• Method
• Source
• StreamTo (see streaming)

Files and folders have the following properties:

• Name The file or folder name
• Size File size (only if exists)
• Modified File or folder last modified date time (only if exists)
• IsFolder True if is marked as a folder
• Exists True if the file or folder is present in the file store
• Path Is relative to the file store
• FullPath For local file stores only, is the complete file path ignoring the file store layer

Examples:

<LocalFolder Name="FolderB" Path="/b" />

<If Target="$FolderB.IsFolder">
	...
</If>

Or:

<LocalFile Name="DepositA" Path="/karen/deposita.tar.enc" />

<If Target="$DepositA.Exists">
	...
</If>

Encryption, compression, splitting and other operations fall under [Advanced Operations] (Feature FileOps Advanced).

Sometimes file operations are more efficient if they are strung together, see issue #1 for an example.
dcScript provides a convenient method for [streaming operations] (Feature FileOps Streaming).