Batch Fluent Interface - Watts-Energy/Watts.Azure GitHub Wiki

The fluent interface for building Azure Batch executions several options available, which are not documented in the main README.md.

The various options are available at certain steps in the process, and will be documented that way here. The first step is the root, BatchBuilder.cs.

Step 1: BatchBuilder

  • UsingAccountSettings(BatchAccountSettings settings): This provides the basic account settings for your batch account. A simpler way is to use an implementation of IPredefinedBatchEnvironment, which will save you a step.
  • InPredefinedEnvironment(IPredefinedBatchEnvironment environment): Specify an environment, which contains the account info as well as the credentials for the storage account linked to the batch account.
  • AsNonPrimaryBatch(AzureMachineConfig machineConfig): This is only relevant when you are building a Hybrid batch. This skips the account and storage account information as that has already been specified in the primary batch. You must specify a machine config (for now) until we get the correct order of things optimized.

Step 2: IBatchCreationWithAccountInfo This step is only relevant if you used UsingAccountSettings above

  • UsingStorageAccountSettings(StorageAccountSettings storageAccountSettings): Specifies the credentials to use the storage account linked with the Batch account.

Step 3: IBatchCreationWithBatchAndStorageAccountSettings

  • RunStartupCommandOnAllNodes(BatchConsoleCommand command): Specify a command that should be run on all Batch nodes as they enter the pool.
  • ResolveDependenciesUsing(IDependencyResolver dependencyResolver): Specify the IDependencyResolver implementation that will find all files that your main executable depends on and return the local paths to them in a list.
  • ResolveDependenciesUsing(List<IDependencyResolver> dependencyResolvers): The same as the above, but with multiple such implementations that each return a subset of the dependencies. This is useful if there are multiple types of dependencies, e.g. dlls and various config files.
  • NoDependencies(): Specify that there are no dependencies for the executable

Step 4: IBatchCreationWithDependencyResolver

  • WithDefaultPoolSetup(): Specify that the batch should use the default pool name (BatchPool) and default job name (BatchJob). Note that this means that it could conflict with an existing batch pool and/or job that you have created, since these names are unique within an account.
  • WithPoolSetup(BatchPoolSetup poolSetup): Specify the name of both the pool and the job.

Step 5: IBatchCreationWithPoolSetup

  • ConfigureMachines(AzureMachineConfig machineConfig): Specify the types of virtual machines to use.
  • WithDefaultMachineConfig(): This is the same as AzureMachineConfig.Small().Instances(2), i.e. 2 Standard A1 windows machines.
  • WithOneSmallMachine(): Equivalent to AzureMachineConfig.Small().Instances(1)
  • PrepareInputUsing(IPrepareInputFiles inputPreparer): Specifies the implementation that will create input files and return a list of strings containing the paths to the created files. This moves to the next step.

Step 6: IBatchCreationWithInputPreparation

  • ReportingProgressUsing(Action<string> progressDelegate): An action delegate to report progress on.
  • ReportProgressToConsole(): Equivalent to ReportProgressUsing(Console.WriteLine).
  • DownloadOutput: Will cause the output of each task to be downloaded. The stdout and stderr of the task is stored as lists of strings in the batch execution and can be retrieved after the task finishes.
  • LogTo(ILog log): Will cause errors to be logged to the specified log. This is not ILog from log4net, but a similar interface with a small subset of log4net methods.
  • SaveStatistics(): If invoked, statistics about the batch execution is saved after it finishes. The table to store this in is created (if it does not already exist) in the storage account linked to the batch account.
  • DontSaveStatistics(): Specifically state that statistics should not be saved. This is default and you don't need to invoke this, but it is never wrong to be specific.
  • SetTimeoutInMinutes(int minutes): Specifies the timeout, in minutes, after which the Watts.Azure will throw an exception unless all tasks have reached the completed state.
  • ExecuteRScript(string scriptFileName): Specifies the file containing the R script to execute in Azure Batch. This moves the process to the next step (7.1).
  • ExecuteRCode(string[] code): Specify the R code to execute. This is just written to a file named main.R (with a Guid prepended) and handled the same was as ExecuteRScript("./main.R"). This moves to the next step (7.1).
  • RunExecutable(string executableFilePath): Specify the path to the executable to run in Azure Batch. This moves to the next step (7.2).

Step 7.1: RBatchCreation

  • UseRVersion(string version): Specify the R version of the application package you've uploaded to batch (see the project README).
  • SetMaxMemoryForRScript(int megaBytes): Set the max memory of the RScript process that will be run in batch.
  • WithApplicationPackageReference(ApplicationPackageReference packageReference): Specify additional application packages that your tasks depend on (in addition to R).
  • WithAdditionalScriptExecutionArgument(string arg): Specify additional arguments to pass to the R script. By default and at minimum, the script will receive the name of the input file it is to execute on, so these are any parameters you need to pass in addition to that. This can be invoked as many times as you like.
  • GetBatchExecution(): Finalizes the building process and returns a BatchExecutionBase ready to execute.

Step 7.2: ExecutableBatchCreation

  • UsingExecutableInfo(BatchExecutableInfo execInfo): Specify the names of the input and application containers. This is already handled by Watts.Azure, so this is only relevant in case you want to override the names it has generated (which are application<some guid> and input<some guid>).
  • WithApplicationPackageReference(ApplicationPackageReference packageReference): Adds a application package reference. This requires you to upload an application package through the Azure Portal in advance.
  • WithAdditionalScriptExecutionArgument(string arg): By default the executable will receive the path to the input file it is to process, so this adds an extra argument. This method can be invoked as many times as necessary.
  • GetBatchExecution(): Finalizes the building process and returns a BatchExecutionBase ready to execute.

Once the tasks have finished, you can retrieve the output of all tasks using GetExecutionOutput(). This returns a list of TaskOutput entities which hold the stdout and stderr lines in the StdOut and StdErr properties respectively.

⚠️ **GitHub.com Fallback** ⚠️