Data Factory Fluent Interface - Watts-Energy/Watts.Azure GitHub Wiki

The Data Factory fluent interface is used to run Copy activities (https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview).

Each step involved in building a copy activity is documented here:

Step 1: DataCopyBuilder

  • InDataFactoryEnvironment(IDataCopyEnvironment environment): specifices the environment settings of the data factory to use, i.e. the Subscription id, storage account connection string, credentials etc.

Step 2: DataCopyBuilderWithEnvironment

  • UsingDataFactorySetup(AzureDataFactorySetup dataFactorySetup): specifies the resource group name and name of the data factory.

Step 3: DataCopyBuilderWithDataFactorySetup

  • UsingDefaultCopySetup(): Specify that the default copy setup should be used, i.e. predefined names for the source/target dataset and linked service and time out of 60 minutes.
  • UsingCopySetup(CopySetup setup): Specify specific names for the datasets and linked services as well as the time out. Both of these methods moves forward to the next step:

Step 4: DataCopyBuilderWithCopySetup

  • WithTimeoutInMinutes(int numberOfMinutes): Set the number of minutes before the copy activity times out.
  • CreateTargetTableIfNotExists(): Means that the target table (or file in the case of Data Lake) will be created if they don't exist already.
  • AuthenticateUsing(IAzureActiveDirectoryAuthentication authenticator) Specify the Azure Active Directory authenticator containing credentials, etc. used to authenticate towards the Data Factory. This moves forward to the next step.

Step 5: DataCopyBuilderWithAuthentication

  • CopyFrom(IAzureLinkedService source): Sets the source linked service (e.g. an AzureTableStorage instance) to copy from.

Step 6: DataCopyBuilderWithSource

  • WithSourceQuery(string query): Optionally specify a query to limit the results from the data source. If the source is e.g. Azure Table Storage you can use OData syntax (e.g. "PartitionKey eq 'x').
  • To(IAzureLinkedService target): Specifies the target service and moves to the next step

Step 7: DataCopyBuilderWithSourceAndTarget

  • DoNoCleanUpAfter(): Specify that the data factory should not be deleted when the pipline completes.
  • ReportProgressToConsole(): Progress and various status messages will be written to console.
  • ReportProgressTo(Action<string> progressAction): Is an alternative to ReportToConsole where you can specify an action where messages will be directed.
  • StartCopy() Starts the copy activity and waits for it to finish.
⚠️ **GitHub.com Fallback** ⚠️