Consume data - Mobility-Data-Space/mobility-data-space GitHub Wiki

Contract negotiation

Click on the tab Catalog Browser and enter the endpoint URL of the provider's connector into the Connector Endpoints field. Your connector will automatically show you all available offerings of the provider's Connector, which this Connector is entitled to show by the offerings access policy.

You can also connect to multiple providing Connectors at once by entering multiple Connector endpoints and seperating them in via a comma.

The information symbol in the "Connector Endpoint"-search bar provides basic information on the status of the connection process to the entered providing Connectors as well as potential error messages.

The details of the contract offers for an asset and their policies attached become visible when clicking on an asset. If there are several different contract offers for the offered asset which may have different policies attached, several contract offers will be displayed. The appropriate data offering can then be selected based on the set policies.

Click Negotiate to start a contract negotiation for the desired data offer. You will see a prompt asking you to accept the standard license and policies that come with the contract offer.

If you haven’t done so already, take a moment to review the "Standard License" and "Policy" sections in the contract offer. Click anywhere on the screen to go back to the contract offer. Once you’ve reviewed everything, click on the checkbox and on the Confirm button.

As soon as the contract negotiation ends successfully with an agreement, the new contract will be visible in the Contract Agreements tab.

Data transfer

To transfer the data to your desired data sink go to the Contract Agreements page. On this page, all your contract agreements are displayed, consuming as well as providing contract agreements. You can quickly distinguish consuming and providing contracts with the arrow left to the Contract Agreement (Up = providing, Down = Consuming). You can inspect details on the contract agreements by clicking on a single contract agreement.

When clicking on a contract agreement a pop-up window will show up and display details on the contract agreement.

transfer

After pressing the Transfer button, you have following options.

Push

In a Push transfer, the provider sends the data directly to the consumer’s endpoint.
To initiate a Push transfer, fill in the following fields:

  • Type
    The transfer type : REST-API Endpoint.
  • Method
    The HTTP method used to send the data (e.g., POST).
  • Base URL *
    The consumer’s endpoint that will receive the pushed data.
    This field is mandatory.
  • Path
    Optional URL path appended to the Base URL.

If your data-sink is protected by, for example, required headers or authentification, these can also be added in here.

For test purposes and only for public data you can use services like Webhook, which show all information from incoming HTTP requests.

Pull

In a Pull transfer, the consumer retrieves the data from the provider. Pull is typically used when the consumer initiates the data request and the provider exposes an endpoint where the data can be retrieved.

Be aware that only the first part of two-steps pull process is supported through the frontend.

To enable Pull, select the transfer type REST-API Endpoint and activate the checkbox:

  • Data address PULL
    Check this box to switch the transfer mode to Pull and start the transfer.

As answer you will receive from the provider a so called EDR (Endpoint Data Reference) which contains a provider's endpoint and an authorization token (if the authorization is needed), which are available under EDRs tab of the frontend.

The final step can be done using the Management API

Kafka

For Kafka-based assets, the transfer establishes a streaming connection between the provider’s Kafka cluster and the consumer. Please select Kafka Streaming option.

If you don't provide a callback address you will get an EDR from the provider (visible under the tab EDRs) with all informations you needed to initiate the transfer process manually

If you provide a callback endpoint the connection with your Kafka client will be established manually.

Amazon S3

For S3-based assets, a transfer allows the consumer to retrieve one or multiple objects from the configured S3 bucket. Please select Amazon S3 option. Fill in the fields shown in the transfer dialog:

  • Multiple S3 Objects
    Enable this option if the transfer should include more than one object from the bucket.
  • Region *
    The AWS region where the S3 bucket is located.
  • Bucket Name *
    The name of the bucket that contains the S3 object(s).
  • Object Name
    The exact file to be retrieved during the transfer.
    Required when Multiple S3 Objects is not selected.
  • Folder Name
    Used when Multiple S3 Objects is enabled.
    Defines the folder/prefix inside the bucket from which objects will be retrieved.
  • Keyname
    Optional. Reference to S3-related client properties stored in your environment.

During an S3 transfer, the connector fetches the selected object or all objects under the specified folder and provides them to the consumer.

Azure Blob Storage

For Azure Blob–based assets, the transfer retrieves one or multiple blobs from the configured Azure Storage account. Please select Azure Blob Storage option.

  • Multiple Azure Blobs
    Enable this option if the transfer should include multiple blobs from the same container.
  • Container *
    The Azure Blob container that holds the blob(s) to be retrieved.
  • Account *
    The name of the Azure Storage account that contains the container.
  • Folder Name
    Used when Multiple Azure Blobs is enabled.
    Defines the prefix/folder inside the container from which blobs will be retrieved.
  • Blob Name
    Required when Multiple Azure Blobs is not selected.
    Specifies the exact blob to be transferred.
  • Keyname *
    A required reference to Azure Blob client properties stored in your environment.

During the transfer, the connector fetches the blob or blobs from the container and makes them available to the consumer.

Transfer history

The Transfer History tab provides a list of all incoming and outgoing data transfers that were started with this Connector.

transfer3

You can for example identify the date, the involved asset, the state, the counterpart MDS Connector ID, the involved connectors or look up further details with a click on Show details.