Data Consumer Setup Guide - Mobility-Data-Space/mobility-data-space GitHub Wiki

This guide covers how to set up and configure a data consumer in the MDS ecosystem to consume Kafka streams through the Eclipse Dataspace Connector (EDC) and the MDS Kafka Client Backend.

Prerequisites

• MDS Connector or MDS CaaS
• Java 21+
• Docker and Docker Compose
• MDS Kafka Client Backend deployed and running

Overview

The consumer workflow involves:

1. Catalog Browsing: Discover available datasets
2. Contract Negotiation: Agree on usage terms
3. Transfer Process: Initiate data access
4. EDR Processing: Automatically handle access credentials
5. Data Consumption: Consume Kafka streams via the backend

Catalog Browsing

Discover available datasets from a data provider:

 curl -X POST <http://localhost:7182/management/v2/catalog/request> \\ 
   -H "Content-Type: application/json" \\ 
   -H "X-Api-Key: your-api-key" \\ 
   -d '{
     "@context": {
     "@vocab": "<https://w3id.org/edc/v0.0.1/ns/>" 
     }, 
     "counterPartyAddress": "<http://provider:8183/protocol>", 
     "protocol": "dataspace-protocol-http" 
    }'

Contract Negotiation

Initiate contract negotiation for the desired dataset:

  curl -X POST <http://localhost:7182/management/v2/contractnegotiations> \\ 
    -H "Content-Type: application/json" \\ 
    -H "X-Api-Key: your-api-key" \\ 
    -d '{ 
      "@context": { 
      "@vocab": "<https://w3id.org/edc/v0.0.1/ns/>" 
     }, 
    "counterPartyAddress": "<http://provider:8183/protocol>", 
    "protocol": "dataspace-protocol-http", 
    "offer": { 
      "offerId": "mobility-events-stream:streaming-policy", 
      "assetId": "mobility-events-stream", 
      "policy": 
        { "permissions": [{  
             "action": "USE", 
             "constraint": { 
                "leftExpression": "PURPOSE", 
                "operator": "EQ", 
                "rightExpression": "RESEARCH" 
             } 
          }] 
        } 
     } 
   }'

Transfer Process

Once the contract is finalized, initiate data transfer:

curl -X POST <http://localhost:7182/management/v2/transferprocesses> \\ 
  -H "Content-Type: application/json" \\ 
  -H "X-Api-Key: your-api-key" \\ 
  -d '{
    "@context": { 
    "@vocab": "<https://w3id.org/edc/v0.0.1/ns/>" 
   }, 
  "counterPartyAddress": "<http://provider:8183/protocol>", 
  "contractId": "your-contract-agreement-id", 
  "assetId": "mobility-events-stream", 
  "protocol": "dataspace-protocol-http", 
  "dataDestination": { 
     "type": "HttpProxy" 
  }, 
  "callbackAddresses": [{ 
     "transactional": false, 
     "uri": "<http://consumer-backend:8181/events>", 
     "events": ["transfer.process.started"] 
   }] 
 }'

Kafka Consumer

The consumer backend can (automatically) handle EDR callbacks and create Kafka consumers.