Getting started with a new Data Collector development - noi-techpark/odh-docs GitHub Wiki
Getting started with Java
Step 1: Install Docker and Docker Compose
The recommended way to test and develop data collectors for the Open Data Hub writer API is to use Docker.
You need to install two packages:
Step 2: Install the Open Data Hub Writer API
- Go to https://github.com/noi-techpark/bdp-core
git clonethat repository to your local machine- Follow the instructions in bdp-core/README.md/Getting started with Docker
Step 3: Install the Helloworld data collector
- Go to https://github.com/noi-techpark/bdp-commons
git clonethat repository to your local machinecd data-collectors/helloworld- Try it out, if you like:
cp .env.example .envdocker-compose up
- Eventually connect to the DB to see what came in, or have a look at the logs
Step 4: Clone the Helloworld data collector to get started with your own development
- Read and follow our Getting Started guidelines
- Copy/paste the helloworld example in a new folder under
data-collectors, choose the name of your data collector for that folder - Find
TODOcomments and follow their instructions - See and alter code inside
SyncScheduler.java - Start the writer API locally and test everything:
- Writer API with Docker
- Start your data collector
- Check log outputs of the writer and the data collector to find issues
- Connect to the DB, and see what is their after some tests
- Create a pull request as described in the guidelines above
Generic Data collector
If you don't use Java you can still push data to the Open Data Hub trough it's JSON API. Following are some generic curl examples on how to do that.
There is also a basic openapi spec available
Authentication
We use Keycloak to authenticate. That service provides an access_token that
can be used to authenticate your request to writer API.
See here for further details.
curl -X POST -L "https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token" \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'grant_type=client_credentials' \
--data-urlencode 'client_id=odh-mobility-datacollector-development' \
--data-urlencode 'client_secret=7bd46f8f-c296-416d-a13d-dc81e68d0830'
With this call you get an access_token that can then be used as follows in all
writer API methods. Here just an example to get all stations:
curl -X GET "http://localhost:8999/json/stations" \
--header 'Content-Type: application/json' \
--header 'Authorization: bearer your-access-token'
You should get an empty JSON list as result.
Write an email to [email protected], if you want to get the client_secret
and an Open Data Hub OAuth2 account for a non-development setup.
Set provenance
The provenance is the combined by the following values, so that later on we can determine from which data collector and which version the data came from. The provenance need to be set only once on program startup.
- uuid: is always null
- dataCollector: name of the data collector itself like 'odh-mobility-dc-flighdata-skyalps'
- dataCollectorVersion: latest git commit sha of deployed version to server or version number of the data collector like 'zh7shz' or '1.0.7'
- lineage: name of the origin of the data like 'Skyalps'
curl -X POST "https://share.opendatahub.testingmachine.eu/json/provenance"
--header 'Content-Type: application/json'
--header 'Authorization: bearer your-access-token'
-d '{"uuid":null,"dataCollector":"odh-docs-example","dataCollectorVersion":"docs-example-version","lineage":"docs-example"}'
You should get the provence uuid as response like "jxkjBCqd". Save that uuid because you'll need it to push the data.
Sync data types
Every data entry can have its own datatype and if they do not already exist in the Open Data Hub, you need to create a new data type. If there are no new data types after the application started, they need to be set only once on program startup. With the following call you can create mutiple datatypes with by sending a json list with hte following values:
- name: Unique name (ex., air-temperature)
- descirption: (optional) Description of the data type (ex., Air temperature)
- period: (optional) Interval on how often a measurement with this data type gets collected (seconds)
- unit: (optional) Unit of the data type (ex., °C)
- rtype: (optional) Metric of the data type (ex., mean or instantaneous)
- metadata: (optional) Further detail information about given datatype as JSON
curl -X POST -L "https://share.opendatahub.testingmachine.eu/json/syncDataTypes" \
--header 'Content-Type: application/json' \
--header 'Authorization: bearer your-access-token' \
-d '[
{
"name":"docs-example-type",
"unit":"odh-docs-unit",
"rtype":"docs-example-rtype",
"description":"docs-example description",
"period": 600,
"metadata":
{
"extra":"fields"
}
},
{
"name":"docs-example-type-2",
"unit":"odh-docs-unit",
"rtype":"docs-example-rtype",
"description":"docs-example description",
"period": 300,
"metadata":
{
"even-more":"fields"
}
}
]'
You should get a 201 response code if everything is ok.
Sync stations
To create new stations or synchronize their properties with the Open Data Hub you need to make the following request. The data needs to be a list with Station objects with the following fields:
- id: A unique id for the station
- name: A unique name for the station, can be also the id
- origin: The origin from where the data comes from
- latitude: (optional) GPS latitude value like "46.454"
- longitude: (optional) GPS longitude value like "11.254"
- municipality: (optional) Name of the municipality the station is in
- metaData: (optional) Some further information to the station as JSON object
curl -X POST -L "https://share.opendatahub.testingmachine.eu/json/syncStations/ExampleStation" \
--header 'Content-Type: application/json' \
--header 'Authorization: bearer your-access-token' \
-d '[
{
"id": "example-station-id-1",
"name": "example-station-name-1",
"origin": "docs-example",
"latitude": 46.333,
"longitude": 11.356,
"municipality": "Bolzano",
"metaData" : {
"additional": "fields"
}
},
{
"id": "example-station-id-2",
"name": "example-station-name-2",
"origin": "docs-example",
"latitude": 46.333,
"longitude": 11.356,
"municipality": "Bolzano",
"metaData" : {
"additional": "fields"
}
}
]'
Push data
After having created the provenance, datatype and stations you can finally push the data to the Open Data Hub. The root of the data JSON is this:
{
"name": "(default)",
"branch": {},
"data": [],
"provenance": ""
}
Then every branch can have multiple station branches that then can have multiple data-type branches, that then has the data filed filled with data records.
{
"name": "(default)",
"branch": {
"example-station-id-1": {
"name": "(default)",
"branch": {
"docs-example-type": {
"name": "(default)",
"branch": {},
"data": [
{
"timestamp": 1668522653150,
"value": 1234,
"period": 100,
"_t": "it.bz.idm.bdp.dto.SimpleRecordDto"
}
]
}
},
"data": []
}
},
"data": [],
"provenance": "jxkjBCqd"
}
Here the curl example sending 6 data records for 2 different stations and 2 different data types:
curl -X POST -L "https://share.opendatahub.testingmachine.eu/json/pushData" \
--header 'Content-Type: application/json' \
--header 'Authorization: bearer your-access-token' \
-d '{
"name": "(default)",
"branch": {
"example-station-id-1": {
"name": "(default)",
"branch": {
"docs-example-type": {
"name": "(default)",
"branch": {},
"data": [
{
"timestamp": 1668522653150,
"value": "plapla",
"period": 100,
"_t": "it.bz.idm.bdp.dto.SimpleRecordDto"
},
{
"timestamp": 1668522653150,
"value": "plapla",
"period": 100,
"_t": "it.bz.idm.bdp.dto.SimpleRecordDto"
}
]
},
"docs-example-type-2": {
"name": "(default)",
"branch": {},
"data": [
{
"timestamp": 1668522653150,
"value": "plapla",
"period": 100,
"_t": "it.bz.idm.bdp.dto.SimpleRecordDto"
},
{
"timestamp": 1668522653150,
"value": "plapla",
"period": 100,
"_t": "it.bz.idm.bdp.dto.SimpleRecordDto"
}
]
}
},
"data": []
},
"example-station-id-2": {
"name": "(default)",
"branch": {
"docs-example-type": {
"name": "(default)",
"branch": {},
"data": [
{
"timestamp": 1668522653150,
"value": "plapla",
"period": 100,
"_t": "it.bz.idm.bdp.dto.SimpleRecordDto"
},
{
"timestamp": 1668522653150,
"value": "plapla",
"period": 100,
"_t": "it.bz.idm.bdp.dto.SimpleRecordDto"
}
]
}
},
"data": []
}
},
"data": [],
"provenance": "jxkjBCqd"
}'
Support
For support, please contact [email protected].
Glossary
BDP (Big Data Platform)
Old name of the mobility/timeseries platform that today is called Open Data Hub. We now use BDP as a shorthand to refer to the timeseries writer API.
Data collector
Small program (microservice) that collects data from an external source, and sends it to the Open Data Hub via the BDP writer API
Elaboration
A data collector that instead of getting external data, transforms data from the Open Data Hub (e.g. some mathematical model, or a simple sum aggregation by time)
Idempotence
Data collectors should generally be idempotent, which means they should be able to called multiple times and still lead to the same result. Syncing stations and data types are idempotent operations by default. When records are pushed, the BDP API silently (!!) discards any records that are older than what it already has in the Open Data Hub. This means that if you push the same record twice by mistake, only the first one will be written. This also means that you cannot write old missing data, if there is already newer one existing. This constraint works on a station/datatype/period level. e.g. if the period is different you can still push older data
Running a local Instance
clone the bdp-core project copy .env.example to .env docker compose up
This runs the API and a local empty postgres instance. You can now execute example calls in calls.http
When developing a data collector, authorization should also just work (for local development):
OAUTH_AUTH_URI=https://auth.opendatahub.testingmachine.eu/auth
OAUTH_TOKEN_URI=https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token
OAUTH_BASE_URI=http://localhost:8999/json
OAUTH_CLIENT_ID=odh-mobility-datacollector-development
OAUTH_CLIENT_NAME=odh-mobility-datacollector-development
OAUTH_CLIENT_SECRET=7bd46f8f-c296-416d-a13d-dc81e68d0830
Our 2024 bootcamp project has a complete docker-compose file including bdp, ninja, db and analytics working without configuration
Example implementations
BDP client library (Java) https://github.com/noi-techpark/bdp-core/blob/main/client/src/main/java/com/opendatahub/timeseries/bdp/client/json/JSONPusher.java