ICAR Animal Data Exchange Working Group – Technical Meeting 18 November 2021

• In attendance: Andrew C, Andreas S, Arjan L, Anton H, Craig V, Graham M, Thomas P

Agenda:

Items to discuss today are: a streaming API for bulk data sharing or update; type classification; how to use UTC time when only date is needed.

1. How to use UTC time when only date is needed

Thomas raised the question of how to use UTC time when only the date is needed. Arjan pointed out that Dealing with time is included in the Design Principles.

2. Draft generic API (#263)

The pull request is a draft outline of the generic data sharing specification and also plays with ideas around semantic typing, identifiers on resource level. That is, it plays around with some ideas highlighted in the original issue #257.

We considered the files attached to the request starting with the description of a Generic Data Exchange API. It discusses the rationale then the beginnings of the core endpoints to support this. Each resource declares its type in a data form. The idea is that these datasets could contain entities and resources of many different types. If the dataset declares what kind of types are in them and resources can tell you what kind of thing they are, it is a very flexible system. It gives you as a client a programmatic way to know how to consume objects of a particular type. Organisations could pump all changes through one dataset. This is a key concept that would be good for us to discuss further.

Changes endpoint: basic idea of the protocol is that a client is trying to keep a local database or data representation in sync with a service offering this data. Can make a Get request to the /Changes endpoint and optionally pass a sinceToken. The changes endpoint returns a JSON array including resources of the type curently available. The last object in the array is the continuation token. It can be used for paging through initial sets of data when you are syncing for the first time and also it is used as the token you pass every time you want to get changes that have occurred. The resources you get are the most up to date representation of those resources.

Graham added to the base resource two properties, identifier and class. Likely we also need location here or ensure it is in every resource.

This is quite a large change so one suggestion was to split it up into different topics. Also concern about the requirement that this protocol (in particular continuation tokens) would place on the server. There was discussion about the complexity on the server.

Continuation tokens solve the issue of clients and servers having different date-time representations (discussed at past meetings).

The approach sets up a proper data sharing highway between organisations who want to do this. When new data types come along they can flow down the same roads. You don't need to define new endpoints - it just works.

We could prototype pieces of this, even from a specification perspective, and learn from the process.

3. Conformance and data model changes

Arjan suggested we link this to a layered model. We could have multiple levels of compliance (e.g. compliance on a message level). Might help us to structure this into the basic stuff (resources and semantic class definitions, id discussions); the streaming API next to the current locations API.

We could first determine the minimum change we would need to make to the data model, doing it in a way that is backwards compatible and enable us to get to having a working API. A good way to make progress could be to make id an attribute of all icarResource types, rather than just events. This currently does not exist in animals (which have official and alternative identifiers) but could be added, and then in future made mandatory.

It was suggested we work out what the changes need to look like so we can work out how it feels.

There was discussion on whether any of these changes could be made in a 1.x version or whether they need to be held over to version 2.0

The URN scheme is a bigger change and Graham said it felt to him more like a definite 2.0 thing.

Action: Graham will create an issue with some justifications to identify the minimal set of non-breaking backwards compatible changes.

4. Other discussion

As well as working on these technical changes we do need to keep working on the functionality that people want us to add. That is, are there more resource types we should be adding or issues with existing resources we need to be improving?

Andrew met with the broader Animal Exchange Working Group and we have done most of their priorities. One thing they want concerns semantics of how you record an event like a generalised feeding event or a generalised health treatment event against a group of animals. We may need to deal more with sensor data.

Producing a road map and how that stacks up with the 2.0 release was suggested.

Next meeting scheduled for 2 December 2021 at 8:15am CET