ICAR Animal Data Exchange Working Group – Technical Meeting 2 December 2021

• In attendance: Alexey S, Andrew C, Andreas S, Arjan L, Anton H, Craig V, Erwin S, Graham M, Thomas P

Agenda:

Items to discuss today are: add semantic type tp resource; add isDeleted to meta; a streaming API for bulk data sharing or update; type classification.

1. Type classification (#262)

Erwin outlined his pull request in which he added an API for type-classification and changed the conformation API to reuse the conformation part. Andrew pointed out that this is a breaking change, but if the icarConformationScoreType was incorporated using "allOf" it could be a non-breaking change. Anton to review change request

2. Introduce Streaming API (#257)

Graham believe a streaming API can be introduced with zero breaking changes to the existing data model. He has created a project which identifies the core issues for introducing a streaming API to support mass synchronisation of data from many locations between different organisations and systems which maintain these sets of data. We will start today looking at the first 2 or 3 issues.

3. Add isDeleted to meta (#265)

For the streaming protocol to work it is necessary for resources to indicate if they have been deleted. The proposal is to add a new boolean field isDeleted to the meta structure. This is backwards compatible and makes sense alongside the other meta properties. The value is optional and defaults to false. If the resource is deleted, the assumption is that the modified timestamp is when it was deleted. The next thing is to create a pull request and review it. It was suggested we could add an issue to provide for an isDeleted filter in the API.

4. Add semantic type to resource (#266)

In general it would be useful if you looked at a fragment of JSON and it told you what kind of thing it was. Proposed adding a new optional property to resource called resourceType. Its value is a URL. We should create a list of URIs for all of the existing ICAR types. Andrew will talk to the ICAR ADE team to find out what we can do about domains.

5. Add location or context to resource (#267)

For the streaming API it is necessary to provide context regarding which farm or account the data is related to. We have 2 options.

1. Add location as a property of resource. This is backwards compatible. Any resource definitions on subtypes would be moved up to resource.
2. Add a 'context' property to meta that allows for this to be conveyed in an application independent way.

It was agreed to proceed with option 1 with location as a property of resource.

6. Add id to icarResource.json (#264)

For the streaming API to work all resources must have an id. There are a number of options:

1. add id field to icarResource. Simple string value. Aligns well with the current use of id on many sub resources. e.g. icarResource.json and many others. Means some resources without the id property will now need to be given it. Potential semantic clash with id of events?
2. add id field to meta. Protocol woudl state: resource instance representations must have the meta property and this must contain meta and id. Backwards compatible. Requires protocol to define the need for this to be present. No change to any resources as is. id would be defined as string. Maybe call this resourceId?
3. Mandate the use of @self property in the protocol. This by definition must be a globally unique value. No changes needed for this. Slightly mixes addressing and identity.

Following discussion a fourth option was suggested, namely adding the field sourceId to icarResource. Needs to be unique in the system within the scope of Source. We should consider these options.

Next meeting scheduled for 16 December 2021 at 8:15am CET