Technical Minutes 2020 10 23 - adewg/ICAR GitHub Wiki

ICAR Animal Data Exchange Working Group – Technical meeting 23 October 2020

  • In attendance: Andrew C, Anton H, Arjan L, Craig V, Erwin S, Kor O, Marvin R, Thomas P.

Agenda:

  1. Feed Intake API
  2. Semantics of POST Method
  3. Identifiers for POSTing resources
  4. Meeting times

1. Feed Intake API (#93)

  • Erwin will follow up progress on this with Thomas D and Kor.
  • Andrew added some feed type categories #172. There is an official set of categories defined from 1980. There may be value in adding some more categories, but would need care.
  • Andrew added, and Erwin provided more information on Feed Properties #173. There would be benefit in having a predefined list of common properties, but this must be able to be extended.

2. POST method semantics (#154)

  • Should posts allow only a single resource to be added, or allow batch record insertion (Arjan and Anton had added comments)
  • Suggested to focus on a single resource first and then add batches. Batches will be necessary for many use cases, especially for integration with older systems. Encourage developers to send data "when it is completed" rather than "all at the end of the day/week/month".
  • Noted that REST APIs can be limited in POST payload size - so care would be needed to avoid body/payload/buffer size limits. We don't want pagination. The client needs to ensure it can post its data without getting a 413 Payload too large error.
  • Noted Zalando and JSON-API both support posting batches/collections - they vary in how they handle errors.
  • Agreed that if batches are used they should be at the same level as a collection URI -- for a single location and message type.
  • Discussed whether to POST a single item or collection multiple items at the same end point, or separate end points.
  • Agreed that we preferred a separate endpoint for batches vs. single:
    • Single: POST a single item to the collection URI for the location and message type
    • Batch: POST a collection of items to the collection URI with "/batch" added to the collection URI string.
  • Asynchronous vs. synchronous posting - return 201 Created for successful synchronous (location error and return posted data with updated IDs) and 202 Accepted for asynchronous posting (and return location to GET status/data).
  • Discussed how to handle errors within a batch. If all failed because of auth etc, return 4xx error, but if batch accepted but details incorrect (e.g. syntax errors or conflicts in the details), then better to return a 207 Multi-Status and a status array.
  • Agreed that we liked the idea of supporting a 207 response and will need to document how the results are to be returned.
  • Discussed how to provide the resources to the server for the /batch - is this a collection of resources OR a collection of Requests (Google extension). We preferred a collection of resources, not a collection of requests.
  • Discussed the way of returning multipart statuses and errors. Looked at https://github.com/adewg/ICAR/issues/154#issuecomment-701133516.
    • We would need to add "status" (HTTP status for each object) to the error array object.
    • We didn't like the idea of a JSON pointer to an error position in the request; it is necessary if the client is not providing instance IDs.
    • In order to remove the JSON POinter, we would need to require that clients provided unique identifiers for submitted resources.

3. Unique resource identifiers when sending new data to a server (#153)

  • Our discussion on JSON Pointers vs. Instance IDs for errors led to discussion of resource identifiers.
  • Our strong preference is for clients to use UUIDs when creating resources and submitting these to servers.
  • Some clients do not generate UUIDs but create composite IDs based on a location and number within that location and resource type.
  • Require that a resource ID MUST be unique for the location and recommend that it SHOULD be a UUID (to encourage the right behaviour for new developments).
  • We discussed the "unique" ID that might be generated by a client vs the internal ID that might be generated by a server. Which should be used by another client when using PATCH/PUT to update data?
    • We currently have @ID for every icarResource which is the URL to the item on the server (which presumably includes any ID the server might want to provide)
    • We have id which is the identifier for an icarEventResource
    • If the server generates its own identifier for an event, it will return this in the id field. If it knows it will get UUIDs, the server may choose to return those UUIDs.

4. Meeting times

We agreed to move the meeting times for European winter to every second Thursday 8:15am to 9:15am CET (7:15 am GMT, 8:15pm NZDT). Noted that some people will need to leave at 9am CET. Others won't join until 8:30am CET, but this covers most of us.

Next meeting scheduled for 05 November 2020 at 8:15am CET