Implementing a service - adewg/ICAR GitHub Wiki

This page describes the ICAR ADE standard from a service perspective; what do you need to know to build API's or offer ADE messages as described by the standard. You don't need to be a member of ICAR in order to use the standards (although there are many benefits to membership).

For more general information on ICAR and this standard, see About ICAR and the Animal Data Exchange Working Group.

Introduction

The standard consists of JSON-schema definitions defining the messages and an accompanying (recommended) URL scheme. The messages can be transported via many channels and thus can be used by themselves. We also provide a recommended URL scheme which allows REST-style access to these messages.

Most developers should be familiar with REST and JSON files. To simply view the standard, you can open the exampleUrlScheme.json file in your favorite OpenAPI Specification editor. Most developers should be familiar with REST and JSON files. To simply view the standard, you can open the exampleUrlScheme.json file in your favorite OpenAPI Specification editor. For example, open it in the popular Swagger Editor. For a quick guide on getting started with code generation, see Generating code.

Backward and forward compatibility

The standard aims to cover most cases in a generic way. However, there will always be regional differences or vendor specific fields that may be required to be part of a message. You are free to simply add more fields to your messages than described in the standard. Clients using the standard are recommended to use a tolerant-reader pattern. They should ignore fields that they do not recognize. Adding new fields to a message is thus possible.

When doing so, please take note of the following recommendations:

• Consider if your field is potentially standardizable. If other regions or vendors may require something similar, define it in such a way that it may become part of the standard at one time. By doing this, no technical changes may be required by the time a new version comes out which includes this field.
• If your field is not potentially standardizable, then make sure that the name you choose does not potentially conflict with future fields. E.g., use a prefix for your region or company name.

Synchronization with a client

There are use cases where a client has a requirement to know for certain that all events of your service have been consumed. For example, for a milk recording agency it is essential to make sure all milking events have been recorded. This synchronization between a client and a source is not trivial and there are many ways to solve it. The ADE standard prescribes the following pattern to allow for synchronization.

Each event message has modified and source fields. The modified field is a datetime indicating the latest change to this messages record. The source field indicates the system of record for the message. All data sources are required to fill these fields. The modified field must be guaranteed to be monotonically increasing, e.g. it never goes back in time. This holds for each event related to the same source. A client thus may keep track of the latest received modified datetime per source and query all events from that time on. This guarantees that all changes will be retrieved.

Note that the monotonic requirement is for the modified datetime. It is not necessary for specific event timestamps. Devices that are directly accessible and use a local clock should configure that clock to be monotonic. The source scope should then be tied to the device to indicate that each device has its own clock. Device data that is published via a cloud could instead use the cloud server's clock to fill the modified field. The source scope could then be tied to the cloud environment. Again, make sure that your systems clock is set to a monotonic mode.

There is a scenario that we do not cover in this specification: in case a device with a local clock has drifted too much, an operator may decide to reset its clock either forward or backward. If it is reset backward in time, clients may miss data recorded in that correction period. In case of such a hard reset, clients should be prepared to recapture a larger period in time. How to detect this is out of the scope of the spec and assumed a manual proces, similar to reparations needed with hardware failure or other kinds of data loss. In this scenario, the client should be able to rely on the ‘eventId’ being unique for the ’source’.

The url scheme should thus allow to query for meta-modified-from=x and meta-source=y. See Filtering resources for more details.

If your device cannot meet these requirements, please contact us. The WG discussed future extensions: if messages (or rather, devices) are defined for which relying on a datetime ('modified') cannot be implemented, the standard could define a tolerance period for those messages (e.g. it cannot go back in time more than 24hrs), or the standard could be extended with specific synchronization fields (for example, a specialized Sync group with sourceID and syncOffset allowing other synchronization methods). This will be considered only when these messages present itself with a use case.