Shard implementation notes - saalfeldlab/n5 GitHub Wiki

General support

  1. Will the N5 format support sharding?
  2. Will N5Reader and N5Writer have shard methods?

If the N5Reader and N5Writer have shard methods, then the the readers / writers for the N5 format will have to have those methods, and so will either have to implement them, or UnsupportedOperationException or similar, which would be confusing / misleading.

Responsibility and delegation

  1. Is sharding a property of a reader/writer or a dataset?

Should

Choices for defaults and backward compatibility

  1. Should we aim to have all existing methods readBlock and writeBlock work for sharded data.

Yes, absolutely. If not, ALL old code will have to be re-written.

  1. Should we aim to have old methods be as performant as possible.

Yes, absolutely.

  1. Is it possible that using only old methods will be as performant as using newer methods when dealing with shards.

Certainly not realistic. Likely not possible.

Other links