Overview

We preserve Moab objects on storage by zipping each version's files (without compressing so there's no data loss) and replicating the zipped file to geographically and organizationally diverse endpoints. Very large zip files are broken into parts by the zip software, so individual writes to endpoints are less vulnerable to replication problems.

Because the Moab format stores each object update as a new version, we are able to preserve new objects as version 1, and subsequent updates according to the new Moab version created.

Replication Flow Diagram:

graph TD;
    PreservedObject.after_update --> PreservedObject.create_zipped_moab_versions!;
    MoabRecord.after_create --> PreservedObject.create_zipped_moab_versions!;
    MoabRecord.after_update --> PreservedObject.create_zipped_moab_versions!;
    PreservedObject.create_zipped_moab_versions! --> ZipmakerJob;
    ZipmakerJob --> DeliveryDispatcherJob;
    DeliveryDispatcherJob --> S3EastDeliveryJob;
    DeliveryDispatcherJob --> S3WestDeliveryJob;
    DeliveryDispatcherJob --> IbmSouthDeliveryJob;
    S3EastDeliveryJob --> ResultsReporterJob;
    S3WestDeliveryJob --> ResultsReporterJob;
    IbmSouthDeliveryJob --> ResultsReporterJob;

Details of Replication Steps

1. What Starts Replication

• Where in the code?:
  • PreservedObject.after_update hook
  • MoabRecord.after_create hook
  • MoabRecord.after_update hook
• Description: after_xxx hooks are used to ensure objects new to PreservationCatalog and new versions of objects already in PreservationCatalog are replicated.
• Next job(s) called: PreservedObject.create_zipped_moab_versions!
• Where to find job failures: Honeybadger

2. PreservedObject.create_zipped_moab_versions!

• Where in the code?: PreservedObject is an ActiveRecord model
• Description: Ensure we have ZippedMoabVersion db table rows for every ZipEndpoint for each Moab version by calling ZipmakerJob for any missing moab version on a zip endpoint.
• Next job(s) called: Replication::ZipmakerJob for each missing version for each endpoint
• Where to find job failures: Honeybadger

3. Replication::ZipmakerJob

• Description:
  • Creates Replication::DruidVersionZip objects for the given replication endpoint.
    • calls DruidVersionZipParts.new if zip is over threshhold of 10g.
  • If needed, writes zip file(s) to zip cache and calculate checksum(s); otherwise, touches the existing cached zip file(s)
• Next job(s) called: Replication::DeliveryDispatcherJob for each zip file.
• Where to find job failures: Sidekiq queues

4. Replication::DeliveryDispatcherJob

• Description: for each ZippedMoabVersion, dispatches unreplicated parts to the indicated zip endpoint
• Next job(s) called: S3 endpoint delivery jobs for each zip part, specifically
  • S3WestDeliveryJob
  • S3EastDeliveryJob
  • IbmSouthDeliveryJob
• Where to find job failures: Sidekiq queues

5. Replication::S3WestDeliveryJob, ...::S3EastDeliveryJob, ...::IbmSouthDeliveryJob

• Where in the code?: these jobs are subclasses of AbstractDeliveryJob which has the perform method.
• Description: Posts individual zip part to S3 endpoint if not there already.
• Next job(s) called: Replication::ResultsRecorderJob
• Where to find job failures: Sidekiq queues

6. Replication::ResultsRecorderJob

• Description: When this completes the replication of all ZipParts for a ZippedMoabVersion for each S3 endpoint, send message to event service. E.g. if we have 3 endpoints, 3 events will be created.
• Where to find job failures: Sidekiq queues