Messaging and Queue Options

In general, AWS provides tremendous services and allows for rapid development. However, some inherent limitations mean we may want to explore other options. For instance, if an enterprise customer requested their own instance of axe-linter that they could create custom workers for, an aws-only deployment would prevent that.

SQS/SNS (queue, pubsub)

Pro:

• Scalable (out of the box)
• Non-FIFO by default (we don’t really care who goes first)
• Guaranteed message delivery Con: Small(ish) messag* e size (limit 256kb)
• Hard to run locally (below solutions need testing)
• Moto - run a local mock of aws infrastructure for tests
• Localstack - fully functional local aws stack
• SNS needed so we don’t constantly poll the queue

Postgres Listen/Notify (pubsub)

Example project Pro:

• We can run it locally and also run it in RDS Con:
• Scaling could be tricky
• We don’t have Postgres “pro” on staff. Unsure if it would require db maintenance.
• RabbitMQ (queue, pubsub) Pro:
• We can run locally, runs in EC2 on aws
• High message size (2MB, theoretically much higher) Con:
• Can fail to deliver messages (usually misconfiguration or dead listener)
• Could be tricky to scale
• New tech for the team

Kafka

Pro:

Con:

NSQ

Pro:

Con:

Automerge (worker)

A CRDT that runs over webRTC to create a JSON-like structure. It fully supports arrays so a queue would be trivial. Pro:

• Javascript (team’s native language)
• Easy to mock or run locally, runs on virtually anything (including EC2 or lamda)
• Resilient, immutable (think web torrents)
• Multi-writer approach allows workers to be written at any scale (could even connect to chrome)
• Unlinking a torrent essentially clears the data from the “database” (persisting a document just means writing to disk/db)
• No message size limits (needs POC) Con:
• Relatively new concepts for the team to grasp
• webRTC over aws networks would need to be tested