Connector - Twasi/twasi-core GitHub Wiki

Twasi-Twitch-Connector

Until now we connected to the Twitch IRC via sockets directly from Twasi-Core. Although this works and is very simple, it has massive problems when it comes down to scaling.

For example every time someone restarts the bot (for example because of a bug or deployment), the bot needs to disconnect from each and every socket and reconnect. That is not a problem for approx. 100 users, since it only takes about one minute (that's one minute downtime every restart, although). But imagine if it were 1000 users. 10 minutes is a lot!

Introducing Twasi-Twitch-Connector

  • [Me]: Hey Twasi, please reduce the loading time from 10 minutes to 10 seconds. Ohh and when you're on it, please also not make starting time dependent on number of users.
  • [Twasi-Twitch-Connector]: Ok!

By separating the connection to Twitch IRC, we gain several advantages:

  • We can restart Twasi-Core without bothering about reconnecting etc. since the connector stays connected.
  • Even if we restart and a messages comes in at that exact moment, it gets queued and will still be handled! Wow!

Details

Twasi-Twitch-Connector is a seperate application, that connects to Twasi-Core via RabbitMQ.

With that in mind, an incoming message from Twitch will be processed like that: Incoming message: Twitch --- IRC --> Twasi-Twitch-Connector --- RabbitMQ --> Twasi-Core Outgoing message: Twasi-Core --- RabbitMQ --> Twasi-Twitch-Connector --- IRC --> Twitch

The instance of Twasi-Twitch-Connector needs to have some kind of state, because he needs to know in which channel he has to listen. All this traffic will be controlled over a seperate RabbitMQ channel.

Channels

There are three RabbitMQ channels:

MASTER_CTRL

MASTER_CTRL channel is for general communication. Specific tasks:

  • Submitting and acknowledging list of Twitch-Channels
  • Heartbeat, Status
  • Version check

MESSAGE_IN

Only specific (message) packets, that get forwarded from Twitch IRC and contain all the information that is available via Twitch IRC.

MESSAGE_OUT

Only specific (message) packets, that just will get forwarded to Twitch IRC and contain all the information that is needed.

Input & Output

Because the input and output channels are shared among all users, every packet sent will have a reference to a certain user. This will probably be the Twitch ID and the channel name.

Example

  1. Twasi-Twitch-Connector is started
  2. RabbitMQ channels are created by Twasi-Twitch-Connector
  3. Twasi-Twitch-Connector waits for Twasi-Core to send initial packet
  4. Twasi-Core is started
  5. Twasi-Core places initial packet, containing version number and list of channels
  6. Twasi-Twitch-Connector joins all supplied channels (if not already), responds if okay (startup of Twasi-Core will be blocked until then)
  7. Twasi-Twitch-Connector notifies Twasi-Core that it's ready
  8. Message forwarding will work

This is the basic setup. Now if Twasi-Core has to be restarted, the following will happen:

  1. Twasi-Core goes offline. Twasi-Twitch-Connector doesn't get notified.
  2. All incoming messages in registered channels will be queued up
  3. Twasi-Core is again starting
  4. Twasi-Core places initial packet containing version number and list of channels
  5. Since Twasi-Twitch-Connector is already in some channels, it only has to fix the delta. This is much faster, and often it doesn't have to take any action at all.
  6. Twasi-Twitch-Connector notifies Twasi-Core that the changes were applied and it's ready
  7. All previous sent messages will be read by Twasi-Core and proceeded. This could lead to some delay but still better thant don't handle them at all.
  8. Message forwarding will work again