Handling Sequence Latency - adamcfraser/cbnotes GitHub Wiki

We want the Sync Gateway to be able to better adapt to occasional slow sequence numbers. Currently we're buffering all sequence numbers, and will block any updates past the current highest sequence number until it shows up on the feed.

Chris proposed a composite sequence number approach, where incoming _changes requests could spawn two processes - a 'fast feed' that returned sequences as they show up on the feed, and a buffered feed that would fill any gaps.

This conceptual approach doesn't align well with the current feed handling by the SG, unfortunately. We don't spawn new TAP/DCP feeds per incoming request - we just have a single instance of the feed which is used to populate the channel caches. Incoming _changes requests are then served from the caches (with backfilling by view when needed).

The following proposal tries to follow the spirit of Chris's proposal - using a composite sequence number to allow SG to send sequences while waiting for gaps to be filled, with some tweaks to have it work with the channel caches.

Summary

Use the existing buffered feed, with a lowered timeout for giving up on a pending sequence number (1 sec or less). When the timeout is reached, instead of skipping the sequence, track the missed sequence numbers in an ordered set (missed). When missed isn't empty, sending sequence numbers to the client as compound sequence numbers of the form missed[0]:seqno.

When an skipped sequence number eventually shows up on the feed, check whether that sequence is present in the missed set. If so, send that sequence to the appropriate channel caches and remove it from missed.

Handling client requests with compound sequences

When a client makes a _changes request with since=low:high, Sync Gateway will handle as if it were a request with since=low. Even though the client may have previously received some values between low and high, we don't know what gaps may have existed in that range, and need to resend the entire set. Sequences greater than the current missed[0] on the server will be sent as missed[0]:seqno.

SG doesn't use the high component on an incoming _changes request - it's only sent to the client to identify the real sequence number of a revision with a compound key, and to distinguish between multiple keys with the same 'low' value.

Filling missed values during continuous processing

When a missed sequence arrives, it triggers (new) processing to push that value to the channel cache, and alert any current continuous processes that have channel visibility to the sequence.

Each time the continuous feed handling completes an iteration (see How _changes works), it should update the value it writes as 'low' in the compound key to the current missed[0] value.

The implementation will need to ensure that there isn't a race condition where the missed[0] gets pushed to the continuous feed at the same time a new iteration is rolling over, so that sequence numbers are sent with low=(the new missed[0]) before the filled value has been sent.

Purging long-missing sequences from missed

We want to handle the case where a sequence is actually missing (and not just coming late on the feed). This should be an infrequent scenario (such as SG crash while holding a sequence for update), so we can set a relatively large timeout for purging from the missed set.

Store a timestamp with the entry in the missed set to represent when the sequence was added to the set. Set up a new goroutine to remove entries from missed after the timeout has expired. Can review whether it makes sense to attempt retrieve the sequence using a view query before purging, if there's a scenario where it doesn't appear on the feed but is still available from a view.

Handling SG restarts

Propose to store the missed set in memory only, at least initially. On SG restart (today), we listen on the TAP feed for sequences greater than the high sequence number for the server (from the _sync:seq document), so we shouldn't have to worry about gaps created during startup.

The problematic scenario is the case where, on SG startup, sequences earlier than _sync:seq aren't yet available to a view query. Need to investigate whether this is a legitimate concern. If so, one approach would be to execute a view query for the n sequences prior to _sync:seq, and use those to both warm the channel caches and initialize the missed[] set.

Periodic persistence of the missed set for a SG node is another possible approach that could be considered as a future enhancement - the current high sequence number and missed set could be persisted periodically, and on restart SG could warm the channel caches from the previous missed[0].

Handling multiple SG nodes

It's possible that different SG nodes talking to the same CBS cluster would each have a different missed set. This shouldn't be a problem - a client that jumps between SG nodes will just get different missed[0] values in their compound key if they switch between SG nodes, but at worst will retrieve redundant documents (that will get handled by the client revs_diff).