X Sync Model - tomboy-notes/tomboy-ng GitHub Wiki

Still trying to reverse engineer the Tomboy Sync Model

Note : Sadly, the Tomboy sync policy was not well documented. This document is based on what can be concluded by observing the process and does, to a limited extent document the changes made by tomboy-ng. It does explain some of Tomboy's problems during sync but thats not its purpose. If you must know how tomboy-ng does its sync, please look in the source directory, sync.pas Most of the tricky stuff is implemented there, there is no GUI, and there is a reasonably complete text description at the top of the file.

Davo

Found a presentation, slide 22 that appeared to indicate that the Tomboy Sync worked in a different order than I thought. And different from how tomboy-ng's file sync is implemented. This might be a much better way of doing Network Sync and help us understand how the Servers like Rainy might behave. So, filling in the gaps, this looks pretty good. The slide says, and I quote, the sync is like subversion and -

  • There is a running version number for the notes on the server
  • Local copy of notes are like working directory
  • When sync starts Tomboy requests changes since the latest version in its local repo (sadly, further analysis says that this statement is misleading. Perhaps the slide author was simplifying ..)
  • Download changes - apply against local copy (svn up)
  • Upload local changes (svn commit).

So, I postulate that the client asks the server for a list of notes, new since the client's last sync (which the client nominates). The client knows the date of its last sync and its last revision number. The list comes back with (at least) the note ID and the revision number applying to that note. This process then happens -

Flow Chart

(See note below about the "rejoining a sync" problem.) Once completed above, the client needs to

  • Delete notes listed in current local manifest but not in remote one. These have been deleted by other clients.
  • Upload the notes that are not mentioned in its existing local manifest. They have not been seen by the sync system before. Make reference to them in the new local manifest along with the new Rev Number.
  • Then update its local manifest. The local manifest should, at the end of the sync process, list all local notes present in the notes directory along with their actual revision numbers (why ?). It should list no deleted notes.

Issue - get a list, from server, of all notes it has, a copy of remote manifest in other words, and make our own decisions about which ones to check ? The remote server need only fill in last-change-date for notes after nominated revision number so not a massive time penalty. But we need all remote note IDs so we can check what notes have been deleted by other clients.

The server.

The server needs to (apart from initialization and authorization stuff) -

  • Provide a list of all notes in repo, ID and Revision Number and, in the case of notes with a later revision number than the Client's, its last-changed-date.
  • Deliver a requested note.
  • Accept a list of uploaded notes.This will increment revision number.
  • Accept advise that a note (or list of notes) has been deleted locally. This will increment revision number.
  • Maintain, in what ever way it chooses its current rev number.

I do not know if it makes sense to combine all activities that inc rev no or better to let lots of number happen.

The Client

Writes a new local manifest unless nothing happens. By nothing I mean none of -

  • New file down loaded (add that file to list).
  • Advised server of a delete (can then empty delete section).
  • New file uploaded (add that file to main list). A synced client also needs to maintain the local manifest during day to day operations, if a (synced) note is deleted, it must be added to the Deleted section.

The Manifests

  • Firstly, the two manifests are quite different, even using different syntax for similar data.
  • Both store a Server ID, and a revision number. The Local Manifest also has a date of last revision. Obviously the Client Local Manifest can have a revision number equal to or less than the Server. The ID is to ensure you are talking to the right server.
  • The Local Manifest, stored next to the config file, records the ID of locally existing previously synced notes and (in a separate block) the ID of previously synced notes that have now been locally deleted. Its updated at each sync and whenever a previously synced note is deleted in the client. The list of deleted notes is flushed after each sync. The synced note entries also includes the latest revision number (but I am unsure why).
  • The Server or Remote manifest lists all the notes it has in it's 'database'. Along with the most recent revision number. Here the revision number is important, in file sync at least (where revision number corresponds to a directory), it tells us where to look for a file.

The "rejoining a Sync" Problem

I (David) believe there is a problem in the Tomboy model that shows up when you re-join a sync. By definition, if you are joining a sync, your revision number is zero. But even if you do have notes in your local directory with same ID as ones in the remote repo, the server will indicate you should download every note it it has, based on revision numbers. This is how you end up with a lot of duplicate notes.

The Tomboy File sync does not store last-change-dates in the remote manifest, tomboy-ng does. If tomboy-ng encounters a remote manifest without last-change-dates, it can cope, it reads the notes mentioned but having that data in the manifest reduces traffic and is a huge speed improvement.

Further, when joining a repository, or, more specifically, re-joining, if a note with the same ID and Title is found at both ends, tomboy-ng considers the last-change-date stamp. The chance that the notes were edited similtanously (or within 10uSec) is sufficently low and if the last-change-dates match, they are considered identical.

Implementation

At this stage, we identify 5 object layers.

  1. (most of) Application, eg tomboy-ng
  2. The upper part of my sync, the bits that work with 1.), its syncgui in tomboy-ng.
  3. Part of my filesync, the bits that claim to know how to sync. Much of the logic, no file handling.
  4. Part of my filesync, the bit that touch the network. Does local file handling.
  5. eg the part of my filesync but this time that writes to the remote repo, in a network sync, this is the remote server.

So, 4.) and 5.) communicate over the network. When the user switches between file sync and network sync, they from layer 4.) and 5.).

Layer 3, in tomboy-ng does not change when changing between netsync an file sync, Layer 4 does is quite different. In a Network Sync Layer 4 communicates over the network to layer 5, the remote server. In a filesync model, we'll keep the layers clear but obviously all are part of the same binary and source tree. But different pascal Units.

Layer 4 functions made available to (eg) Layer 3.

  • function TLevel4.GetNewNotes(NoteMeta : TNoteMetaList; LocRev : integer) : boolean; // Request a list of new notes since nominated revision with, ideally, last-change-date for each. Parameter is a List of Records,[ID, RevNo, LastChangeDate]
  • function TLevel4.DownloadNotes(const DownLoads : TStringList) : boolean; // Request that all the notes mentioned in the simple list be downloaded and, if necessary, an existing note be moved to Backup.
  • function TLevel4.DeleteNote(const ID : string) : boolean; // Advise server that a note has been deleted (does this trigger a rev ??). Can it be a list of notes ? Does this trigger a new revision.
  • function TLevel4.UploadNotes(const Uploads : TStringList) : boolean; // Push a note, or ideally, a set of notes up to server (triggering a new release).

Layer 4 interaction with Layer 5 Layer 4 will initiate communications, over the network, with the remote server in the case of network Sync. With file sync, it may well write directly to disk ....