How to sync realm with own back end - a1k89/Blog GitHub Wiki

Prepare

  • iOS application (swift) with realm (or android)
  • Django with PostgreSQL
  • We want to provide full offline work (now it's work by default) and synchronize offline data through all account devices

Problems

  • Which algorithm?
  • How to resolve:
    • We have 2 devices for current user
    • Device A work offline
    • Device B work offline
    • Both devices got online and start synchronization
  • How to sync media? (photo/audio/video/files)? iCloud? Base64? Upload to our media server?

Thinking about

  • We must to split our work:
    • Realm-part (swift)
    • Back-end part (python/Django)
  • Server: is our single place of truth
  • Conflicts: last writer wins
  • We don't delete any instance. We set is_delete flag only

Thinking about mobile part

Plan A:
  1. Add to all models field is_synced (bool).
  2. When created: is_synced=false
  3. When we change model set is_synced=false
  4. Run service and collect all no-synced models and send it to server
  5. Get response from server and update our models. Set is_synced=true.
  6. We must to update all models or create new (if not exist in our database)
  7. We must provide full background async task
  8. For full control send to server model-by-model. Send one - update. Send the next one - update
Plan B:
  1. We create separate model SyncModel
  2. We write a service which observe all models
  3. When something happen (update/create) we write on SyncModel: modelType + identifier
  4. Create some trigger(?) to run synchronization our SyncModel
  5. For full control: send to server model-by-model
  6. Run service on every application start
Plan C (Plan B mixin):
  1. Create SyncService
  2. Register models to observe in SyncService
  3. When something change in model (create/update) we send our model instance to server immediately
  4. If has error or something else mark our model in SyncModel (from Plan B) and synchronize SyncModel in the future

Thinking about server side

Plan A:
  1. Create models 1:1 as realm
  2. Create SyncService
  3. Create SyncSerializer (entry point and first serialization)
  4. client data -> SyncSerializer -> SyncService -> get uid, modify_at, model, data -> compare modify_at -> update/ignore -> 200 response

Plan B:

  1. Create single SyncModel: owner, created_at, json_data, model_name, uid
  2. Write a SyncService
  3. Got a data to sync
  4. Got owner, model_name and created_at. Got body (our data)
  5. Find a row: owner + model_name + uid
  6. If row exist check created_at field and follow point4 from Plan A
  7. If not exist create new row
  8. In this we not thinking about relations or another field. We simple create/update rows

Realization

Swift (realm)
  1. Create Uploadable protocol and some helpers:
protocol Uploadable: Codable, Mappable {
    var resourseType: String { get }
    var uid: String { get }
}

typealias Syncable = Object & Uploadable
typealias SyncableMappable = Object & Uploadable & Mappable

struct UpdateStruct {
    let identifiers: [String]
    let resourceTypes:[String]
}
  1. Create extension for Uploadable. Use notification system from realm. Static func is a class method to register models and check all updates. Closure of this method return type of model and identifier.
extension Uploadable where Self: Object {
    static func registerNotificationObserver(for realm: Realm,
                                             callback: @escaping(UpdateStruct) -> Void) -> NotificationToken{
    
        let objects = realm.objects(self)
        let notificationToken = objects.observe { changes in
            switch changes {
                case .update(let collections, _, let insertions, let modifications):
                    if collections.isEmpty { return }
                    
                    var response = insertions.map{collections[$0]}
                        response += modifications.map{collections[$0]}
                                    
                    let prepare = UpdateStruct(identifiers:  response.map{$0.uid},
                                               resourceTypes: response.map{$0.resourseType})
                    

                    callback(prepare)
                    
                    break

                default:
                    break
            }
        }
        return notificationToken
    }
}
  1. Create SyncService:
import Foundation
import RealmSwift
import ObjectMapper
import AlamofireObjectMapper
import Alamofire


final class SyncService:NSObject, ApplicationService {
    static let shared: SyncService = SyncService()

    private override init() {}
    
    var tokens: [NotificationToken] = [NotificationToken]()
    
    var modelTypes:[Syncable.Type] = [
        User.self
    ]
    
}

extension SyncService {
    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey : Any]? = nil) -> Bool {
        return true
    }
}


extension SyncService {
    func register() {
        let realm = try! Realm()
        tokens = modelTypes.compactMap{$0.registerNotificationObserver(for: realm) { updates in
            print("updates: ", updates)
        }}
    }
    
}

Now if we change User model we will see in console type of model and identifier. Good.

  1. The next step: after got a Updates structure try to send changes to server (through API). If has error we will save Updates structure in our cache SyncQueueModel. We save in one string: <userUID>_<modelName>_<instanceUID> 
protocol SyncQueueModelProtocol {
    var compoundKey: String { get }
    var createdAt: Date { get }
}

class SyncQueueModel: Object, SyncQueueModelProtocol {
    @objc dynamic var compoundKey:String  = ""
    @objc dynamic var createdAt: Date = Date()
    
    override static func primaryKey() -> String? {
        return "compoundKey"
    }
    
    func configure(userUID:String, modelType: String, identifier: String, data:Data){
        let key = "\(userUID)_\(modelType)_\(identifier)"
        self.compoundKey = key
    }
}

Backend

Rules:

  • All models must have: owner, uid, modify_at fields

Mark: As example use User model

  1. Create app data. Add models.py
  2. Add to models.py:
from django.db import models
from django.contrib.auth import get_user_model

class User(get_user_model()):
    class Meta:
        proxy = True

    @property
    def owner(self):
        return self
  1. Create serializer classes
serializers.py

class UserSerializer(ModelSerializer):
    name = serializers.CharField(source='first_name')
    jwt_token = serializers.CharField(read_only=True)
    modify_at = TimestampField()

    class Meta:
        model = User
        fields = [
            'uid',
            'name',
            'email',
            'modify_at',
            'jwt_token',]

        read_only_fields = (
            'uid',
            'jwt_token',
        )
  1. Create entry point:
serializers.py

class SyncSerializer(serializers.Serializer):
    uid = serializers.UUIDField()
    model = serializers.CharField()
    modify_at = TimestampField()
    data = serializers.JSONField()
  1. Create SyncService:
import pytz

from django.apps import apps
from django.db import transaction
from django.contrib.auth import get_user_model

from rest_framework.exceptions import PermissionDenied

from api.v1.exceptions import ServiceException

User = get_user_model()


class SyncService:
    def __init__(self, request, model, uid, modify_at, data):
        app_label, model_name = model.split('.')

        self.request = request
        self.identifier = uid
        self.modify_at = modify_at
        self.model = apps.get_model(app_label=app_label,
                                    model_name=model_name)
        self.data = data

    @classmethod
    def execute(cls, *args, **kwargs):
        instance = cls(*args, **kwargs)

        with transaction.atomic():
            return instance.sync()

    def get_serializer_class(self):
        return self.model.get_serializer()

    def get_serializer(self, *args, **kwargs):
        serializer_class = self.get_serializer_class()

        return serializer_class(*args, **kwargs)

    def check_object_permissions(self, instance):
        if self.request.user != instance.owner:
            raise PermissionDenied()

    def is_newest(self, instance):
        """
        Check incoming and instance datetime. Both must be use utc

        """
        if instance is None:
            return True

        if self.modify_at.tzinfo != pytz.utc:
            raise ServiceException('Date is not utc')

        return self.modify_at >= instance.modify_at

    def get_instance(self):
        kwargs = {'uid': self.identifier,
                  'owner': self.request.user}

        if issubclass(self.model, User):
            kwargs.pop('owner')

        instance = self.model.objects\
            .filter(**kwargs)\
            .first()

        return instance

    def sync(self):
        instance = self.get_instance()
        self.check_object_permissions(instance)
        serializer = self.get_serializer(instance=instance,
                                         data=self.data)

        if serializer.is_valid():
            if self.is_newest(instance):
                serializer.save()

            return serializer.data

        raise ServiceException(serializer.errors)
  1. Create APIView view:
api.v1.sync.py:

from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status

from api.v1.serializers import SyncSerializer
from apps.data.services.sync import SyncService


class SyncGenericApiView(APIView):
    def post(self, request, *args, **kwargs):
        serializer = SyncSerializer(data=request.data)

        if serializer.is_valid():
            validated_data = serializer.validated_data
            validated_data['request'] = self.request
            result = SyncService.execute(**validated_data)

            return Response(result)

        return Response(serializer.errors,
                        status.HTTP_400_BAD_REQUEST)
  1. Add endpoint url:
urls.py

urlpatterns += [
    path('sync/', SyncGenericApiView.as_view())
]

And this is all. Now we have single class to synchronize our incoming data to models.

⚠️ **GitHub.com Fallback** ⚠️