1000% faster than Notes Replicator: Synchronisation of Notes data to iPhone, iPad, Android, Blackberry with Domino To Go

Note: this post has been migrated from another blog. Some links may be broken.

I nearly finished all those performance improvements that Domino To Go will get. The goal for Domino To Go 2.0 is to be able to synchronisation a lot of data to a mobile device such as the iPhone or iPad.

My test suite consist of more than 210.000 contact datasets in a Notes CRM application. This contact data should be synchronized to an iOS app, so that all 210.000 contacts can be used when the device is offline. And, furthermore, the synchronisation should run as fast as possible!.

There were a lot of interesting challenges on the way. For example, I needed to speed up building JSON data on the Domino server dramatically, see 1600% faster – building JSON data out of a view with 75.000 documents in 26 seconds instead of 4 minutes for further reading.

Then there were even harder challenges on the mobile device side. A device like the iPhone has limited ressources. For my testcase, the JSON data’s size that has to be processed on the mobile device is more than 60MB (but only about 6MB during the download thanks to Domino’s build-in GZIP compression for HTTP transfers).
Previous versions of Domino To Go simply stored the data in memory for processing – with more than 60MB of data, that is out of the question.  

So I needed to re-engineer the logic so that the data is downloaded to a file on the mobile device. Then the file is read in chunks, so that only some KB of data is in the device’s memory at a time. Since the data was in UTF8 charset, that lead to problem’s of it’s own (which are solved by now, by the way).

The next step was to speed up processing the JSON data on the mobile device – being able to process megabytes of data is great, but what it is good for when it takes ages? I learned a lot about what kind of JavaScript code is faster than other, and how to get the maximum performance out of the local SQLite database.

Finally I needed to make sure that the generation of JSON data on the Domino server does not take too much memory, and has no memory leaks. I just finished that optimization today.

As of today I think all major problems are solved and so far, the code seems to be stable, although a lot more testing will happen during the next weeks. But so far, I’m very satisified with the outcome!

Numbers regarding the performance

Test suite: synchronizing two views, each with 10 columns or so. The first view displays about 150.000 documents, the seconds view about 60.000 documents.
So the Domino To Go on the server needs to run through 210.000 documents to generate JSON data for the mobile device.

The mobile device needs to download this JSON data, store if locally and process it. During processing, the data is split into chunks, interpreted as JSON and each JSON data item is unescaped and stored in the local SQLite database.

Here is how long the Notes replicator needs to replicate the data from the server to local:

The number changes during the replication, but alltogether it’s save to assume that Notes needs about 50 minutes to replicate the data.

And here is the performance of Domino To Go:
(measured on an iPhone 4S)

a) Generating the JSON data on a Domino server running on a small laptop (!): 40 seconds for the first view, about 10 seconds for the second view. Alltogether: about 50 seconds.
(so this will be much faster on good server hardware!)

b) Downloading the data from Domino to the device: about 4 seconds (remember, the JSON data is compressed during the transfer).

c) Processing of JSON data for the first view on the mobile device: 216 seconds

d) Processing of JSON data for thge second view on the mobiel device: 59 seconds

All together: 329 seconds or 5.5 minutes.

OK, it’s only about 820% faster than Notes, but nevertheless, I’m satisified with the result 🙂

Note: the comparison is not entirely exact because the Notes replicator synchronizes full Notes documents, while in my test Domino To Go synchronizes the content of two views. Nevertheless, the documents do not contain much more data than displayed in the views, so this comparison is a good indicator for how fast Domino To Go is.

Comments are closed.