Jun 30

One-Day tutorial for Titanium Alloy

You know, to develop mobile apps for IBM Notes and Domino I’m recommending Appcelerator Titanium together with DominoToGo.

Appcelerator Titanium has a pretty cool MVC framework named Alloy. Fokke Zandbergen, a well known Titanium consultant from The Netherlands, published a one-day tutorial for Alloy. Have a look!


Jun 24

Appcelerator Titanium Introduction

Good slides explaining the basics of Appcelerator Titanium. I’m working with Titanium for some years now, and I can highly recommend it. If you want to do mobile apps right, Titanium is a very good option.

And if you want to work with IBM Notes and Domino data in your mobile app, Titanium and DominoToGo is your solution.


Jun 23

Upcoming changes in DominoToGo: Titanium 3.3 support and new documentation

Appcelerator announced that starting with Titanium SKD 3.3 the Ti.include() method will be deprecated. DominoToGo was added to each project with this single line of code:

Ti.include("dtg/dtg_base.js");

So Appcelerator’s move to get rid of Ti.inlcude() means some refactoring work for me. But I see the advantages of the CommonJS model which should be used instead, so I’m not complaining.

On the contrary, I’m embracing the change and using the opportunity to refactor the DominoToGo code in various areas. So in a nutshell: DominoToGo will support Titanium SDK 3.3 and the CommonJS model soon.

And while I’m digging through tons of code, I’m improving inline documentation a lot so that it can be parsed by JSDuck. JSDuck is a very, very nice tool to create documentation from JavaScript code. Have a look at this Titanium SDK documentation. My DominoToGo documentation will look like this, too.

Here is a screenshot:

screenie

 

If you’re in the JavaScript business, I highly recommend having a look at JSDuck in order to document your code. It’s simple, creates results fast and produces beautiful documentation.

 


Jan 07

Speedup Android Simulator for Appcelerator Titanium development

Android development can be very annoying, mostly due to the very sluggish performance of the Android simulator.

There is an add-on driver from Intel called “HAMX” which promised to speed up the simulator, have a look at http://software.intel.com/en-us/articles/intel-hardware-accelerated-execution-manager/.


Sep 17

What a difference an even or odd number makes… or: how to split an UTF8 stream into chunks

Note: this post has been migrated from another blog. Some links may be broken.

Today I got a couple of more gray hairs over an issue I needed some hours to understand:

In a Domino To Go App for iPhone and iPad I have to deal with huge amounts of data (at least from a handheld device perspective). It’s a CRM App, and I need to synchronize 150.000+ contact datasets from IBM Domino to the App using Domino To Go. As you may know, the Domino To Go framework transfers the data as JSON, and for all contacts the JSON data’s size is 60MB.

Previous versions of Domino To Go simply processed all JSON data in memory, but with 60MB of data that is not possible anymore on an iPhone or iPad. It simply provokes out-of-memory errors and the App will crash.
So it was time to teach Domino To Go a new trick: I re-engineered the logic so that the JSON data is saved to a file on the device, and the file is consumed and processed in chunks of 512K.

In general, this is quite easy in Titanium, here is a code snippet:


var xhrData;

var xhrBufferSize = 524288; // 512K

var xhrFile = Titanium.Filesystem.getFile(Ti.Filesystem.getTempDirectory() + "dtg_download.txt");

var xhrDataSize = xhrFile.size;

var xhrBuffer = Ti.createBuffer({

     length : xhrBufferSize
});

var xhrStream = xhrFile.open(Ti.Filesystem.MODE_READ);

var xhrBytesRead = 0;

while (xhrBytesRead >= 0) {

     xhrBytesRead = xhrStream.read(xhrBuffer, 0, xhrBufferSize);

     xhrData = xhrBuffer.toString();

     if (xhrData) {

       // process the data

     }

}

In my case the code is a little more complex, but the general idea is the same. So I was very happy since now I had a way to process nearly any size of JSON data on a handheld or tablet device. And it was relatively fast, too.

But yesterday I found a weird problem: sometimes the xhrBuffer.toString() only returned “undefined” instead of a string! But why?

I examined the JSON data but didn’t find any unusual. No problematic chars like ‘ or ” or , the JSON data was perfectly fine. Nevertheless, I thought the cause has to be some special character somewhere in the big 60MB text file. So, how to find one problematic character in 60 MB of data? A needle in the haystack is an easy find against this task…

I created some code so that I can easily test the data file, then I divided the file in two halfs and tested each of them. The half containing the problem was divided again and so on, until I had the exact position of the problem located. I found that the problem occured with this character: ®
But how can that be? I’m dealing with UTF8 data here, how can that character lead to such a problem? Nevertheless, as I removed the character and replaced it with something else, the problem was solved. Weird.

So I added an automatic replacement of ® to “(R)” in the Domino To Go system and thought the matter was resolved. But the problem remained, there were still some chunks of data that could not be converted to a string by Titanium. What the f….?!?

I tried various approaches to change the way the stream is read, read every piece of documentation about files, streams and buffers in Titanium, researched every corner of the net.. without any luck. The problem remained. Since there is no other one who solved this kind of problem, it was up to me to dig deeper into the problem!

My old friend: the hex editor

As next step I saved every chunk of data into a single file. And as I examined the files, I found that around the position where the problem lies very strange things happened. One chunk of data was repeated three times: the first time it looks fine, the second time it started with one strange character, the third time it started and ended with a strange character. How can THAT happen….?

At this time, I faded out the normal world entirely and had a near direct brain-machine connection established. I was very curios to find the solution, so no call, no voice from the outer world was allowed to come through 🙂

The next step was to get an hex editor. A tool that I didn’t needed for quite some time, but since I’m around in this job for 20+ years I still know what an hex editor is and how to use it… to make a long story short, by looking at the bytes I found that my chunk-reading-logic cracked the two bytes that form one character into two chunks, so that the first byte of the character was at the end of one chunk, and the second byte at the beginning of the next chunk.

I know that in UTF8 one character consists of two bytes, not one byte. But I didn’t thought about that when I decided how big one chunk (that means, the buffer) should be. I simply thought “512K seems to be a good size” and multiplied 1024  by 512. Boy, what a mistake!

When one character consists of two bytes, it means the first character starts at byte #0 and spans byte #1, too. The next character starts at byte #2, the third at byte #4 and so on. Now with an even number of bytes for each chunk, each chunk’s end divides the two bytes of a character, which is obviously a bad thing. You would feel bad if your head would be in one room and the rest of your body in another, too!

Normally this should lead to immediate problems. I thought in my case some additional logic where the end of one chunk was concatenated with the beginning of the next chunk healed the broken characters in most cases.

The solution

After I got this insight, the fix was obvious: use an odd number of bytes for each chunk. So in my case the buffer size now is 524289 instead of 524288. That’s a very easy fix, isn’t it?

…or not?

Indeed, as I changed the buffer size to an odd number of bytes, the problem didn’t occured anymore. But is it really a general fix?

Unfortunately: no, it is not.

In UTF8, ASCII characters are encoded as ONE byte, only non-ASCII characters are encoded in two or even more bytes! So, there is no general rule that and even or odd buffer size avoid this problem. Pretty bad, huh?
It seems that I have to think about this a little bit more. Good ideas are always appreciated 🙂
 

Update I:

I am aware that I could simply encode the JSON data so that it only contains ASCII characters. Then there would be no two-byte characters, and the problem would be solved. But encoding and decoding the data would take time, and I want the whole process to be as fast as possible. So I will try to find a way without encoding first, and will use encoding only as last resort.