Feb 13

Thanks IBM for agent.runWithDocumentContext() in XPages – which I use in DominoToGo mobile solutions

One project I’m currently working on is a mobile version of a Notes based workflow application. The mobile app should enable users to approve or decline workflow documents like vacation requests.

Again, offline availability is important, so a web app is out of the question. Therefore I’m using my winning combination of Appcelerator Titanium and DominoToGo.

Now the challenge is that there is a lot of LotusScript code which handles all the workflow logic in the Notes application. No one wants to re-invent the wheel and implement all the logic again inside the mobile App. There has to be an easier solution.

And for sure, there is: DominoToGo provides a Server Side JavaScript library on the Domino side which allows to hook into several DominoToGo events.

One event is named “custom_write(doc)”. This event allows me to do something with the Notes document that has been uploaded from the mobile app to Domino, for example I can change fields or send a mail.

How does that help with re-using existing LotusScript code for workflow logic? Quite simple: I extracted the LotusScript code (which was stored in a Notes form) to a script library. Then I wrote a simple agent like this:

Use "workflow"

Sub Initialize
	On Error GoTo errorHandler

	Dim session As New NotesSession
	Dim doc As NotesDocument
	Dim action As String

	Set doc = session.Documentcontext
	action = doc.getItemValue("$dtgWorkflowAction")(0)

	Call WflAktion(action, doc)

errorExit:
	Exit Sub
errorHandler:
	Print "mobileWorkflowAction init "+CStr(Erl)+" "+Error$
	Resume errorExit
End Sub

This agent simply calls workflow logic which is stored in the “workflow” script library and runs an action which is identified by the content of the $dtgWorkflowAction field. This field is set in the mobile app by the click on a button.

Now the only thing I needed to do was to start the agent right after the mobile app uploaded Notes document data to Domino. That’s a piece of cake with DominoToGo’s custom_write event:

function custom_write(doc:NotesDocument) {
	var agent:NotesAgent = database.getAgent("mobileWorkflowAction");
	agent.runWithDocumentContext(doc);
	return true;
}

Here is the chain of events:

  1. A user in the mobile app clicks on a button “approve” or “decline”.
  2. The button sets the $dtgWorkflowAction field in the workflow document on the mobile device to either “approve” or “decline” and saves the document on the mobile device.
  3. DominoToGo on the mobile device uploads the change in the $dtgWorkflowAction field from the mobile device to Domino.
  4. DominoToGo on Domino processes the changed field and starts the ServerSide JavaScript custom_write function after that.
  5. The Server Side JavaScript custom_write function gets the changed document as parameter and starts the LotusScript agent “mobileWorkflowAction” with the document as parameter.
  6. The LotusScript agent imports the LotusScript library where all the workflow logic is stored and calls a LotusScript function to start the workflow action.

All of that is only possible because IBM gave us the agent.runWithDocumentContext() method in XPages / ServerSide JavaScript. So a big “thank you!” goes to the engineers that implemented this feature!

 


Dec 16

This is how a modern IBM Domino App looks like!

Rene Richter of notesanwendungen.de, a german Notes application developer, asked me to develop a top modern Web version of his successful classic Notes application “Easy-Support”. Quite a nice assignment! He told me what features to implement, but other than that I was free to develop as I like. So I was able to use various modern web technologies like:

  • Implementation with XPages on Domino 9.
  • Layout and styling with Bootstrap 3.
  • Animations, client business logic etc. implemented with Dojo.
  • No reloading of the page at all! Every change in the UI is done via Ajax, the app feels like a desktop application.
  • A lot of nice and up-to-date features like: visual indicator when some part of the UI is loading, nice visualization of field validation errors, drop-down menus, scalable icons etc.


But you can see it for yourself! The app itself is in german, but I added various video comments in english, enjoy!

Drop me a note if you want to modernize a classic Notes application like this – I can help you with that 🙂


Dec 16

So sieht eine moderne IBM Domino Anwendung aus!

Rene Richter von notesanwendungen.de hat mich gebeten, eine moderne Web-Version von der erfolgreichen Notes-Anwendung “Easy-Support” zu entwickeln. Ein schöner Auftrag! Der zu entwicklende Funktionsumfang war natürlich vorgegeben, aber ansonsten war ich frei in der Umsetzung und konnte voll in die Kiste mit modernen Web-Technologien greifen. Dabei ist folgendes herausgekommen:

  • Die Anwendung ist natürlich mit XPages implementiert.
  • Layout und Styling habe ich mit Bootstrap 3.0 gemacht.
  • Animationen, Client-Logik etc. sind mit Dojo umgesetzt.
  • Es wird nicht neugeladen, jede Änderung der Oberfläche (z.B. beim öffnen eines Dokuments) wird via Ajax gemacht. Die Web-Anwendung fühlt sich daher wie eine Desktop-Anwendung an.
  • Viele nette und zeitgemäße Kleinigkeiten: visueller Indikator, wenn etwas nachgeladen wird, schicke Anzeige von Feldvalidierungsfehlern, DropDown-Menüs, skalierbare Icons etc.

Aber seht selbst!

Möchte noch jemand eine Classic Notes Anwendung derart modernisieren? Ich mach das gerne 🙂


Sep 17

What a difference an even or odd number makes… or: how to split an UTF8 stream into chunks

Note: this post has been migrated from another blog. Some links may be broken.

Today I got a couple of more gray hairs over an issue I needed some hours to understand:

In a Domino To Go App for iPhone and iPad I have to deal with huge amounts of data (at least from a handheld device perspective). It’s a CRM App, and I need to synchronize 150.000+ contact datasets from IBM Domino to the App using Domino To Go. As you may know, the Domino To Go framework transfers the data as JSON, and for all contacts the JSON data’s size is 60MB.

Previous versions of Domino To Go simply processed all JSON data in memory, but with 60MB of data that is not possible anymore on an iPhone or iPad. It simply provokes out-of-memory errors and the App will crash.
So it was time to teach Domino To Go a new trick: I re-engineered the logic so that the JSON data is saved to a file on the device, and the file is consumed and processed in chunks of 512K.

In general, this is quite easy in Titanium, here is a code snippet:


var xhrData;

var xhrBufferSize = 524288; // 512K

var xhrFile = Titanium.Filesystem.getFile(Ti.Filesystem.getTempDirectory() + "dtg_download.txt");

var xhrDataSize = xhrFile.size;

var xhrBuffer = Ti.createBuffer({

     length : xhrBufferSize
});

var xhrStream = xhrFile.open(Ti.Filesystem.MODE_READ);

var xhrBytesRead = 0;

while (xhrBytesRead >= 0) {

     xhrBytesRead = xhrStream.read(xhrBuffer, 0, xhrBufferSize);

     xhrData = xhrBuffer.toString();

     if (xhrData) {

       // process the data

     }

}

In my case the code is a little more complex, but the general idea is the same. So I was very happy since now I had a way to process nearly any size of JSON data on a handheld or tablet device. And it was relatively fast, too.

But yesterday I found a weird problem: sometimes the xhrBuffer.toString() only returned “undefined” instead of a string! But why?

I examined the JSON data but didn’t find any unusual. No problematic chars like ‘ or ” or , the JSON data was perfectly fine. Nevertheless, I thought the cause has to be some special character somewhere in the big 60MB text file. So, how to find one problematic character in 60 MB of data? A needle in the haystack is an easy find against this task…

I created some code so that I can easily test the data file, then I divided the file in two halfs and tested each of them. The half containing the problem was divided again and so on, until I had the exact position of the problem located. I found that the problem occured with this character: ®
But how can that be? I’m dealing with UTF8 data here, how can that character lead to such a problem? Nevertheless, as I removed the character and replaced it with something else, the problem was solved. Weird.

So I added an automatic replacement of ® to “(R)” in the Domino To Go system and thought the matter was resolved. But the problem remained, there were still some chunks of data that could not be converted to a string by Titanium. What the f….?!?

I tried various approaches to change the way the stream is read, read every piece of documentation about files, streams and buffers in Titanium, researched every corner of the net.. without any luck. The problem remained. Since there is no other one who solved this kind of problem, it was up to me to dig deeper into the problem!

My old friend: the hex editor

As next step I saved every chunk of data into a single file. And as I examined the files, I found that around the position where the problem lies very strange things happened. One chunk of data was repeated three times: the first time it looks fine, the second time it started with one strange character, the third time it started and ended with a strange character. How can THAT happen….?

At this time, I faded out the normal world entirely and had a near direct brain-machine connection established. I was very curios to find the solution, so no call, no voice from the outer world was allowed to come through 🙂

The next step was to get an hex editor. A tool that I didn’t needed for quite some time, but since I’m around in this job for 20+ years I still know what an hex editor is and how to use it… to make a long story short, by looking at the bytes I found that my chunk-reading-logic cracked the two bytes that form one character into two chunks, so that the first byte of the character was at the end of one chunk, and the second byte at the beginning of the next chunk.

I know that in UTF8 one character consists of two bytes, not one byte. But I didn’t thought about that when I decided how big one chunk (that means, the buffer) should be. I simply thought “512K seems to be a good size” and multiplied 1024  by 512. Boy, what a mistake!

When one character consists of two bytes, it means the first character starts at byte #0 and spans byte #1, too. The next character starts at byte #2, the third at byte #4 and so on. Now with an even number of bytes for each chunk, each chunk’s end divides the two bytes of a character, which is obviously a bad thing. You would feel bad if your head would be in one room and the rest of your body in another, too!

Normally this should lead to immediate problems. I thought in my case some additional logic where the end of one chunk was concatenated with the beginning of the next chunk healed the broken characters in most cases.

The solution

After I got this insight, the fix was obvious: use an odd number of bytes for each chunk. So in my case the buffer size now is 524289 instead of 524288. That’s a very easy fix, isn’t it?

…or not?

Indeed, as I changed the buffer size to an odd number of bytes, the problem didn’t occured anymore. But is it really a general fix?

Unfortunately: no, it is not.

In UTF8, ASCII characters are encoded as ONE byte, only non-ASCII characters are encoded in two or even more bytes! So, there is no general rule that and even or odd buffer size avoid this problem. Pretty bad, huh?
It seems that I have to think about this a little bit more. Good ideas are always appreciated 🙂
 

Update I:

I am aware that I could simply encode the JSON data so that it only contains ASCII characters. Then there would be no two-byte characters, and the problem would be solved. But encoding and decoding the data would take time, and I want the whole process to be as fast as possible. So I will try to find a way without encoding first, and will use encoding only as last resort.


Aug 20

1600% faster – building JSON data out of a view with 75.000 documents in 26 seconds instead of 4 minutes

Note: this post has been migrated from another blog. Some links may be broken.

In Domino To Go we are using an XPage to build JSON data out of a Domino view, so that a mobile device can download the data and process it locally.

Running through a Domino view and building this JSON data can be a time consuming task, or it can be quite fast – depending on how the code is implemented, what classes are used and so on. Yesterday I did some more tests using a standard address book with 75.000 contacts.
The XPage that runs through the Domino view and builds the JSON data already works with the view index only, that means uses a NotesViewNavigator, NotesViewNavigator.getFirstEntry() and NotesViewNavigator.getNextEntry() to iterate over the view.
It sets NotesView.autoUpdate to false and uses a java.lang.StringBuilder to concatenate single strings so that it becomes a full set of JSON data.

So, it was not that bad after all. Still, on a test machine (Lenovo Thinkpad with Domino 9) it took nearly 4 minutes to run through all 75.000 contacts, compute JSON data and deliver it to the browser (or mobile device).

So, there is room for improvement, isn’t it?

I got the time down from 4 minutes to 26 18 seconds, which still is not the end of the optimization. Here is what I did:

1.) Check for unnecessary calls to Java or Notes methods, especially in loops

In the code, I needed to use NotesViewEntry.getColumnValues() in order to get the values of a view entry, the number of columns and so on. There were multiple NotesViewEntry.getColumnValues() calls in the code, for example:


var size = viewentry.getColumnValues().size();

for (j = 0; j < size; j++) {

var v = viewentry.getColumnValues().elementAt(j);

...

}

So, depening on the number of columns, there were a lot of getColumnValues() calls.
It’s faster this way:


var colValues:java.util.Vector = viewentry.getColumnValues();

var size = colValues.size();

for (j = 0; j < size; j++) {

var v = colValues.elementAt(j);

...

}

2.) Avoid loops if possible

I had this code:


var v = colValues.elementAt(j);

for (k = 0; k < v.size(); k ++) {

s += v.elementAt(k)+(k < (v.size()-1) ? "##YNSEP##" : "")

}

There are two issues with this code: first, it’s very bad to get the size of the vector on every iteration of the loop, even twice in this example.
This would be quite faster:


var v = colValues.elementAt(j);

var vSize = v.size();

for (k = 0; k < vSize; k ++) {

s += v.elementAt(k)+(k < (vSize-1) ? "##YNSEP##" : "")

}

Second, I asked myself what does this code do in the first place? It’s a simple join operation, all elements of the vector are joined together to a string. It’s expensive in several ways:

– a loop is always expensive
– multiple calls to methods of the vector

– multiple concatenations using a simple string object

But why coding a loop of my own when there is an build in method for that? What about this code:


var v = colValues.elementAt(j);

s = v.toArray().join("##YNSEP##");

It’s much simpler to read, and the internal method of converting the vector to an array and using the internal join seems to be much faster than my own logic.

3.) Avoid string concatenations

Stuff like this is expensive:


s = "";

s += "some string";

s += "some other string";

It’s not important when doing this only a couple of times. But concatenating strings this way a hundreds of thousands times needs more time than using a class that has been  optimized for such operations, such as java.lang.StringBuilder.
My code already used a java.lang.StringBuilder, but still there were several occasions where a simple string concatenation was used. I removed them all, so that only the java.lang.StringBuilder is used now.

4.) Use NotesViewNavigator.setBufferMaxEntries(400)

Setting a buffer before running through the view is simply faster. This method is available since Domino 8.5.3, and now I enabled it. If a customer still has Domino 8.5.2, he can disable it again.

Result of these changes so far: time was down from 4 minutes to under one minute.

That was not bad, but I wondered how expensive the usage of Server Side JavaScript is compared to pure Java. So I did one other change:

5.) Use pure Java instead of Server Side JavaScript

In created a simple static Java class like this:


public class ReadView {

  private ReadView() {
         
  }

 
  public static void doWork(com.ibm.xsp.domino.context.DominoFacesContext facesContext, com.ibm.xsp.designer.context.ServletXSPContext context, Session session, java.util.AbstractMap applicationScope) {

  }
}

In the doWork() method I placed the code I formerly used in Server Side JavaScript and migrated it to pure Java.
The afterRenderResponse event in the XPage now looks quite clean:


com.dominotogo.ReadView.doWork(facesContext, context, session, applicationScope);

Result of using pure Java instead of Server Side JavaScript: the time went down again notably, from about 50 seconds to 26 seconds. The time saving will be even higher with bigger datasets. That’s simply because JavaScript is interpreted and all the conversion work between Server Side JavaScript and Java is not needed anymore.
Using Java instead of SSJS is not worth the effort when your code runs fast anyway, but if you want to optimize, it’s a good option.

There are some more small chances for code improvement in my Java class, but I believe the only possibility to get another notable speedup would be implementing the code in pure C.

But so far, I’m quite satisfied with this improvement 🙂

Update: my XPage compared to Domino’s own JSON output

As you may know, Domino is able to deliver JSON data from a view out of the box, using the ReadViewEntries&outputformat=JSON URL.

In a quick test I was only able to get 9464 entries out of Domino, although I defined “0” as limit for view rows in the server document.
Therefore I compared Domino’s JSON output to my own based on 9464 view entries:

Domino’s build in JSON output

my own JSON output

time to compute and deliver the data to the browser

6 seconds

3.8 seconds

size of the data

6.03MB

2.46MB

It seems that my code is not only faster, it produces much less data. Since my JSON output is downloaded by mobile devices, size is very important.

2nd Update: time is down to 18 seconds

Karsten Lehmann gave the hint to save more time when writing directly to the output stream and skipping the StringBuilder stuff. I need the StringBuilder when the JSON data should be cached, nevertheless I tried it – and indeed, it saved additional 8 seconds.
So I will use the StringBuilder only if the result should be cached, otherwise I will write directly to the output stream. Thanks, Karsten!


Jun 04

Developing a top modern web application with XPages is so freaking cool.

Note: this post has been migrated from another blog. Some links may be broken.

I’m working on a new mail-management application for an enterprise customer, and I got the freedom to do it as web application with IBM XPages (although the customer normally uses Notes client  applications). There are no compatibility or design restrictions, I’m free to implement the way I want.

And so I took the opportunity to use three techniques that I only used in experimental projects so far:

  • A UI based on Bootstrap.
  • Single-Page approach where every page is loaded via partial updates so that the browser never needs to reload the entire page.
  • Keyboard navigation (for example, allow closing a preview window with ESC key).


and beside that, I use various fancy stuff I already have in my toolbox, like:

  • A visual indicator that shows when a partial update is running, including changing the cursor to a wait-cursor and blocking any user interaction until the update is completed.
  • A technique to do multiple partial updates at once.
  • Communication from Client Side JavaScript to Server Side JavaScript, for example to store the UNID of a mail selected in a custom-designed view into a sessionScope variable.
  • A keep-alive technique to keep the session open when the user doesn’t do any action for some time (similar to the “Keep Session Alive” control of the ExtLib).
  • Custom designed views, based on repeat controls, using standard Bootstrap CSS techniques.
  • A lot of Dojo, including charts.
  • Some Java backend stuff, for example PDF generation.


Guess what? Developing this way is really fun and quite productive!


I know, all the stuff needed to do serious XPages development is hard to learn if you don’t know HTML, CSS, JavaScript, Dojo et al yet. But if you did learn all of that, you can develop top modern apps and have fun on the way.


Jun 03

Quick tipp: forEach function in XPages Server Side JavaScript

Note: this post has been migrated from another blog. Some links may be broken.

If you’re a XPages developer, chances are good that you already know the dojo.forEach() function. In that case, I’m pretty sure you miss this functionality in SSJS, don’t you?

I added a jbForEach(array, function) function to my SSJS toolbox, and furthermore, I added a jbForEachDocument(unidarray, function) function, too. Here is the code:



function
jbForEach(a, f) {
       
if (!a || !a.length) return;
       
for (var i = 0; i < a.length; i++) {
               f(a[i]);

       }        

}


function
jbForEachDocument(unids, f, db:NotesDatabase) {
       
if (!unids || !unids.length) return;
       
for (var i = 0; i < unids.length; i++) {
               
var doc:NotesDocument = (db ? db.getDocumentByUNID(unids[i]) : database.getDocumentByUNID(unids[i]));
               
if (doc && doc.isValid()) {
                    f(doc);
                    doc.recycle();
                }
       }        

}



UPDATE 4th June: I added a recycle() call as David suggested.

You can use the jbForEachDocument function as follows:



var unids = ... // an array of UniversalIDs

jbForEachDocument(unids, function(doc) {

 doc.replaceItemValue("myfield", "somevalue");

 doc.save();

});



If you want to use documents from another database, you can give the database object as third parameter like this:



var unids = ... // an array of UniversalIDs

var db = session.getDatabase("", "somedb.nsf");

jbForEachDocument(unids, function(doc) {

 doc.replaceItemValue("myfield", "somevalue");

 doc.save();

}, db);



Using this kind of function spares the effort to do a for…next loop on your own and makes your code more readable.


May 28

New content on xpageswiki.com

Note: this post has been migrated from another blog. Some links may be broken.

Some new stuff on xpageswiki.com:

28. May 2013: Set startKeys property of a ViewPanel with SSJS
28. May 2013:
About multithreading in XPages
4. Apr. 2013:
Using Twitter Bootstrap CSS Framework
4. Apr. 2013:
Info about new xsp.properties in Domino 9 added
25. Mar 2013:
Getting the referer for the current page
3. Mar 2013:
Referencing Dojo Root Directory
18. Feb 2013:
Relative URLs to another NSF

Happy coding!


Apr 02

Domino 9 XPages problem: Date Time Picker always has a default value

Note: this post has been migrated from another blog. Some links may be broken.

I want a Date Time Picker control WITHOUT a default value. Doesn’t seem to be possible anymore since Domino 9.0 🙁
To reproduce, create a blank XPage and place a Date Time Picker control. Open the XPage in the browser and you will see that it defaults to today.


I didn’t found any way to set the default to an empty value. I tried setting all properties/data/default to 0, null, empty string and so on – no luck.

I tried the data-dojo-probs attribute with value:”, this sets the default to 1970-1-1, but not to blank.


That worked in Domino 9 beta with Dojo 1.7, so it seems to be an issue with Dojo 1.8.1.


Any ideas?


Mar 26

CSS media queries in XPages for Internet Explorer 8 (responsive web design)

Note: this post has been migrated from another blog. Some links may be broken.

As you should know, you can adjust your XPages based application to the size of the browser window with CSS media queries. See here for a small summary about CSS media queries in xpageswiki.com.

Unfortunately, Internet Explorer only supports CSS media queries in version 9 and up. But, there is a very simple solution to that: you can simply include a JavaScript library in your page (at the bottom), which emulates CSS media queries for Internet Explorer 8 (and maybe 7, too, I didn’t tested that). Very cool.

I used it as follows:

1.) Download css3-mediaqueries-js

2.) Import as image resource in Domino Designer

3) Include this code at the end of your XPage: