Jun 04

‘Field is too large (32k) or view’s column and selection formulas too large’ problem – finally solved!

The internet is full of questions like “I get the message ‘Field is too large (32k) or view’s column and selection formulas too large’ – how do I find and delete the problematic document?”. And there are several attempts to solve this problem, most of them without much chance or quite complicated.

Now I just read the changelog for the new Ytria ScanEZ 12.1 and found this: New scanning method allows easy management for ’32k error’ afflicted documents

I just tested this against a big database of my own and indeed found two documents with summary data of more than 32k. Very nice. Thanks Ytria!

Dec 23

Experiences from the first days after migrating from Domino Blog to WordPress

Some days has passed since I migrated this blog from a NSF based Domino Blog to WordPress. Some observations:

  • I got spam comments incredibly fast. Mostly trash in chinese. Solution: a plugin named “captcha”, which adds an easy to solve math question to the comment form. Simple and effective.
  • Bruce notified me about some CSS glitches I missed. Since my WordPress theme allows me to add custom CSS, I could fix all of them easily. Thanks, Bruce.
  • After some comments were written, I wondered myself if there is some way to let my readers subscribe to a comment thread. Found the plugin “Subscribe to Comments Reloaded” which adds an “subscribe” checkbox to the comments form. I even could adjust the text of the notification mails.
  • I wanted some “share” buttons below each post – again: quick search for a plugin, tried some and installed the “Canjo” plugin.
  • Tested my blog on iPhone and iPad – looks good.
  • Did some searches on Google to get deep links to my old blog, for example “domino to go performance“. The URL redirection works perfectly.
  • I like the autosave feature in the WordPress editor.
  • The WordPress App for iOS is great.

One open issue is how to migrate comments from the old blog. I did some research and found that there is no easy option. I have to look in the SQL tables in WordPress’ database.

So far, I’m quite happy with the migration. You know the saying “there is an app for that” for the iOS platform. It’s just like that with WordPress: “there is a plugin for that”.

Dec 13

I’m surprised over and over again what kind of cool stuff is possible with Notes these days

A customer has a custom helpdesk / ticket system, which is quite complex and developed entirely with classic Notes elements. Beside these tickets there is a lot of additional data, like hardware devices, customer records, a document management database, purchase orders etc.
So far all this data is displayed in the ticket form using embedded views. This works, but is slow and this technique is not able to display all kind of documents belonging to a ticket in one list.

The customer asked me “can you create a widget for the sidebar which displays a list of all documents belonging to the ticket which is currently selected in the view?” – “sure, should be no problem” was my response 🙂

Some may remember that I worked with Composite Applications some time ago with Notes 8, which was not the best experience… a lot of stuff didn’t work that time, and I got a decent amount of additional gray hair.
Since the easiest way to develop such a widget is using an XPage inside a Composite Application, I had to revisit this technology again. And I have to admit, it works very well in Notes 9!

So far I have an XPage which display some generic data about a ticket and a list of all documents from different databases belonging to this ticket. The list of documents is computed by another XPage and loaded via Ajax (dojo.xhrGet), so that the computation of the data does not hinder the widget from being displayed.
I created a Composite Application which simply displays the start frameset of the ticketing database and loads the XPage into the sidebar. Then I told some views to push the note id of the selected document via property broker to the Composite Application, which in turn pushes that note id to the XPage.

Now when the ticketing database is being opened in Notes, the XPage widget is automatically loaded into the sidebar and when the user selects a ticket in the view, the widget is refreshed automatically so that the appropriate data can be computed and displayed. And when the ticketing database is being closed, the sidebar widget is being closed automatically, too.

I will write more about the exact steps I took, but so far I’d simply like say: using an XPage as sidebar widget works just perfect and I’m very satisifed with the outcome so far.

Sep 20

1000% faster than Notes Replicator: Synchronisation of Notes data to iPhone, iPad, Android, Blackberry with Domino To Go

Note: this post has been migrated from another blog. Some links may be broken.

I nearly finished all those performance improvements that Domino To Go will get. The goal for Domino To Go 2.0 is to be able to synchronisation a lot of data to a mobile device such as the iPhone or iPad.

My test suite consist of more than 210.000 contact datasets in a Notes CRM application. This contact data should be synchronized to an iOS app, so that all 210.000 contacts can be used when the device is offline. And, furthermore, the synchronisation should run as fast as possible!.

There were a lot of interesting challenges on the way. For example, I needed to speed up building JSON data on the Domino server dramatically, see 1600% faster – building JSON data out of a view with 75.000 documents in 26 seconds instead of 4 minutes for further reading.

Then there were even harder challenges on the mobile device side. A device like the iPhone has limited ressources. For my testcase, the JSON data’s size that has to be processed on the mobile device is more than 60MB (but only about 6MB during the download thanks to Domino’s build-in GZIP compression for HTTP transfers).
Previous versions of Domino To Go simply stored the data in memory for processing – with more than 60MB of data, that is out of the question.  

So I needed to re-engineer the logic so that the data is downloaded to a file on the mobile device. Then the file is read in chunks, so that only some KB of data is in the device’s memory at a time. Since the data was in UTF8 charset, that lead to problem’s of it’s own (which are solved by now, by the way).

The next step was to speed up processing the JSON data on the mobile device – being able to process megabytes of data is great, but what it is good for when it takes ages? I learned a lot about what kind of JavaScript code is faster than other, and how to get the maximum performance out of the local SQLite database.

Finally I needed to make sure that the generation of JSON data on the Domino server does not take too much memory, and has no memory leaks. I just finished that optimization today.

As of today I think all major problems are solved and so far, the code seems to be stable, although a lot more testing will happen during the next weeks. But so far, I’m very satisified with the outcome!

Numbers regarding the performance

Test suite: synchronizing two views, each with 10 columns or so. The first view displays about 150.000 documents, the seconds view about 60.000 documents.
So the Domino To Go on the server needs to run through 210.000 documents to generate JSON data for the mobile device.

The mobile device needs to download this JSON data, store if locally and process it. During processing, the data is split into chunks, interpreted as JSON and each JSON data item is unescaped and stored in the local SQLite database.

Here is how long the Notes replicator needs to replicate the data from the server to local:

The number changes during the replication, but alltogether it’s save to assume that Notes needs about 50 minutes to replicate the data.

And here is the performance of Domino To Go:
(measured on an iPhone 4S)

a) Generating the JSON data on a Domino server running on a small laptop (!): 40 seconds for the first view, about 10 seconds for the second view. Alltogether: about 50 seconds.
(so this will be much faster on good server hardware!)

b) Downloading the data from Domino to the device: about 4 seconds (remember, the JSON data is compressed during the transfer).

c) Processing of JSON data for the first view on the mobile device: 216 seconds

d) Processing of JSON data for thge second view on the mobiel device: 59 seconds

All together: 329 seconds or 5.5 minutes.

OK, it’s only about 820% faster than Notes, but nevertheless, I’m satisified with the result 🙂

Note: the comparison is not entirely exact because the Notes replicator synchronizes full Notes documents, while in my test Domino To Go synchronizes the content of two views. Nevertheless, the documents do not contain much more data than displayed in the views, so this comparison is a good indicator for how fast Domino To Go is.

Dec 12

IBM Notes 9 Beta – Praise the lord!

Note: this post has been migrated from another blog. Some links may be broken.

I just reviewed new features coming in the public beta of Notes 9. And until the beta is released, I cannot talk about what I just read. But praise the lord, one very small, but quite important feature (at least for me) has made it into Notes 9.
Hint: think of what every other software can do as an alternative to CTRL+V.

Thank you, IBM.

Nov 03

How Notes could escape the “Notes vs. Exchange” war while staying in the companies

Note: this post has been migrated from another blog. Some links may be broken.

I just talked with a collegue about a customer of ours (some thousand seats). It’s a Notes shop, where it’s planned to implement SAP and Exchange. It’s quite clear that SAP and Exchange will come in the near future for over a year now, nevertheless, the employees are using the features of Notes and our own YouAtNotes CRM software more and more.

In fact none of these employees can imagine how SAP could ever implement all the processes they solve with Notes and YouAtNotes CRM without investing very, very, very much money for custom development.
Therefore it’s foreseeable that it will end like in many other companies: SAP comes, Exchange comes, and Notes stays for the applications.

And now think about this: what if Notes could simply connect to other mail and calendar (for example, Exchange) services as well, just as it does with Domino? What if you have Notes as you know it, but the mailfile and calendar view is filled with data from Exchange?

All existing Notes applications could continue to work.
Even the ones with deep mailfile intergration. The IT could put some Exchange serves in place, while the Domino application servers stay where they are. Management would be happy because they “moved to Exchange”. Nothing would change for IT, since Domino stays there anyway because Microsoft cannot migrate the Notes apps.
And the users would be very happy because they can continue working productively, just as they did before.

And it could be even possible to invest in new Domino/XPages applications, because they run in the client the users already have (Notes) while Management still have the satisfaction of having “moved to Exchange”.

Boy, that would be a smart move by IBM.

Apr 07

The name of Lotus Notes Next

Note: this post has been migrated from another blog. Some links may be broken.

You see that the brand “Lotus” disappears from one product after the other. Last example is “Lotus Connections” which now is named “IBM Connections”.

That brings up the question: how will the next Notes release will be named?
What do you think and what do you would like?

Personally, I’m not quite sure yet. I think removing the brand “Lotus” is a good thing since “Lotus” is a “burned brand” here in Germany, and I would assume in other countries, too. There are many people who simply think that “Lotus Notes” is the pure evil, based on their impression of a Lotus Notes R4 or R5 client.

Rebranding the product could be helpful while dealing with those kind of people.

Question is: is it enough to name it “IBM Notes”? Or does “Notes” have to disappear, too?
Something like “IBM Collaboration Client”, “IBM Groupware Client” or so?

What do you think?

Jan 03

Kickoff: what’s to be expected with Lotus Notes and Domino in 2011?

Note: this post has been migrated from another blog. Some links may be broken.

First of all, happy new year to you all. And much fun at Lotusphere, I’ll miss it – be sure to visit my collegues Walfred and Michael at the YouAtNotes booth in the exhibition hall!

So, a new year, a fresh start – what do I expect for Lotus Notes and Domino in 2011? What are my wishes? Let’s see:

First the easy part
: XPages. This technology is getting more and more important, and that’s even more true in 2011. Any Notes developer who does not invest into XPages knowledge will be caught in legacy (that means: old school) Notes application development. There will be no movement in the classic Notes development area, every new feature will be in the XPages area. Hate or love it.

It’s not hard to believe that Notes 9 will focus very very much on apps developed with XPages, and a lot of the new UI will be created with XPages. The boundaries between the fat client and the browser will blur, but I’m very sure there still WILL be a lot of differences between both clients.

Replication is for sure one of them. It’s still a long way to teach a browser the replication features of Notes, and I don’t think we will see that in 2011.

RichText editing is another issue
. The RichText editor in the browser will improve very much in 2011, but it still will not be as good as in Notes. For example, navigating by keyboard in the RichText will still be a problem (stuff like CMD+LEFT jumps to the first char of the current line, pageup/pagedown and so on). Or things like highlighting and sections… I’m sure we will see some of there features in the browser RichText editor in 2011, but not all of them.

I really would like to see a much lighter Notes clien
t which gets rid of some legacy stuff, I’m not sure how many clients really need compatibility to Notes 3 apps in 2011… despite all the progress IBM made with the Eclipse client, and while it’s “fast enough”, it still feels fat, and there are still times when the client is blocked, the harddisk works like crazy and I wonder what the hell the client is doing.

Nevertheless, I think it’s a long way until we see some kind of “XPages only client”, and I’m pretty sure this will not happen in 2011. Therefore, I just hope for more performance improvements.

I really like the general direction IBM is going
by concentrating on XPages: new developers are able to learn XPages-Notes-Development fast, and it feels like a modern technology which is fun to work with. And it allows to leave the classic Notes development behind in some point in the future.
Furthermore, it motivates (or forces) every Notes developer to learn common stuff like JavaScript, CSS and even Java. That’s good, because it enables a developer to get out of the “weird Notes development cage”.
And finally, XPages apps are FAST and nice looking, and they can change the (bad) image Notes and Domino might have in a company.

So in a Nutshell: 2011 will bring a lot more XPages features and improvements, and Notes 9 will start to blur the boundaries between fat client and the browser.

What’s about Project Vulcan?
Because it’s not a product but a vision, and for me it’s a lot of talk until I see real stuff in a real product. Until then I’ll simply wait until I write about it.

Regarding Domino Designer: it simply gets better and better and better with every release. Maureen and her team did a very good job in modernizing Domino Designer! And they keep going, we will see a lot small and big improvements in 2011.
As of Domino Designer 8.5.2 with the SVN plugin, it’s a good and productive development environment, and we don’t need to be ashamed of it anymore.

Sure, there is still a lot of room for improvement (for example, the JavaScript editors are ok, but not good), but there no blockers anymore, and it is being worked on. So I assume at the end of 2011 with have a Domino Designer which is able to compete with most other modern development environments. Remember how Domino Designer looked like two or three years ago and you see how much effort IBM invested in this area.

And the Domino Server? Same as with Domino Designer, it’s getting better and better with every release! We get new great features, without the need for more hardware. How great is that? What other server software do you know which does not need a better hardware with each release?
I’m sure the Domino team will continue to add new features in 2011, but to be honest, I don’t know what the big ones might be, since I’m very happy with Domino as it is today (but in the end, I’m a developer, not an administrator, so my view is somewhat restricted).

Now to the core NSF database technology:
that’s my biggest painpoint for the near future, because I don’t see any real progress here. Yes, the NSF database is a very good No-SQL database, and it has a lot of great features. Nevertheless, we didn’t see any real progress here for years:

  • We really need old-school views to access data. What about dynamic queries?
  • The fulltext engine mostly delivers results fast, but not always.
  • When we want to select more than 5000 documents, we need a view or the slow database.search(). Therefore, creating reports for more than 5000 documents is still a pain.
  • 32K problem anyone? Having such a limitation is so 1980….
  • Laptops with replicas of some big Notes applications (2 GBs or more) have still problems with a lot of disk I/O, which slows things down a lot.

and so on. I understand that IBM does not want to improve NSF too much so it does not start to compete with “real” SQL databases like DB2, but while IBM modernizes Notes, Domino Designer and the Domino Server I really would love to see at least some progress in the NSF technology, too.

Lotus Live, Quickr, Connections
and so on: that are all good products as far as I can see it. I don’t know that 2011 will bring for them, my only concern is that IBM might try to rip features off Notes and Domino to try to sell these add-on products. And that would simply wrong.

We all remember the times where Websphere should be the solution for everything, and where customers should move applications from Notes to Websphere. That strategy failed completely, since Websphere is big and overengineered and needs so much more manpower and hardware. I really hope IBM does not try to make the same mistake again.

The IBM executives might like it or not
: Lotus Notes and Domino is a very good solution for collaboration and application development for companies of all sizes. It’s easy to manage, and one gets fast results. This is even more true with the new XPages technology. Once a customer gets that, they like it and don’t see any reason to move to another platform.
The “once a customer gets that” is the hard part here – which could be solved with good marketing. Since we talk about IBM here, this is just wishful thinking.

Finally, some thoughs about the cloud: sure, cloud offerings will attract more customers in 2011. But is the cloud the solution to everything? Is the cloud THE final solution for IT problems?
I do not think so. There are scenarios where putting stuff in the cloud is fine, but at least here in Germany there are a lot concerns regarding putting business data and services in the cloud, and I have my concerns, too.

So, this blog post got longer than planned 🙂 I wish you a healthy and successfull 2011, and I wish us all good business with Lotus Notes and Domino!

Jan 07

My wishlist to Notes 9 and Designer 9

Note: this post has been migrated from another blog. Some links may be broken.

I just chatted with Andrejus of Infosana about upcoming Notes versions… and while we talked I thought to myself “what are my wishes for Notes and Designer 9”?

There are lots of small things, but the most important ones are:

Lotus Notes 9:

  • Performance, Performance, Performance.
  • DAOS on the client to keep replicated databases small and save I/O operations.
  • Performance.
  • Keep on with the good UI work.
  • More improvements in reliability and deployment of composite applications, plugins and all the other new stuff.
  • Performance.
  • Improvements for XPages components, for example a view control which is more like the native Notes view. Or better RichText editor and compatibility to the native RichText editor.
  • Performance.

With Performance I mean:

– Start faster.

– Don’t use my harddisk all the time when I have a number of big applications replicated. That’s a pain especially on Notebooks.
– Open documents with complex forms faster (-> speed up rendering of Notes forms).
– Open views faster.
– Faster fulltext searches.
and that kind of stuff. The are much improvements in 8.5.1 and it’s a good start. But still, customers complain about Notes feeling slow.

Event when it’s not fair, but customer compare it to Outlook. Outlook starts in a fraction of time as Notes. Outlook just feels faster.

Designer 9:

  • I’m very happy with XPages, even on the client. We’ll just need the planned improvements here and there, and improvements for XPages components on the client, to make them feel more like classic Notes.
  • SVN integration, that’s the big point. But IBM Is very aware of that, it’s a tough challenge, but I have hope that they will deliver something in that area.
  • Make composite application editor faster, since it reloads a page after each small change it’s sometime a pain to work with it.
  • All the feature requests I already told IBM for composite applications 🙂
  • Other XPages render kits, for example for iPhone and Blackberry.

So, no really big wishes for Designer.

And for the Domino server? Frankly, I don’t know. I really like Domino 8.5.1. It’s stable, feature rich, makes good use of hardware, is available on many platforms… oh, one thing: support for Ubuntu server.
But beside that, I don’t have any big wishes for Domino server.