KDEPIM sprint #2 (Understanding and diagnosing Akonadi)

In this second part I'm going to explain a few techniques to debug Akonadi we use in order to understand why my KMail installation was working bad.

My configuration
I'm a heavy email user, every day  I receive and read hundreds of them, and I love to have my email synchronized across devices (laptop, workstation, phone, tablet) and particulary in my laptop and workstation I want to have all my email offline so I can make use of it with or without internet, specially useful on my laptop while traveling. Because of all those reasons I have to use something called Disconnect IMAP.

My EMail account
The account has 5.6GB of data and according to GMail it is split in 700K emails (more or less) 200K of those emails are in a folder (label in GMail) called kde-commits with a size of 1.8GB, then it has 5 folders with more than 25K emails and around 200MB of data each, and finally the rest are just small folders with 500-5000 emails. In total there are 70 folders.

First import experience
The first problem I have with KMail2 is the experience out of the first email import. Since my account is so huge it takes a while (around 15 hours) to make the full import and I can't use the application while doing it because either the data won't be available or all my CPU power will be used in the task.

To fix this we designed something called "Spanish sync" which basically consist on splitting the task of "Downloading all email while indexing it" into the following steps:

1. Download all the envelopes (subject,from, to, and some other data) starting from the most recent. Once this is done KMail will be fully functional (full email will be downloaded when user tries to read it).
2. Download in a controlled way all the email bodies (where the content of the email is). Right now this goes as fast as your internet line allows which can be a problem for people having a fast internet connection (100Mb in my case) and slow machine (my laptop is 6yrs old). Once this finishes the email content will be available offline.
3. Finally we will index all emails. If the user tries to search before this is done we will show an progress indicator.

The full synchronization of my email is still going to take 15 hours but I will be able to use my application almost since moment one (since it will import most recent emails first).

KMail not showing email body
Even though I set my account as "Disconnected IMAP", more or less 50% of my emails were not being downloaded so I kept getting this "loading email" message from KMail. This is specially annoying when I'm on my laptop since the internet availability while traveling is not reliable. We have not been able to identify the reason why 1 out 2 emails is not being downloaded but at least I learned how to diagnose this.

Execute the debugging application "akonadiconsole" and go to the "DB console" tab and execute the following SQL query:

SELECT count(*) FROM parttable WHERE name = "PLD:RFC822" AND data is null;

The result should be a small amount of your total email (in my workstation it is 4K while in my laptop it is 300K). If you set your account as "Disconnected IMAP" and you are getting a high number you may be suffering the same bug.

Unfortunately there is no way to force Akonadi to download the missing body for all those emails... so you will have to either live with it or re-import your account.

KMail not updating email status, not loading anything, not working !
If all your email is offline and you do not suffer from the bug I mentioned before, you won't even notice this... but if you are using "Normal IMAP" or you suffer the bug this is the most annoying behavior.

Basically all the actions you perform are put in a queue to be processed, for example if you read 100 emails while offline, those "100 read actions" will be stored locally and pushed to the server once online. Where is the problem? the problem is that right now "pushing" this information blocks any other operation to be sure that we are using fresh data so if for instance email XXX is requested akonadi won't provide it until the the "pushing" finishes.

In most IMAP servers pushing this data is FAST, but for some reason in GMail each modification takes around 1/2 seconds... so if I set as read 100 email or I delete 2000 make the math :p

The solution for this is conceptually quite easy, we should only block operations that affect email XXX instead of blocking everything. The technical implementation is a little more difficult than that :p

You can use akonadiconsole to check this "todo list", open it go to "Agents" tab, and in there open the context menu on the agent you want to check and click on "show change-notification log".

Summary:
Akonadi and Kontact2 are an awesome piece of software, I have no doubts that once the issues I mentioned here are fixed we will offer a better user experience than the one you get either in osx or Thunderbird.

As a note, Thunderbird or iMail take also 15 hour to fetch all my email, with the difference that I can use my email right the way, we are not that far.

Finally, a screenshot of how I'd like KMail to look like by default... and I'm not the only one asking this :p

Edit: to have this config just do:

Settings-->Appearance-->Layout-->Show the message preview pane next to the message list
Settings->Appearance-->Message List-->Default Thene: Fancy with Clickable Status