Ryan Cartwright has written an article on how to backup Gmail with Getmail. It is an excellent piece, but Getmail is a command line program, and while most readers of FSM will doubtless be at home there and in configuring XML files, we also like to behave like electric currents and take the course of least resistance to get from A to B. Besides, using the Thunderbird (or Evolution) e-mail clients to backup Gmail has other advantages too. Let me explain.

Thunderbird is go Virgil

Fine granularity is one of the great payoffs of the command line. So, three cheers for the command line

As Ryan said, Getmail is only an apt-get install getmail away. The trickier part comes in configuring and setting it up. After that, if you want to read your email, the venerable Mutt is suggested. GNU/Linux is blessed with a multitude of e-mail clients. Mutt is one of the more basic ones of course, but for the bog standard task of just reading those backed up e-mails from Google's servers it is quite adequate and as the author has indicated, using Mutt to read them obviates the need to set up accounts. And yes, because we're in command line territory we can roll out cron to schedule Getmail as well. Fine granularity is one of the great payoffs of the command line. So, three cheers for the command line. If you want similar functionality with a few bells and whistles too though, you might want to think about using one of the big hitters like Thunderbird or Evolution. The latter comes as part of a default Gnome install and Thunderbird is, like Getmail, only an apt-get install away.

If you're not already using either of these clients, setting up Evolution is very easy and Thunderbird will be in the repositories of virtually all the main GNU/Linux distros. I've been using an alpha version of Thunderbird 3.1 on Ubuntu (Karmic Koala). The final version (3.1) has just been released but it won't be available in the Ubuntu repositories or an Ubuntu PPA. If you want the latest version point your browser to a Debian binary of Ubuntuzilla over at Sourceforge.

Install it, fire up a terminal and type ubuntuzilla.py -a install -p thunderbird and you'll be good to go. The official Ubuntuzilla site has detailed instructions. If you prefer the tarball route Mozilla should be your port of call. The Wizard will carry you through all the necessary steps to set up an e-mail account for downloading and reading your mails on Google's servers. If you are already using a version of Thunderbird to read your e-mail from another domain, including those provided by your ISP (as I used to do before I changed broadband provider and lost access to that account and realised that I needed an "independent" webmail account like Gmail) it is simply a matter of setting up another account by selecting Account Settings from the Edit dropdown menu and adding your Gmail account details.

IMAP

Now, of course, you can connect to your e-mail account via POP; the problem with this protocol is that, by default, it is set to download you e-mails from the server without leaving a copy there. You can change this setting by logging into your ISP mail account but this only allows you to leave messages on the server and that is restricted to the Inbox. You have few configuration options. This is where IMAP comes in. Despite the fact that it has been around for some time (since 1986 in fact) and that Google rolled out IMAP for Gmail back in the last quarter of 2007 it is something of a relatively well kept secret. I was using e-mail for years before I discovered it--and only because I had switched my ISP, lost my e-mail account and didn't want to repeat the mistake. (Google has done this protocol a huge favour by adding it as a connection option but even so, IMAP's creator, Mark Crispin, described the Google implementation as "a work in progress". However, Google's global reach ensures that more and more people are becoming aware of the advantages of using IMAP and as the uptake increases the feature set improves. It is a double edged sword though. If more people access Gmail via a desktop client, POP or IMAP connected, Google's advertising revenues may be affected as a desktop client screens out the advertising you see when viewing Gmail as webmail in a browser.)

When I "discovered" IMAP I realised this was only the tip of the proverbial iceberg

So, I joined the herd and set up a Gmail account which gave me independence by not tying my e-mail account to my ISP. I was quite content with that, before I discovered that I could enable IMAP on Google's servers and extend my options. Before that I was using Gmail via a browser interface. I found it a bit slow and I never really took to the interface or its features. Once I updated Thunderbird and starting using it again with Gmail I actually started to enjoy accessing and reading my e-mail again. When I "discovered" IMAP I realised this was only the tip of the proverbial iceberg. So, what exactly is IMAP and what are the advantages of using it?

The advantages

• can be operated in both online and offline modes
• allows you to manage your Gmail account from multiple desktop clients (unlike POP)
• supports encrypted logins
• supports e-mail traffic encryption via TLS (and its predecessor, SSL)
• ability to retrieve body of an e-mail without downloadng attachments (partial fetch)
• ability to track individual e-mail status (read/unread) from multiple clients
• supports creating, renaming and deleting e-mail folders from client software
• supports access to have shared/public folders with other IMAP users
• some IMAP clients support storing sent e-mail in a client-side folder
• better spam, virus and junk mail filtering from the client side
• ability to drag and drop old email from other accounts in Thunderbird to Gmail folders
• easy to migrate to other e-mail clients (no need to move or convert formats)
• IMAP allows you customise the Gmails folders (labels) that appear in Thunderbird (via subscriptions)
• supports push e-mail (obviates need to poll the server) via the IDLE command
• password protect the message list pane

Configuring it all

Well, that's a pretty impressive list of reasons for using IMAP over POP but the real bonus comes when you can access it via clients like Thunderbird (and Evolution--I'll come back to that later). However, our first port of call is to actually enable IMAP in Gmail before we do anything else. And that's an absolute no brainer. By default, when you set up a Gmail account POP is the protocol you are using. So let's change that to access those goodies. Just log in, click on Settings and select Forwarding and POP/IMAP, scroll down to the section labeled IMAP Access and enable it. Save your changes and exit back to the main Gmail page. Yep, it's that simple. You're done.

Time was when setting up IMAP on desktop clients like Thunderbird (and Evolution) was not straightforward. It is now. If you have set up a POP account then configuring Thunderbird for IMAP is just as easy. Assuming you have already installed Thunderbird and have run the account wizard, setting up a second account for Gmail is simply a matter of going to the File drop-down menu, selecting New and then Mail Account and filling in your Gmail adddress. Click continue and Thunderbird will automatically detect your Gmail settings for both incoming and outgoing e-mail for IMAP. Click Create Account and your ready to roll. (Settings can be checked and/or amended in Edit/Account Settings.)

Folders versus labels

One of the most obvious differences between Gmail and Thunderbird is that where Thunderbird organises e-mails under a conventional folder structure Gmail works with labels. The good news though is that, because IMAP is bi-directional, creating, deleting or renaming folders/labels will be synchronised. What you do in one will be reflected in the other. With both Thunderbird and Gmail open and logged into, create a new folder in Thunderbird and reload Gmail in your browser and the new label will be there. Right click on the folder in Thunderbird and you can delete it and it will be deleted in Gmail too and of course, as they are synchronised, if you use Thunderbird offline for any reason and edit your folder structure in any way, when you are back online Gmail's IMP settings will pick up the changes and reflect them the next time you access Gmail in your browser.

To get Thunderbird to refresh automatically without a restart you need to add two entries via the Config editor. If you've used Firefox (and who hasn't?) you'll know the drill

What happens though if you create a label in Gmail and want to see it as a folder in Thunderbird immediately? I couldn't find a "reload" feature. The only alternative is to close Thunderbird and restart it to see the new folder. Not exactly user friendly. The solution lies in remembering that Thunderbird is, like Firefox, a Mozilla product and like Firefox it not only has addons but that very useful about:config feature. To get Thunderbird to refresh automatically without a restart you need to add two entries via the Config editor. If you've used Firefox (and who hasn't?) you'll know the drill but let's recap it anyway.

Access Preferences via the Edit drop-down menu and select the Advanced button under the General tab and click on Config Editor. The About Config box will open. Two preferences need to be added and to do this, simply right click anywhere and choose string and type in the following: user_pref(“mail.check_all_imap_folders_for_new”, true); and to avoid Thunderbird displaying a sending dialogue also add the following as well: user_pref(“mailnews.show_send_progress”, false);. In both cases, click on OK and the dialogue box will pop up again. Leave it blank and hit OK again and the user preferences will now be visible in bold in the Config Editor. (The effect of creating/deleting or editing labels in Gmail may take a few seconds to synchronize with Thunderbird's folders.) Bye bye restarts. As an added bonus this also ensures that Thunderbird will download all accounts so that if you have multiple Gmail accounts they will all be downloaded.

Yes, subscribe me

Another way to control what Gmail labels display as Thunderbird folders without creating/deleting or editing is to use something called Subscriptions. You can access this feature either by right clicking on any folder in Thunderbird and selecting Subscribe or navigate to it via File > Subscribe. This opens a checklist box which allows the user to both subscribe and unsubscribe to IMAP folders. This means that when synchronising with Gmail you can be selective about what you actually download. (Note to the developers: this feature has confused some users who want to unsubscribe to folders and can only see the Subscribe item in the File drop-down menu. They don't realise that the Unsubscribe option is buried in that menu option. Come on guys, add Unsubscribe as a separate menu item and stop teasing.) This feature is useful for managing the size/number of folders and the time taken to download their content.

Spam, spam, spam, spam

Google doesn't give you the option to turn off spam filtering. It's Google's spam control and you're not invited to the party

Gmail is known for claiming good spam control. (If you want a good visual presentation of how it works, take a look at this video). It uses Optical character Recognition (OCR) to block image spam, machine learning algorithms and multiple authentication systems (SPF and DKM). However, strictly speaking, it does not "block" spam. It simply redirects it to the spam folder where it resides for thirty days and once it is gone there's no chance to retrieve false positives. Google doesn't give you the option to turn off spam filtering. It's Google's spam control and you're not invited to the party (though you do have some limited ability to create filters under Settings). However, if you are using IMAP with Thunderbird you can wrestle back some influence and control.

That lack of control is ominously reminiscent of the worst excesses of proprietary software but no such restriction applies to Thunderbird. It's spam control is based on Bayesian filtering and it "learns" on per user basis to identify spam (or junk, as Thunderbird classifies it, but you need to train it and each e-mil in the Inbox can be individually flagged as junk or not junk by right clicking them. Additionally, Thnuderbird's junk settings (Edit > Account Settings > Junk Settings) allow you to explicitly exclude your Personal Address Book and Collected Addresses from being treated as spam. That's good, but even better is this useful tip: in order to teach Gmail's spam filtering, in Thunderbird's Junk Settings, tick the checkbox marked "Move new junk messages to" and select Gmail's spam folder (label). That way, you can kill two birds with one stone (no pun intended). Thunderbird's adaptive junk filter learns progressively to identify spam and it, in turn, teaches Gmail's spam filter. That should reduce the number of false positives. Thanks Reverend Bayes!

As if all those IMAP goodies weren't enough, Thunderbird itself has some tasty stuff to tickle your palate and leverage Gmail that bit more. The clue is that Thunderbird is out of the Mozilla stable, the same developers who gave us Firefox and that of course invokes one of my favourite words: addons. They're not as prolific for Thunderbird but there's enough to usefully extend it. The official addon site for Thunderbird will give a flavour and a good number from which to choose. Everyone will have their own favourites and here are the ones I think get the functional best out of Thunderbird--and Gmail.

WAT or Thunderbrowse

They have the same purpose but I mention both because, as with all addons, you may run into version compatibility issues. I did. Thunderbrowse wouldn't play nicely but I managed to install WAT. Whichever one you install, the addon turns Thunderbird into compact and minimal tabbed browser. Latest versions of Thunderbird have a tabbed interface for reading e-mail and the WAT addon just extends that to allow you to view web links from Gmail in another separate tab (or an external browser). So, you can read your Gmails whilst browsing their links and never leave the confines of Thunderbird --unless you want to, of course. As an added bonus, WAT (Web Application Tab) also allows you to independently view websites in their own right. From the WAT drop-down menu select Option and you can set up a primitive bookmark system. The link will be added to the WAT menu (and if you want to mimic a browser-style sessions feature just close Thunderbird without closing any tabs and it will reopen with all open tabs intact. Neat.) One final advantage of installing WAT is that you can view and manage Gmail from a web interface in one tab while reading the same mail via Thunderbird, to monitor configuration changes and get the best of both worlds. Better still, if you have more than one Gmail account (POP or IMAP) you can view them simultaneously under one tab (in any browser and not just Thunderbird) without having to log in and out between them--if you follow this excellent howto. But that would just be showing off, wouldn't it?

TEBE

This article is an attempt to take Gmail to the next level by using IMAP in conjunction with Thunderbird. This combination makes for a better and more fully featured e-mail experience and in the process back up Gmail; but I'm the kind of person who thinks it might be unsafe to venture outside in case someone steals my trousers; so when I say backup I don't just mean backing up Gmail by having "local copy" on my hard drive. That's good, especially when , for whatever reason, an internet connection is not available and you want to check mail or compose one for later sending. What happens though, when your laptop has been stolen or has crashed? No, backup has to also mean either removable storage backup (USB hard drive or flash stick) or using an encrypted backup uploaded to sites like Dropbox or Tonido for example.

If the perfect storm happens Thunderbird can be installed on another machine and TEBE ued to restore settings and e-mails

Thunderbird does not have a default mechanism for backing up the actual e-mails, let alone any of the settings (themes, addons etc.). However, there is an app for that. Yes, you've guessed it. Another addon. I can vouch for it (at four million downloads) as I have already used a version of it by the same developer, Chuck Baker, who wrote it for Firefox. It was called FEBE. This gave rise to TEBE, an addon for backing up Thunderbird. It's still in beta (which is presumably why it hasn't reached the official Mozilla addons site yet, as FEBE has) but I did get it to work and it can be configured for complete or selective backups, including message folders. I saved a copy to my local drive and another copy to a flash stick. The hyper security paranoid might want to encrypt and password protect that backup. If the perfect storm happens and my internet connection is down and or my hard drive is inaccessible Thunderbird can be installed on another machine and TEBE ued to restore settings and e-mails.

Of course, the user is plagued by application and addon upgrades causing version incompatibilities. One solution, apart from hacking the install.rdf file of the offending addons, (see my article on fixing broken addons) is to backup the really easy way with Evolution's built in script. Evolution is part of the default Gnome desktop, so it's only a matter of launching it for the first time and following the Wizard to set up Gmail IMAP. Backing up settings and e-mails is as easy as selecting Backup Settings from the file dropdown menu. You will be prompted for a name and place to save the backup. The file will always be tar.gz type and this should not be changed. The only downside to this wonderfully simple method is that you can only restore the backup to the same version of Evolution.

Thunderbird, Evolution and KlamAV?

Gmail's spam filtering and phishing controls are well known and reasonably effective. Less well known is how it handles viruses. Gmail can detect a virus in an attachment and will prevent you from downloading it. How good it is depends on the software and how up to date the virus database is, but Google don't say who their provider is (though rumoured to be Brightmail, acquired by Symantec). At least Yahoo tell you that they use Symantec and Hotmail use Trend. Additionally, users can't disable or configure it either. I did notice that in the Labs section of Settings you can enable an Authentication icon for verified senders to combat spammers but it only works for Paypal and eBay. Thunderbird has this feature but you need to install the addon for it. It might be better to have the option to disable Google's security measures and use your own client side anti-virus software instead but given the inertia and technical illiteracy of many users it's not likely that they will be doing this anytime soon--even if only to protect some users from themselves and to avoid leaving themselves open to accusations of being negligent.

Of course, it's true that there has never been a GNU/Linux virus in the wild but almost certainly, someone in your e-mail address book is using Outlook on Windows, or you work in a heterogeneous environment populated by multiple operating system platforms and you risk acting as an unwitting virus vector. So, if you are not entirely happy with Gmail's first line of defense, can Thunderbird give you an added layer of protection? Well, there may have been a time when you could do it. In addition to installing KlamAV you also needed a Thunderbird addon called Clamdrib but it has fallen off a cliff and seems to be nowhere to be found. So, no possibility of on-access scanning with a combination of Thunderbird, KlamAV and an addon. Evolution might come to the rescue.

There is no extension to enable this functionality in the Ubuntu repositories but the ubuntu forums has a ten-line bash script to call ClamAV to scan incoming e-mail for viruses. It is released and written under the GPL so no worries there. I copied and pasted the script to Gedit and saved it as clam-filter to my home directory and made it executable with chmod 777 clam-filter. Create an Evolution directory called virus dump as a repository for infected e-mails, as you will need this when configuring the Evolution filter. From there follow the instructions to set up a filter in Evolution and you're done.

Oops! This may work with your ISP's mail server (POP or IMAP enabled) but with Gmail it's a catch 22 situation. I tried to send myself an e-mail through Evolution but Gmail blocked it, identified it as "an illegal file" and Evolution dumped it in the Outbox folder. I think the only way to test this would be to have a separate non-Gmail account for sending and see if receiving it via Gmail but read through Evolution worked. Even then, I suspect that Gmail would block it and it may not get as far as being scanned and filtered in Evolution. This would constitute a serious nuisance for anyone trying to report an infected file to antivirus vendors--although there are ways around this. It looks like there is a limit to the degree of integration between Gmail, IMAP, Thunderbird and Evolution.

And finally, Spiecbird

One of the pleasures of writing articles like this is that it often lead to serendipitous discoveries. Prior to researching this piece I had never heard of Spicebird. It's a fully GPL'd, tabbed e-mail client based on Thunderbird's Mozilla code. It was developed by Synovel who describe it as a collaboration client which bundles e-mail, calendaring, contacts and instant messaging in one application. It won't do anything more than Thunderbird but it has some nifty features that are worth having, as the screenshots illustrate.

if you like iGoogle then Spicebird's home tab will make you feel right at home

There is no stand alone binary available for any distro or PPA or distro repository binaries but installing is a breeze. Spicebird is available for download as a tarball from Synovel. The Spicebird wiki has an excellent install guide which you can copy and paste into a console. It will happily co-exist with both Thunderbird and Evolution and they can all be run simultaneously whilst logged into Gmail's IMAP account if you want or need to access features unique to any of them. Oh, and if you like iGoogle then Spicebird's home tab will make you feel right at home.

This article has concentrated on using Gmail and IMAP's features in conjunction with Thunderbird (and Evolution, to a lesser extent). I have not discussed the best of the latest release (version 3.1). That's a monograph in its own right. Once you've explored the dense feature set of Thunderbird you'll get even more out of using it to read and manage your Gmail account.