Qbmove: the new email migration system developed by Qboxmail

Switching to a new email service provider entails meticulous planning for the migration of existing email accounts, along with all their associated folders and messages.

During the email migration planning process, it’s imperative to consider specific elements, such as:

• The number of email accounts slated for migration
• The total space occupied by these email accounts
• The speed of the source server (your current provider)
• The number of email accounts that can be migrated in parallel from the new provider

Qboxmail’s Control Panel has consistently offered customers a user-friendly tool to strategically plan and automate IMAP-based email account migrations from their current provider to Qboxmail.

In October, a more sophisticated migration system, named Qbmove, was introduced. 

This system was meticulously developed by our dedicated Research and Development department. In contrast to its predecessor, which was relatively straightforward, clients would upload a CSV file, initiating a Perl process for each email account to handle message transfers via IMAP.

Nevertheless, the former migration system encountered several limitations:

• It failed to scale efficiently without the addition of extra RAM to the individual server running the Perl process.
• Server updates caused disruptions in migrations without any option for resumption.
• The number of parallel migrations was predetermined and couldn’t be customized on a per-client basis.
• IMAP errors reaching the Perl process were undifferentiated, leaving clients with a generic error message, necessitating log file analysis to identify the cause.

As a response, we embarked on developing a new migration system that not only rectified these issues but also provided more adaptability for future implementations.

The architecture of the new migration service:

The idea behind the architecture of the new system is to have a single entry point, the orchestrator, which in turn configures the processes to be executed on other servers and tracks their progress.

The orchestrator is an application written in Nodejs, utilizing the SSH protocol to launch and monitor processes on the servers made available to it.

In this way, new servers can be added for executing migrations by simply specifying a new IP address to the orchestrator. It is also possible to dedicate entire server clusters to specific types of operations or clients using a highly flexible rules system.

Another fundamental detail is that once a process is initiated, it operates independently of the orchestrator. If the orchestrator were to malfunction or if we simply chose to update it, no ongoing migration process would be affected. This has always been a crucial requirement for us, especially since some migration processes can last for several days due to the slowness of the server from which we retrieve emails.

After the processes are launched, the Nodejs application monitors them, extracting useful data such as resource utilization, progress details, and any errors. These pieces of information are then exposed through internal APIs in our control panel, which are displayed to the user.

The servers responsible for the actual transfer do not require many dependencies but simply:

• Are accessible from the orchestrator.
• Contain the necessary Perl scripts for message transfer.

Migrations can be initiated on one server and completed on another, thanks to the sharing of states and data cache. Furthermore, we can allocate a higher number of parallel migrations for each client, thereby speeding up the migration process.

With Qbmove, we have the capability to introduce new features to provide greater flexibility in managing the email migration process for our customers.