Not logged in » Login
Jul 21 2018

“Big Four” Launch Data Transfer Project

Ever since the heydays of GeoCities and Tripod, Internet users around the world have tended to store increasingly larger chunks of information online. Today, that (mega-)trend remains unbroken, not least due to the fact that there's practically no limit to the purposes such platforms can be used for. From operating a web shop through writing a travel blog to keeping a personal diary, pretty much everything is in the mix, and it's only logical that everyone involved will at some point upload documents (like invoices or receipts), notes, conversations with friends and colleagues, graphs and other drawings, and many, many photographs and videos. Then one day, the inevitable happens: Your trusted hosting platform informs you that it will raise prices, loosen privacy protections, merge with another service or simply close up shop – and that you have 90 or sometimes 30 days left to transfer all your files and data to a new host.

Anyone who's ever had to manage such a digital relocation process will sooner or later tell you that the job isn't as easy as you'd like to think or could reasonably expect. The reason for such widespread consensus is at least twofold. On the one hand, direct service-to-service data transfers are an exception rather than the norm, which means that unless they've kept local copies, customers first have to re-download their files from their original provider and then move them to the new platform. One of the few that chose a different way was Google, whose Download Your Data tool (previously known as Takeout since 2011) enabled users to export data from its mail, storage and calendar services headlong to those of competing vendors. However, some time in 2017 the company decided this wasn't enough and that it was time to convert it into a more comprehensive solution with the help of other big players, namely Microsoft. The result of this collaboration was the Data Transfer Project or DTP, a collaborative effort to create a common framework that uses open source code to "connect any two online service providers, enabling a seamless, direct, user-initiated portability of data between the two platforms." Preliminaries started last fall, and by the end of January, the first pieces of code appeared on GitHub, which has since been acquired by Microsoft. Lately, the co-founders invited two other big players to give their endeavor more clout: With Facebook and Twitter on board, the DTP is now poised to serve as the 'mover of choice' for all file and data transfers that happen outside of the Amazon and Apple ecosystems.

Technically, DTP builds on the various services' existing APIs and authorization mechanisms to access (or rather ingest) any data that must be moved. In order to simplify transfers, the data is then converted into a common format with the help of service-specific adapters. Once a transfer is completed, another  adapter translates the data from the common to a target format so that it will work with the new service's APIs. Meanwhile, a third set of mainly OAuth-based adapters take care of the necessary authentication processes to ensure that all transfers are legitimate, and a couple of libraries handle so-called background tasks such as calls between adapters, failure handling, and secure data storage. Thanks to this setup, users will be able to skirt the second great obstacle that turns mass file relocation into the job everyone hopes to avoid: the risk of losing files and/or conversion failures. In a best-case scenario, the relocation process could become almost entirely transparent; all a user might have to do is log into his old and new accounts to kick off the move, and then return after a while to find it's all done.

However, while the idea of using hassle-free, foolproof and largely 'unmanaged' data transfer services is definitely fascinating, it should be noted here that DTP is still pretty much under development and far from being a mature platform or technology. For more information, please visit the project home page linked above and read the white paper Data Transfer Project Overview and Fundamentals.


Comments on this article

No comments yet.

Please Login to leave a comment.


Please login

Please log in with your Fujitsu Partner Account.


» Forgot password

Register now

If you do not have a Fujitsu Partner Account, please register for a new account.

» Register now