Building the Data Highway to the Clouds

Moving data between terrestrial and cloud computers is simply an age old challenge with a new twist. Once upon a time information technology architects spent a LOT of time on data in motion so they could guarantee response times and availability. A good example pitted DEC against IBM where the former would transmit characters and the latter would buffer entire screens to maintain an "illusion" of responsiveness.

Cloud computing presents a similar set of challenges because some of the luxuries we've grown accustomed won't always be in play. For instance, as LAN speeds have risen to the gigabit level, we rarely ponder the physical aspects of moving a file. When we browse the web, our pages are popping but we give no thought to the web services infrastructure behind it. Some of these little considered aspects will now move back to the front of the class as we try to leverage the power of cloud computers.

Starting from the basic premise that building an effective cloud computing environment for any organization will happen over time, we can anticipate a LOT of back and forth movement of data between the ground and grids built by Amazon and Google. Of the current solutions, Amazon's is closest to a standard data center with their Elastic Computing Cloud (EC2) delivering gigahertz of compute power and their Elastic Block Storage (EBS) promising terabytes of data storage for substantially less than hosted solutions.

Taking advantage of this new frontier will mean that we need to pay attention to the arcane tools of compression, encryption, delta differencing, versioning, and especially time shifting data movement because simple copies between hosts will take too long. Specifically, the LANs on the ground and in the grid will be used to quickly transmit files to a data pump that will manage the trip between the earth and the sky, or vice versa. This data pump will securely hold files for use on an as needed basis when new cloud computers or terrestrial virtual machines are deployed.

In other words the logistics of traveling the data highway will lead to data abstraction from the operating system images so systems can be built and torn down as needed. Last but not least, this new frontier will put a premium on lightweight, and license free, deployments of specialized compute kernels that will request their data files "at birth".

This promises to be an exciting time and we look forward to meeting the challenge.