Compute Cloud Abstraction APIs ... who needs 'em?

This blog is about open source cloud computing libraries. Specifically, this is about libraries that abstract away differences between compute clouds such as EC2 and Rackspace. In the following blog, you'll find opinions from consultants, business founders, users of the cloud and even those who run a cloud themselves. They will tell you their opinion on the value proposition of cloud aggregation and abstraction libraries. Some mention their expectations for the future. In a follow-up, I'll discuss some of the tools these leaders mention.

Here are the commentators and their backgrounds. They are listed in order of appearance in the blog. We should all thank them for taking time out to help us understand the issue better.

------------------------------------
[1] Tom White http://www.lexemetech.com/
Tom works at Cloudera on the popular Map/Reduce engine: Hadoop. He's also a committer on the libcloud project.

[2] Bradford Cross http://measuringmeasures.blogspot.com/
Bradford is co-founder of Flightcaster, and founded crane: an open source cluster management project.

[3] Shlomo Swidler http://www.orchestratus.com/
Shlomo is basically "the" expert on EC2. He's also an expert on cloud ecosystems generally, deeply involved in OCCI and the popular cloud connect conference.

[4] Thorsten von Eicken http://www.voneicken.com/
Thorsten founded the most popular cloud management platform: rightscale.

[5] Peter Bryant http://rimuhosting.com
Peter founded the compute cloud RimuHosting.

[6] Michael D Neale http://michaelneale.blogspot.com
Mic works for RedHat on a rules-based cloud deployment project called Cooling Tower.
------------------------------------

Tom [1] wrote us a great introductory piece on what led us to multi-cloud apis. Let's review where this all began:

------------------------------------
The advent of Amazon EC2 a few years ago meant that for the first time independent developers could run clusters of machines just by turning up with a credit card and paying for the resources they used on an hourly basis. More recently other cloud providers have started offering comparable services, which again is great news for developers, since it introduces competition into the market. With the multitude of providers comes a multitude of APIs, however, each with slightly different nomenclature, features, and language bindings. In the absence of cloud standards, developers have had to pick a provider and write to its API, making migration to other providers difficult at best. This is why multi-provider APIs like jclouds and Apache's libcloud are so attractive: they promise to provide a single API (typically for a single language: jclouds is Java, libcloud Python) for a multitude of providers. Of course, the difficultly is now: how do you provide a uniform API to a set of services that are so disparate?

Having worked with a number of providers, I think cloud providers need to offer at least the following minimal set of features to succeed. This is in some sense the "Turing complete" set of cloud compute services, since they offer enough for a developer to conveniently develop virtually any application on any provider - any less and some things are incredibly difficult or just impossible to achieve.

1. Instance lifecycle. The lifecycle defines the basic commands to provision, start, stop, and terminate instances. The bare-bones of a compute service.
2. Shared images. While it is possible to bootstrap from plain OS images created by the cloud service provider, the ability to build your own customized image, and crucially to share it with others provides a social element that helps drive adoption of a cloud platform.
3. Instance metadata. This feature allows you to inject small amounts of user-specific metadata to each instance at boot time - e.g. secret keys, or custom boot parameters - which allows another level of customization. This feature works well in conjunction with shared images: the common non-user specific code is baked into the shared image, and the user-specific code is supplied at launch time as metadata.
4. Network controls. Cloud providers need to think about the network environment that the user's instances run in. Offering such services as DNS, firewalling, VPNs (and exposing it via an API) makes it easy for developers to get started quickly without having to build this infrastructure themselves.

The multi-provider APIs have made a great start on abstracting these features, but there is more to do, so please get involved and help out!
------------------------------------

One point Tom raises is that cloud providers are indeed offering comparable services, and that leads to further competition. I rather enjoyed Brad's [2] succinct angle on where this leads.

------------------------------------
The value proposition of cloud is the commoditization of compute resources and services - perhaps most specifically scalable, distributed services. If we want utility computing services, they must first be homogenous, so that one can switch between them. We are not there yet, but abstraction layers over compute, storage, and other services are a step in that direction.
------------------------------------

So, what progress are we really seeing on this continuum to homogeneity? Shlomo's [3] take on the topic gives us a wider look at the issues that are being attempted here.

------------------------------------
Multi-cloud libraries play a key role in today's cloud ecosystem. These libraries promise the programmer the freedom to code without being dependent on any specific cloud's interface. This allows business decisions - for example, "Which cloud is the most cost-effective place to to operate?" - to be taken without recoding the application, and without the programmer needing to learn each available cloud API. Applications that can operate equally well with multiple clouds will encourage openness, drive competition between cloud vendors, and lower prices for all players.

Today, the features and APIs provided by each cloud vendor are so different, with new ones being introduced all the time. How can a developer be sure that his application produced today will support tomorrow's best choice? This question raises multifaceted issues, and these issues are being tackled today by the industry. We're developing standard protocols so new clouds can be driven by existing libraries. We're assembling taxonomies of services so their features and variations can be described in a machine-understandable manner. And we're developing ways to express service-level demands so applications can be automatically orchestrated to obey an SLA. With library support for these capabilities application developers will be able to shift their attention away from cloud management, focusing instead on the application's domain-specific challenges. This vision will be made possible by multi-cloud libraries.
-------------------------------------

Let developers stay focused on doing work as opposed to recoding apis to the latest feature. This is an easy one to get, but let's let Thorsten [4] challenge this.

-------------------------------------
I'm quite skeptical of a standard cloud API at this point in time, there are still too many degrees of experimentation and in the end the syntax of the API isn't that much work to write a driver for. The only real benefit now would be that perhaps we wouldn't need to explain to every vendor that their list_instances call has to list the actual details of each instance and not just the names and ids, or that we need to be able to specify something like the EC2 user data when launching an instance. If you look at our open source cloud Ruby libraries (RightAws, RightRackspace, RightGoGrid, etc.) you'll see that it's not actually that much work to abstract the API details away. The real benefit of standards would be around the semantics of resources like vlans, security groups, reassignable IP addresses, network attached block storage devices, etc. But vendors today barely support these and if they do they have quite different semantics from one another that reflect different underlying architectures and also use-cases. This is not something that can just be standardized at this point.

The "higher level libs" I have seen aren't much higher level and all they seem to do is allow you to launch a server and stop it (or very little more). We're working on a gateway that provides true multi-cloud abstractions for things like disk volumes, security groups, IP addresses, etc. but it's too early for us to publish it. I'm following deltacloud and while I think it's an interesting project I don't see the leverage yet for any sophisticated cloud use. Doesn't mean they're on the wrong track, though.
-------------------------------------

Thorsten claims there isn't much work to do. Peter [5] has a relevant response to this, also amusing considering the context of cloudslamevent!

-------------------------------------
I read that as cloudslaveevent. The amount of work RimuHosting, other providers and the API makers put into our APIs and aggregating APIs it is no wonder.

What I like about multi-compute APIs (= libcloud, deltacloud, jclouds, etc?). I've been referring to those as cloud compute api aggregators.

Easy want to get alternative language (Python, Java) bindings for our API.

We're seeing better adoption of our API than we may have seen otherwise. e.g. it is very easy for outfits like serverdensity and cloudkick to plug us in as a provider. Giving us access to new customers. As well as giving our existing customers access to tools or services that they may not otherwise get (e.g. in the case of server density and cloudick they get the monitoring services built into those services).

In the future we're looking for aggregating APIs that let us express our unique points of difference (e.g. flexible sizing of VPSs in our case, public/private IPs in the case of other providers) but are implemented so that clients can rely on a basic level of functionality from all implementing providers. Perhaps a fallout of this is that we may end up with some 'simple' APIs and some more expressive APIs (which may in fact be further 'wrapped' to simplify them).
-------------------------------------

If Peter's looking for ways to express unique value, Mic [6] hopes cloud aggregator apis can help prevent these differences from becoming lock-in.

-------------------------------------
I view cross cloud apis as having 2 key benefits:

1) for the internal IT developer and "tech consumer" organisations: they avoid unnecessary lockin to specific cloud vendors
2) for the ISV - they have an API to target that also avoids locking them into a cloud vendor.

I view this as very similar to the ye olde app server days of yesteryear (the 90s) when there were many proprietary APIs, yet things coalesced into some standard apis (J2EE, .Net, PHP etc) which both meant that applications were portable between middleware vendors and OSes (to some extent) and also often tended towards open source. In any case the end aim is avoiding unnecessary lock-in due to proprietary APIs being used (sometimes accidentally) when functionality is in-fact more in-common than unique.
-------------------------------------

So, is this history all over again? Are these apis valuable? Is it time for standards or points of difference?

You'll have to decide this for yourself. There are multiple correct answers.

What are these projects? and what do they do about these concerns?

Next blog, I'll review the latest from the apis mentioned, and maybe even a couple more.

Thanks for reading!

Adrian Cole
founder jclouds
http://code.google.com/p/jclouds/