What is the “Unit” of Cloud Computing? Virtual Machines, Virtual Networks, and Virtual Data Centers

October 21, 2009

This is a post that summarizes some conversations that Stuart Bailey (from Infoblox) and I have been having.

There is a lot of market clutter today about cloud computing and it can be challenging at times to identify the core technical issues. Sometimes it is helpful with an emerging technology to ask the question: “What is the ‘unit’ of deployment for the technology?” There are two important related questions: “How are the units named?” “How do the units communicate?”

Sometimes the perspective matters.

Sometimes the perspective matters.

Before we think about the answers for cloud computing, let’s warm up with some other examples.

  • For the web, the “unit” is the web page, web pages are identifid by URLs (or URIs), and the units “communicate” using HTTP and related protocols. Of course, web pages aggregate into web sites.
  • In networking, the “unit” is the IP address (at Layer 3) or the MAC address (at Layer 2) and DNS is the link between URLs and IP addresses (allowing them to communicate), while ARP (or NDP in IPv6) is the link between MAC addresses and IP addresses.
  • In grid computing, the “unit” is a computer in a cluster (“a grid resource”) and computers commnicate using the Message Passing Interface (MPI).

Depending upon your perspective and your role in the cloud computing eco-system, you could argue that any of the following are the units:

Infrastructure Perspective

  • A virtual machine (VM).
  • A virtual network (VN), consisting of multiple VMs and all required information to network the VMs.
  • A virtual data center (VDC), consisting of one or more VNs.

Data/Content/Resource Perspective

  • An identifier specifying the name of a resource for a cloud storage service. Examples include an object managed by Amazon’s S3 service, or a file managed by the Hadoop Distributed File System (HDFS).
  • An identifier specifying the name of a data resource for a cloud data service. Examples include a domain (database table) manged by Amazon’s SimpleDB service or a table (or row) manged by a BigTable-like service.

Once we take this point of view, a number of issues become much easier to discuss.

Intercloud Protocols. Today with clouds, we are in the same situation that networking was before Internet protocols enabled internetworking by supporting communication between networks. Until TCP and related Internet protocols were developed, there were not agreed upon standards identifying the appropriate entities and layers nor for passing names of entities between layers. We can ask what are the appropriate mechanisms for naming VMs, VNs and VDCs, as well as cloud and tables services, how do we pass the names of objects between layers, and how do the objects in the infrastructure stack communicate with objects in the data stack.

Virtual networks also count. Most of the cloud virtualization discussion today focuses on VMs and their migration, but it is just as essential to support VNs and their migration. If we look to how IP addresses arose, then it is tempting to think about using names for VMs that include information about VNs. Today, depending upon the units we feel are important, we will need layers in the cloud for naming and linking VMs, VNs and VDCs, not just VMs.

Removing the distinction between clouds and large data clouds. There are two fundamentally different approaches to cloud services for storage or data. In the first, there is an implicit assumption that the storage or data service must fit in a single VM (S3) or other device (such as NAS). In the second, the whole point is to develop cloud storage and data services that span multiple VMs and devices (Google’s GFS/MapReduce/BigTable), Hadoop HDFS/MapReduce, Sector Distributed File System/Sphere UDFs, etc.).

Services that link virtual infrastructure and data. In many discussions, no effort is made to span the virtual infrastructure perspective entities (VMs, VNs) with the data perspective. One simple approach is to provide a dynamic infrastructure service so that data/content/resource services could easily determine which VMs and VNs support their service (there is usally done with static configuration files today). With this approach, large data cloud services are simply data/content/resource services that are engineered to scale to multiple VMs (and perhaps VNs).

Scaling to services to data centers. One of attributes that I think is a core attribute of certain types of clouds, is for a service to scale beyond a single machine or VM to an entire data center or VDC. Defining these types of scalable services is something that is relatively easy to do from the perspective here.

Acknowledgements: The photograph is from the Flickr photostream of bourget_82 and was posted with a Attribution-No Derivative Works 2.0 Generic Creative Commons License.