Digging into Xen at EC2
During my master’s thesis I experimented with co-residence of virtual machines on the public cloud (Amazon EC2). The goal was to create a setting with co-resident VMs in which one VM offers specialized security services to another co-resident VM. Doing this in the public cloud was not academically very relevant, but a fun experiment nonetheless.
Mapping the cloud has already been done before (see Ristenpart et al.) and that Amazon leaks some information has been pointed out before. Nonetheless, there is still much more to be discovered.
Co-residence
Achieving co-residence is (of course) not endorsed by the cloud provider (in this case Amazon). The chance that two random VMs are co-resident in the cloud is very small, since the cloud consists of a huge number of physical hosts. However, if you launch n=20 VMs in a single burst (via the AWS API), there is a non-insignificant chance that at least two VMs of these are started on the same physical host.
Detecting co-residence can be done in multiple ways. For example by looking at the first hop (cf. Ristenpart et al.) or by exchanging “ping-pong” data through the Xenstore. A simple indication can be given by simply looking at the domain ID that your VMs have received. If these are sequential, then you have a good indication that these VMs might be co-resident.
Domain ID samples
During my experiments I recorded information on all the VMs which were launched in order to get co-resident VMs. The following figure gives an overview of the domain IDs that my VMs received.
This graph conveys that the domain IDs can vary quite a lot. Also shown is the Xen hypervisor version that was running on the respective hosts.
EC2 Xen versions
I encountered at least three distinct versions of Xen hypervisors in the US East region. This information is exposed by Xen via the xen_version hypercall.
| 3.1.2-128.1.10.el5 | 3.4.3-kaos_t1micro | 3.4.3.amazon | |
| Compile date | Thu Sep 2 19:09:17 SAST 2010 Tue Mar 15 19:28:19 SAST 2011 Mon Nov 21 07:15:03 SAST 2011 Fri Jun 1 17:29:43 SAST 2012 |
Sat Apr 28 07:06:40 UTC 2012 Mon May 28 23:52:08 UTC 2012 Mon Aug 27 08:26:20 UTC 2012 |
Thu Aug 24 17:00:00 UTC 2006 |
| Compiled by | amazon, builder |
mockbuild | amazon |
| Compiler | gcc version 4.1.1 20070105 (Red Hat 4.1.1-52) | gcc version 4.1.2 20080704 (Red Hat 4.1.2-48) | amazon_cc |
| Compile domain | ec2.internal | (none) | amazon.com |
| Domain-ID Median | 2438 | 424 | 258 |
From this information, we can infer a few observations.
- The variety in versions that are used seems to indicate that Amazon does not have an automated hypervisor updating scheme. This immediately raises the concern whether Amazon can patch running hypervisors against publicized vulnerabilities. It seems more likely that, unless they hotpatch their hypervisors, Amazon is running (vulnerable) outdated hypervisors on some of their machines.
- The version listed in the first column (3.1.2) lists its timestamps in South African timezones. This is a clear correlation with the fact that Amazon EC2 has its roots in South Africa.
- It has been noted before that Amazon is likely using Red Hat, and the above table shows (in the GCC strings) that the Xen packages are compiled on a Red Hat machine.
- Mockbuild (listed in column two) is a Red Hat tool typically used for building RPMs, giving additional indication that Dom0 must be running a Red Hat based Linux, too.
- In the third column we see a misleading compilation timestamp (3.4 was not out yet in 2006). All the compilation metadata for this version seems to indicate that Amazon is using a custom compilation toolchain now.
Up next: learning more from the hypervisor and the Xenstore.
These 220 measurements were taken throughout November 2012 on Amazon EC2 US-East region. All instances were micro instances.