Apache conditional websockets

If you have Apache >= 2.4 and enabled proxy_wstunnel, then this example may help to show how to redirect websockets connections to the right place. (Note that there are easier ways if you want to redirect all traffic to the websocket location. This example is for if you want to have the normal HTTP side-by-side with the websockets.)

<IfModule mod_proxy.c>
  RewriteCond %{HTTP:Upgrade} websocket [NC]
  RewriteRule /(.*)           ws://localhost:1234/$1 [P,L]

Search and replace in .docx using Python

A simple PoC to get you going:

#!/usr/bin/env python2
import zipfile as zf
import StringIO as si
import sys

candidates = ["word/document.xml"]

sio = si.StringIO()
with zf.ZipFile(sio, "w", compression=zf.ZIP_DEFLATED) as zio:
    with zf.ZipFile(sys.argv[1], "r") as za:
        for infile in za.namelist():
            indata = za.read(infile)
            if infile in candidates:
                indata = indata.replace("token", "replacement")
            zio.writestr(infile, indata)

with open("modified_{}".format(sys.argv[1]), "w") as oufile:

Spam level headers in Thunderbird

Spamassassin adds several special headers to every email that it scans. One of these is X-Spam-Level, that is, the degree of “spaminess” that the email scored.


When training your spam filters, it can be useful to see the score of each scanned email. In Thunderbird, this can easily be done by changing a setting using the config editor. In the field mailnews.headers.extraExpandedHeaders add the value X-Spam-Level.


As a bonus, to see the user agent string, you can also use the config editor by setting mailnews.headers.showUserAgent to value true. This simple flag renders a separate plugin for this functionality superfluous.

Gmail-style email address aliases with Postfix

As some might know, Google offers alternative ways to refer to Gmail email addresses.

For example, you can use the “plus” technique to make distinctions between different organizations that you share you email address with. For instance, if you own example@gmail.com, then example+nospam still delivers to the same address. Indeed, everything after the plus symbol is ignored (Gmail is simply adhering to the standard here).

One advantage of this technique is that you can hand out different email aliases to different organizations. If you start receiving spam on any of these aliases then you can trace this back to the organization which leaked your email address to spammers.

The Postfix mail transfer agent also supports such a feature. If you set

recipient_delimiter = +

then Postfix tries to deliver to example+nospam, and if this doesn’t exist, will deliver to example.

Of course, you can change this parameter. This is especially interesting if you own the whole domain name and do not need to consider other users. Changing the parameter to a dot symbol is not a bad idea, since many web forms tend to inadvertently block out the plus symbol.

Digging deeper in EC2

If you have two VMs which are co-resident on a Xen machine then the Xenstore will play a pivotal role for the inter-VM communication. To confirm that your VMs are indeed co-resident, a simple “ping-pong” through the Xenstore is a fast verification method. It can even be done using the Xenstore tools without using the API.

To find out your domain id, xenstore-read domid will read from the the domid field in your part of the Xenstore. Your part of the Xenstore is normally under /local/domain/<your domain id>/ in the Xenstore tree, and it is not possible to read the subtrees of co-resident VMs.

However, if one of the co-resident VMs cooperates, this situation changes. For instance, the simple ping-pong becomes possible if you make your VM subtree accessible for other VMs. It turns out that there is only one field in this subtree which is not owned by domain 0 and which you can change the permissions of yourself: the data field. Hence, if you do, where <ID> is the domain ID of the first VM:

 xenstore-chmod /tool/test-hugo b<ID>

Then you have shared read and write access (b<ID> is both) with all other VMs not explicitly listed. The next step is then simply to write a string there using xenstore-write and reading it using xenstore-read in the second VM. Moreover, I have used the world-readable data field extensively for creating a passtrough-encryption virtual block device.

Co-residence information leak

It is in the cloud providers interest that the cloud computing resources appear homogenous to the cloud consumers. When only looking at the Xenstore, to a certain degree this is true. It is not possible to list the /local/domain directory in the Xenstore to see which other VMs are running.

However, there is a part of the Xenstore which has not been properly closed down in all deployments of Xen in EC2. In these versions, you can list a part of the Xenstore which will reveal the Xen ID numbers of the co-resident VMs. Using the standard tools, this can be done with:

xenstore-list /local/domain/0/backend/vbd

Additionally, you can enumerate in each VM’s subdirectory to see how many Xen virtual disks they have attached (as well as the major-minor number of the disk device). The affected versions are:

  • 3.4.3-kaos_t1micro, with build timestamp: Mon May 28 23:52:08 UTC 2012
  • 3.4.3-kaos_t1micro, with build timestamp: Sat Apr 28 07:06:40 UTC 2012

Since this relates to the Xenstore daemon running in domain 0, the Xen hypervisor version as listed here should in principle be unrelated. However, the experiments in the field showed a 100% correlation of this bug exclusively to these two versions, and can thus be found to relate to how Amazon has packed their Xen packages together with their customizations. Some results that could be found using this bug are as follows.

scatter plot of co-resident VMs

This plot shows who my neighbors were and how many disks they were using.

histogram of number of co-resident VMs

Furthermore, this plot shows that there are typically 24 micro instances per physical machine in EC2.

histogram of number of disks per VM

These plots belong to a sample size of 26 instances where the leak was applicable. The number of co-resident VMs is a snapshot from the moment of deployment.

Digging into Xen at EC2

During my master’s thesis I experimented with co-residence of virtual machines on the public cloud (Amazon EC2). The goal was to create a setting with co-resident VMs in which one VM offers specialized security services to another co-resident VM. Doing this in the public cloud was not academically very relevant, but a fun experiment nonetheless.

Mapping the cloud has already been done before (see Ristenpart et al.) and that Amazon leaks some information has been pointed out before. Nonetheless, there is still much more to be discovered.


Achieving co-residence is (of course) not endorsed by the cloud provider (in this case Amazon). The chance that two random VMs are co-resident in the cloud is very small, since the cloud consists of a huge number of physical hosts. However, if you launch n=20 VMs in a single burst (via the AWS API), there is a non-insignificant chance that at least two VMs of these are started on the same physical host.

Detecting co-residence can be done in multiple ways. For example by looking at the first hop (cf. Ristenpart et al.) or by exchanging “ping-pong” data through the Xenstore. A simple indication can be given by simply looking at the domain ID that your VMs have received. If these are sequential, then you have a good indication that these VMs might be co-resident.

Domain ID samples

During my experiments I recorded information on all the VMs which were launched in order to get co-resident VMs. The following figure gives an overview of the domain IDs that my VMs received.

domid scatter plot
This graph conveys that the domain IDs can vary quite a lot. Also shown is the Xen hypervisor version that was running on the respective hosts.

EC2 Xen versions

I encountered at least three distinct versions of Xen hypervisors in the US East region. This information is exposed by Xen via the xen_version hypercall.

3.1.2-128.1.10.el5 3.4.3-kaos_t1micro 3.4.3.amazon
Compile date Thu Sep 2 19:09:17 SAST 2010
Tue Mar 15 19:28:19 SAST 2011
Mon Nov 21 07:15:03 SAST 2011
Fri Jun 1 17:29:43 SAST 2012
Sat Apr 28 07:06:40 UTC 2012
Mon May 28 23:52:08 UTC 2012
Mon Aug 27 08:26:20 UTC 2012
Thu Aug 24 17:00:00 UTC 2006
Compiled by amazon,
mockbuild amazon
Compiler gcc version 4.1.1 20070105 (Red Hat 4.1.1-52) gcc version 4.1.2 20080704 (Red Hat 4.1.2-48) amazon_cc
Compile domain ec2.internal (none) amazon.com
Domain-ID Median 2438 424 258

From this information, we can infer a few observations.

  • The variety in versions that are used seems to indicate that Amazon does not have an automated hypervisor updating scheme. This immediately raises the concern whether Amazon can patch running hypervisors against publicized vulnerabilities. It seems more likely that, unless they hotpatch their hypervisors, Amazon is running (vulnerable) outdated hypervisors on some of their machines.
  • The version listed in the first column (3.1.2) lists its timestamps in South African timezones. This is a clear correlation with the fact that Amazon EC2 has its roots in South Africa.
  • It has been noted before that Amazon is likely using Red Hat, and the above table shows (in the GCC strings) that the Xen packages are compiled on a Red Hat machine.
  • Mockbuild (listed in column two) is a Red Hat tool typically used for building RPMs, giving additional indication that Dom0 must be running a Red Hat based Linux, too.
  • In the third column we see a misleading compilation timestamp (3.4 was no out yet in 2006). All the compilation metadata for this version seems to indicate that Amazon is using a custom compilation toolchain now.

Up next: learning more from the hypervisor and the Xenstore.

These 220 measurements were taken throughout November 2012 on Amazon EC2 US-East region. All instances were micro instances.

TikZ node positioning

Drawing nodes in TikZ above each other can be tricky. For example, if you want to draw several nodes above another node in an aligned way.

One way to achieve this is as follows.

\node[minimum width=5cm] (basis) {};
\node[above=of basis.north west, anchor=south west] (a) {A};
\node[above=of basis] (b) {B};
\node[above=of basis.north east, anchor=south east] (c) {C


In this way, the left border of A and the right border of B do not extend beyond the basis node. Note that you need to have the positioning TikZ libary loaded.

To draw the edges in a vertical fashion, use the intersection shorthand for the nonstandard nodes:

\draw[->] (basis.north -| a) -- (a);
\draw[->] (basis) -- (b);
\draw[->] (basis.north -| c) -- (c);

TikZ node connector

In the series of TikZ tips, today we look at connectors between nodes.

TikZ comes with a variety of line types between nodes, such as straight, curved, and right-angled. In particular the last one, right-angled lines, can be very useful in diagrams. Right-angled lines are usually more appealing than direct point-to-point straight lines in diagrams.

Placing right-angled lines in diagrams in a visually appealing way is not always easy, however. Consider for example this diagram in which a chain of nodes have an outgoing arrow into an array.

connector needed

In this figure, the line marked B is well-suited for a right-angled line. On the other hand, the line marked with A is less-suited, since it needs two bends. For the ease of use, try using the following two macros for horizontal or vertical lines.

] (#3) -| ($(#3) !#2! (#4)$) |- (#4);
] (#3) |- ($(#3) !#2! (#4)$) -| (#4);

You can then use macros by specifying a value between 0 and 1 (where to ‘break’) and the source and destination node. For instance

\connectorH[->, red]{0.50}{a}{b}
\connectorH[->, blue]{0.75}{a}{c

Results in:


Note that you need to have the calc TikZ libary loaded.

Disabling Office 2010 shared folder synchronization in a simple way

There are various posts on how to remove the Office 2010 “Shared Folder Synchronization” context menu using either the Windows registry modifications or the Office uninstaller. I prefer a cleaner solution which seems hard to find at Google, so it’s worthwhile mentioning it here.


The approach is to unload the DLL in question. The first step is to start a command prompt as administrator.

In this command prompt, type (assuming 64-bits Office — see comments for details):

regsvr32 /u "C:\Program Files\Microsoft Office\Office14\GROOVEEX.DLL"

Pasting is not good enough, since the command prompt will not swallow the quotes. Therefore, type it, or repair the line with the up-arrow after pasting. An example of how it looks like:

After executing the command, a dialog will inform you of the result. If you get an error, double check whether you opened the command prompt as administrator correctly.


Reloading can always be done by calling regsvr32 as shown above, but without the /u flag.

(Aside: in Office 2007 this feature was called Groove folder synchronization.)

Multi-hop X-forwarding over SSH

Sometimes you’d like to do X-forwarding over multiple hops. For instance:

Home ↔ Office gateway ↔ Office machine.

One way to automate this is with this Bash function:

sshx () {
        [ -z "$2" ] && return
        sudo ssh -Nn $1 -L 2200:$2:22 &
        sleep 1s
        ssh localhost -XYC -p2200
        sudo kill $PID

Choose a random localhost port number; 2200 is merely an example here.

Then invoke with:

$ sshx passthrough_server destination_server

Since the snippet relies on Bash forking, it assumes you have set up SSH for public key based authentication with the passthrough server, and it will not ask for a password. Because SSH requires root privileges on your local machine to set up the port forwarding, your ~/.ssh/authorized_keys on the passthrough server must contains the public key of root from your local machine.