Software testing as a Service

There are many applications that can potentially benefit from the abundance of resources in clustered systems. Software testing is a resource-hungry and time-consuming task that can leverage cloud computing.

We benchmark Cloud9, an automated software-testing platform that parallelizes symbolic execution and scales on clusters of commodity hardware. The Cloud9 software that we benchmark has been developed by DSLab at EPFL. You can get more information on the Cloud9 website.

Prerequisite Software Packages

Cloud9 was tested extensively on Ubuntu10.4 Server Edition 64-bit and Scientific Linux 6.0 64-bit and is expected to work on any other 64-bit Linux distribution. To run the software properly, you need the following packages:

  • gnu g++
  • dejagnu
  • flex
  • bison
  • protobuf-compiler
  • libprotobuf-dev
  • libboost-thread-dev
  • libboost-system-dev
  • python-argparse

Download the software-testing benchmark.

Installing Cloud9

Download a Cloud9 copy either from the benchmark suite package that includes all the required components or you can download it from the original source. Although the Cloud9 website provides detailed instructions for installing and running Cloud9, here, we provide a summary of the required steps to run the software.

The instructions for installing the benchmark:

  1. Building LLVM
    • You need the GCC front-end binaries. You can obtain them either from the benchmark package (unzip llvm-gcc4.2-2.9-x86_64-linux.tar.bz2.tar.bz2.tar.bz2), or from the Cloud9 website, or from the official LLVM webpage.
    • Add 'llvm-gcc4.2-2.9-x86_64-linux/bin' to your path.
    • Download the LLVM 2.9 source either from the benchmark package (unzip llvm-2.9.tar), or the Cloud9 website, or the official LLVM repository "svn co https://user@llvm.org/svn/llvm-project/llvm/trunk llvm".
    • Inside the 'llvm-2.9' folder run './configure --enable-optimized --enable-assertions'.
    • Run 'make'.
    • Add the resulting 'llvm-2.9/Release+Asserts/bin' directory to your path.
  2. Building KLEE
    • Use the modified KLEE C library uclib in the Cloud9 package (unzip klee-c9-uclibc.tar).
    • Run './configure --with-llvm=[Your 'llvm-2.9' path]'.
    • Edit the file Rules.mak, change this "LLVMTOOLDIR = $(LLVMROOTDIR)/Release/bin" to "LLVMTOOLDIR = $(LLVMROOTDIR)/Release+Asserts/bin".
    • Edit the file Rules.mak.llvm, change this "LLVMTOOLDIR = $(LLVMROOTDIR)/Release/bin" to "LLVMTOOLDIR = $(LLVMROOTDIR)/Release+Asserts/bin".
    • Run 'make'.
  3. Building Cloud9
    • Inside the Cloud9 directory run './configure --with-llvm=[The 'llvm-2.9' path] --with-uclibc=[The 'klee-c9-uclibc' path] --enable-posix-runtime --enable-optimized'.
    • Run 'make'.

Running Cloud9 Single Instance

We are going to run an experiment on a single node using one core (one Cloud9 instance).

  1. Go to the Cloud9 'cloud9/Release+Asserts/bin' folder.
  2. Run './c9-worker --stand-alone --posix-runtime --libc=uclibc -coverable-modules=[The 'cloud9/infra' path]/coverable/coreutils.coverable [The 'cloud9/targets' path]/src/printf.bc --sym-args 0 2 5'.
  3. We can actually run all the coreutils binaries that are inside the folder targets/src.

* The instructions for a single instance (c9-worker) of Cloud9 is for ensuring that the program has been installed correctly and it runs without crashing. The instructions for running Cloud9 on a Cluster let you configure a script that will manage several instances (c9-worker) of Cloud9 across one or several nodes.

Running Cloud9 on a Cluster

  1. Install Cloud9 on all the machines you will use. There are two types of machines: the workers and the load balancer. Cloud9 has to be installed on all of them. However, the following steps are required to be done only on the load balancer. Following this terminology, the local host is the load balancer and all the other hosts are the workers. Note that the machines can be either single or multi-core machines. Multi-core machines can run multiple independent Cloud9 worker tasks.

  2. Create a .hosts file under the cloud9/infra/hosts/ directory (e.g., epfl.hosts or example.hosts) and specify:
    • The machines you will use including the local host.
    • The number of cores available on each machine (0 for the local host).
    • The Cloud9 path on each machine.
    • The SSH username used to connect to the machines, not required for the local host.
    • A directory for the logs (it must exist).
    • The path where the targets folder is. In our case it is 'cloud9/targets'. This is not required for the local host.

    Note that the Cloud9 distribution comes with a sample configuration file as a starting point.

  3. We will use the CoreUtils targets provided by the Cloud9 distribution. Therefore, you can rely on coreutils.cmdlines and coreutils.kcmd that are provided by the Cloud9 distribution for the target commands and the KLEE options respectively. The coreutils.cmdlines defines an ID that specifies the file that Cloud9 should analyze inside the folder indicated by the file .hosts and the arguments needed to run the binary. The coreutils.kcmd defines the arguments of the instances of Cloud9 on each machine (workers).

  4. In the cloud9/infra/coverable/ directory, create the file that describes the source files accounted for, when computing the code coverage (e.g., example.coverable). You may use the provided coreutils.coverable file to run Coreutils.

  5. To create an experiment schedule, we will create a .exp file in the cloud9/infra/exp/ directory (e.g., coreutils.exp or example.exp). Inside the .exp file, each line corresponds to one iteration of the experiment - and all the items in one line are executed in parallel. The items are space-separated, and have the format "[testing target id] [# workers] {[host] [# cores]}*" that describes the number of cores on a worker node allocated for a testing target, and their allocation per physical host. Note that the [testing target id] refers to the entry specified in the .cmdlines file.

  6. As the last step, we are going to perform the run. Inside the 'cloud9/infra' folder, run the following command:

    ./run-experiment.py -t [time in seconds] --strategy random-path,cov-opt epfl coreutils coreutils-12 coreutils coreutils

    In this example, the command uses the following files defined above: epfl.hosts, coreutils.cmdlines, coreutils-12.exp, coreutils.cmdlines, coreutils.coverable.

Cloud9 Evaluation

To evaluate Cloud9, we consider the time needed to reach a certain, fixed coverage level. For instance, the time required to cover 80% of the code. When running Cloud9 on a single node, the coverage is plotted automatically on the screen. On the other hand, when running Cloud9 on a cluster of machines, the amount of code covered is reported inside the log folder of the local host. Due to the random-path search strategy, we recommend to run Cloud9 several times using the same input to increase the confidence levels of the evaluation.


2010 EPFL PARSA 1015 Lausanne, Switzerland tel. +41 21 693 1111 all rights reserved webmaster web d esigned by atelier.f