Software testing as a Service
There are many applications that can potentially benefit from the abundance of resources in clustered systems. Software testing is a resource-hungry and time-consuming task that can leverage cloud computing.
We benchmark Cloud9, an automated software-testing platform that parallelizes symbolic execution and scales on clusters of commodity hardware. The Cloud9 software that we benchmark has been developed by DSLab at EPFL. You can get more information on the Cloud9 website.
Prerequisite Software Packages
Cloud9 was tested extensively on Ubuntu10.4 Server Edition 64-bit and Scientific Linux 6.0 64-bit and is expected to work on any other 64-bit Linux distribution. To run the software properly, you need the following packages:
- gnu g++
Download the software-testing benchmark.
Download a Cloud9 copy either from the benchmark suite package that includes all the required components or you can download it from the original source. Although the Cloud9 website provides detailed instructions for installing and running Cloud9, here, we provide a summary of the required steps to run the software.
The instructions for installing the benchmark:
- Building LLVM
- You need the GCC front-end binaries. You can obtain them either from the benchmark package (unzip llvm-gcc4.2-2.9-x86_64-linux.tar.bz2.tar.bz2.tar.bz2), or from the Cloud9 website, or from the official LLVM webpage.
- Add 'llvm-gcc4.2-2.9-x86_64-linux/bin' to your path.
- Download the LLVM 2.9 source either from the benchmark package (unzip llvm-2.9.tar), or the Cloud9 website, or the official LLVM repository "svn co https://firstname.lastname@example.org/svn/llvm-project/llvm/trunk llvm".
- Inside the 'llvm-2.9' folder run './configure --enable-optimized --enable-assertions'.
- Run 'make'.
- Add the resulting 'llvm-2.9/Release+Asserts/bin' directory to your path.
- Building KLEE
- Use the modified KLEE C library uclib in the Cloud9 package (unzip klee-c9-uclibc.tar).
- Run './configure --with-llvm=[Your 'llvm-2.9' path]'.
- Edit the file Rules.mak, change this "LLVMTOOLDIR = $(LLVMROOTDIR)/Release/bin" to "LLVMTOOLDIR = $(LLVMROOTDIR)/Release+Asserts/bin".
- Edit the file Rules.mak.llvm, change this "LLVMTOOLDIR = $(LLVMROOTDIR)/Release/bin" to "LLVMTOOLDIR = $(LLVMROOTDIR)/Release+Asserts/bin".
- Run 'make'.
- Building Cloud9
- Inside the Cloud9 directory run './configure --with-llvm=[The 'llvm-2.9' path] --with-uclibc=[The 'klee-c9-uclibc' path] --enable-posix-runtime --enable-optimized'.
- Run 'make'.
Running Cloud9 Single Instance
We are going to run an experiment on a single node using one core (one Cloud9 instance).
- Go to the Cloud9 'cloud9/Release+Asserts/bin' folder.
- Run './c9-worker --stand-alone --posix-runtime --libc=uclibc -coverable-modules=[The 'cloud9/infra' path]/coverable/coreutils.coverable [The 'cloud9/targets' path]/src/printf.bc --sym-args 0 2 5'.
- We can actually run all the coreutils binaries that are inside the folder targets/src.
* The instructions for a single instance (c9-worker) of Cloud9 is for ensuring that the program has been installed correctly and it runs without crashing. The instructions for running Cloud9 on a Cluster let you configure a script that will manage several instances (c9-worker) of Cloud9 across one or several nodes.
Running Cloud9 on a Cluster
- Install Cloud9 on all the machines you will use. There are two types of machines: the workers and the load balancer. Cloud9 has to be installed on all of them. However, the following steps are required to be done only on the load balancer. Following this terminology, the local host is the load balancer and all the other hosts are the workers. Note that the machines can be either single or multi-core machines. Multi-core machines can run multiple independent Cloud9 worker tasks.
- Create a .hosts file under the cloud9/infra/hosts/ directory (e.g., epfl.hosts or example.hosts) and specify:
- The machines you will use including the local host.
- The number of cores available on each machine (0 for the local host).
- The Cloud9 path on each machine.
- The SSH username used to connect to the machines, not required for the local host.
- A directory for the logs (it must exist).
- The path where the targets folder is. In our case it is 'cloud9/targets'. This is not required for the local host.
Note that the Cloud9 distribution comes with a sample configuration file as a starting point.
./run-experiment.py -t [time in seconds] --strategy random-path,cov-opt epfl coreutils coreutils-12 coreutils coreutilsIn this example, the command uses the following files defined above: epfl.hosts, coreutils.cmdlines, coreutils-12.exp, coreutils.cmdlines, coreutils.coverable.
To evaluate Cloud9, we consider the time needed to reach a certain, fixed coverage level. For instance, the time required to cover 80% of the code. When running Cloud9 on a single node, the coverage is plotted automatically on the screen. On the other hand, when running Cloud9 on a cluster of machines, the amount of code covered is reported inside the log folder of the local host. Due to the random-path search strategy, we recommend to run Cloud9 several times using the same input to increase the confidence levels of the evaluation.