Web Serving

Web serving is a fundamental application in any Internet-based service. We use CloudStone in CloudSuite to benchmark Web 2.0 applications. CloudStone is a Web 2.0 benchmark, which includes a Web 2.0 social-events application (Olio) and a client implemented using the Faban workload generator.
CloudStone can be used to benchmark various web technologies and web server software stacks. Below, we describe the procedure to set up a CloudStone version based on the Nginx web server, the MySQL database and the PHP implementation of Olio.

Prerequisite Software Packages

  1. MySQL v5.5
  2. Nginx v1.0
  3. PHP v5.3
  4. APC v3.1
  5. Faban v1.0
  6. Olio v0.2
  7. Tomcat v6.0

Download the Web Serving Benchmark.

You need JDK, GCC, and Apache Ant installed on all machines. The JAVA_HOME enviroment variable must point to the JDK installation path. We use OpenJDK 1.6.

Benchmark Components

The benchmark consists of three main components: A web server, a database backend and a client to emulate real world accesses to the web server, each running on a separate machine (Client and backend can run on the same machine). The following instructions provide the necessary details to install the benchmark. We provide the instructions to install the three main components and prepare the applications for benchmarking. Please note that CloudStone provides the option to add an additional layer of memcached machines. However, in CloudSuite, we focus on benchmarking the web server behavior using this benchmark and we do not setup a memcached server.

Setting up the Client Machine

The following instructions will help the user set up the Faban driver which is used to benchmark the Olio application. We start by the client machine because we will need Faban installation on the backend and frontend. Faban must be installed on all nodes on the same absolute path. It has a single master and several agents. The Faban master must run on the client machine. You just need to set up the Faban master and then copy it to the backend and frontend machines, where it will be used as an agent.

Setting up the Faban driver
  1. Use the Faban kit in our web benchmark package or download it from link.
  2. Untar faban-kit-022311.tar.gz in a directory on the client node. A subdirectory named faban will be created which we will refer to it as $FABAN_HOME.
    tar xzvf faban-kit-022311.tar.gz
    export FABAN_HOME=/path/to/faban
    Note: More details on Faban installation can be found at link.
  3. Download a copy of Olio's PHP implementation from link by selecting apache-olio-php-src-0.2.tar.gz or use the copy included in our web benchmark package.
  4. Untar apache-olio-php-src-0.2.tar.gz in a directory. This creates an apache-olio-php-src-0.2 subdirectory, which we will refer to it as $OLIO_HOME.
    tar xzvf apache-olio-php-src-0.2.tar.gz
    export OLIO_HOME=/path/to/apache-olio-php-src-0.2
    Note: Detailed installation and configuration instructions for Olio can be found at link.
  5. Download MySQL Connector/J (JDBC Driver for MySQL) and unpack it (or use the one provided in our package). Copy the mysql-connector-java-5.0.8-bin.jar file into $OLIO_HOME/workload/php/trunk/lib.
    tar -xzvf mysql-connector-java-5.0.8.tar.gz
    cp mysql-connector-java-5.0.8/mysql-connector-java-5.0.8-bin.jar $OLIO_HOME/workload/php/trunk/lib
  6. cd $FABAN_HOME/
  7. cp samples/services/ApacheHttpdService/build/ApacheHttpdService.jar services
  8. cp samples/services/MysqlService/build/MySQLService.jar services
  9. cp samples/services/MemcachedService/build/MemcachedService.jar services
  10. cd $OLIO_HOME/workload/php/trunk
  11. cp build.properties.template build.properties
  12. Edit build.properties and set faban.home to $FABAN_HOME and faban.url to http://IP.address.of.client_machine:9980, where the Faban master will run.
  13. ant deploy.jar
  14. cp $OLIO_HOME/workload/php/trunk/build/OlioDriver.jar $FABAN_HOME/benchmarks
  15. Start the Faban master on the client machine:
    $FABAN_HOME/master/bin/startup.sh
  16. Point your browser to: http://IP.address.of.client_machine:9980. You should see the OlioWorkload welcome note.
    Note: This step is necessary because it will unpack the content of OlioDriver.jar which will be required later.
  17. Copy the faban directory ($FABAN_HOME) to the frontend, backend, and client machines. Faban directories must be in the same path on every machine.

Setting up the Backend Machine

Setting up MySQL

The following instructions assume a Linux-based installation. For more detailed instructions, please refer to link.

Note: You need to have a copy of MySQL Community Server. You can find it in our web benchmark package or download it from link.
You need to select the Linux platform, and then, download the Linux-generic, x86_64-bit, compressed tar archive.

Here are the installation commands:
(You need root privileges to execute some of the commands below)

  1. Install the libaio1 (e.g., sudo apt-get install libaio1)
  2. groupadd mysql
    (addgroup and adduser commands might differ in different Linux versions)
  3. useradd -r -g mysql mysql
  4. tar xzvf mysql-5.5.20-linux2.6-x86_64.tar.gz
  5. chown -R mysql mysql-5.5.20-linux2.6-x86_64
  6. chgrp -R mysql mysql-5.5.20-linux2.6-x86_64
  7. cd mysql-5.5.20-linux2.6-x86_64
  8. cp support-files/my-medium.cnf /etc/my.cnf
  9. scripts/mysql_install_db --user=mysql
  10. To start mysql, in the installation directory, run:
    bin/mysqld_safe --defaults-file=/etc/my.cnf --user=mysql &
Setting up and populating the database
  1. Run MySQL from the mysql installation directory:
    • bin/mysql -uroot
  2. Create the Olio user:
    • mysql> create user 'olio'@'%' identified by 'olio';
    • mysql> grant all privileges on *.* to 'olio'@'localhost' identified by 'olio' with grant option;
    • Note: This command grants privilege to an olio user running on the local host. In CloudStone, the olio users running on all frontend machines need the same privileges:
    • mysql> grant all privileges on *.* to 'olio'@'ip.address.of.frontend' identified by 'olio' with grant option;
  3. Crate Olio database and tables:
    • mysql> create database olio;
    • mysql> use olio;
    • mysql> \. $FABAN_HOME/benchmarks/OlioDriver/bin/schema.sql
    • mysql> exit
  4. Populate the database:
    • cd $FABAN_HOME/benchmarks/OlioDriver/bin
    • chmod +x dbloader.sh
    • ./dbloader.sh <dbserver> <load_scale>
      dbserver is the address of the database server (e.g., localhost) and load_scale is the scaling factor. This number shows the maximum number of concurrent users that can be simulated. The database will be loaded with 100*load_scale users. For information on the scaling factor to populate the database, refer to link.
Setting up Tomcat
  1. Download Tomcat or use the copy in our package. In the current CloudSuite distribution, we are using Tomcat v6.0.35.
  2. Untar the package. We refer to the root folder of Tomcat as CATALINA_HOME.
    tar xzvf apache-tomcat-6.0.35.tar.gz
    export CATALINA_HOME=/path/to/apache-tomcat-6.0.35
  3. cd $CATALINA_HOME/bin
  4. tar zxvf commons-daemon-native.tar.gz
  5. cd commons-daemon-1.0.7-native-src/unix/
  6. ./configure
  7. make
  8. cp jsvc ../..
Setting up the Geocoder Emulator

The Geocoder Emulator is a servlet deployed on Tomcat and required by CloudStone.

  1. Copy $OLIO_HOME/geocoder from the client machine to the backend machine in a directory which we will refer to it as $GEOCODER_HOME.
    export GEOCODER_HOME=/path/to/a/directory
    mkdir $GEOCODER_HOME
    scp -r IP.address.of.client_machine:$OLIO_HOME/geocoder $GEOCODER_HOME
  2. cd $GEOCODER_HOME/geocoder
  3. cp build.properties.template build.properties
  4. Edit build.properties and set the servlet.lib.path to point to the $CATALINA_HOME/lib directory.
    servlet.lib.path=/usr/local/apache-tomcat-6.0.13/lib
  5. ant all
  6. cp dist/geocoder.war $CATALINA_HOME/webapps
  7. Start Tomcat:
    $CATALINA_HOME/bin/startup.sh

Setting up the Frontend machine


Setting up the web application
  1. Create a directory which will be used as the root of the web application. You can use the root directory of the web server (e.g., /var/www) to install the web application. We will call this directory $APP_DIR.
    export APP_DIR=/path/to/webapp/root
    mkdir -p $APP_DIR
  2. Download a copy of Olio's PHP implementation from link by selecting apache-olio-php-src-0.2.tar.gz or use the copy included in our web benchmark package.
  3. Untar apache-olio-php-src-0.2.tar.gz in a directory. This creates an apache-olio-php-src-0.2 subdirectory, which we will refer to it as $OLIO_HOME.
    tar xzvf apache-olio-php-src-0.2.tar.gz
    export OLIO_HOME=/path/to/apache-olio-php-src-0.2
  4. cp -r $OLIO_HOME/webapp/php/trunk/* $APP_DIR
  5. We created a patch for the CloudStone which includes a few bug fixes and resolves compatibility issues with PHP 5.3. The patch is available in our package with the name cloudstone.patch. Copy this patch to $APP_DIR and apply it to the Olio installation:
    cp cloudstone.patch $APP_DIR
    cd $APP_DIR
    patch -p1 < cloudstone.patch
  6. You need to use $APP_DIR/etc/php.ini as the setting for your PHP installation (see below) or copy the settings appropriately to the php.ini for your installation (e.g., /usr/local/lib). We will use $APP_DIR/etc/php.ini through the rest of the document.
  7. Edit $APP_DIR/etc/config.php:
    • Set the database host name:
      $olioconfig['dbTarget'] = 'mysql:host=IP.address.of.dbserver;dbname=olio';
    • Comment out the 'MemCached' component and uncomment the 'NoCache' line:
      //$olioconfig['cacheSystem'] = 'MemCached';
      $olioconfig['cacheSystem'] = 'NoCache';
    • Edit the geocoderURL parameter and replace GEOCODER_HOST:8080 with the actual host address and port number, where Tomcat is running (i.e., the backend machine).
      $olioconfig['geocoderURL'] = 'http://IP.address.of.backend:8080/geocoder/geocode';

Installing Nginx
  1. The following dependencies are required, make sure to install them before you continue:
    libpcre3 libpcre3-dev libpcrecpp0 libssl-dev zlib1g-dev (Depending on the distribution the names might be pcre, pcre-devel, ...)
  2. You need the Nginx web server which is included in our web benchmark package. Alternatively, you can download it from here.
  3. tar zxvf nginx-1.0.11.tar.gz
  4. cd nginx-1.0.11
  5. ./configure
    Note: You can change the installation directory by specifying --sbin-path=/path in the configure command.
  6. make
  7. sudo make install
  8. To test the installation:
    • Start the server: sudo /usr/local/nginx/sbin/nginx
    • Point your browser to: http://IP.address.of.webserver. You should see a welcome note.
    • Stop the server: sudo /usr/local/nginx/sbin/nginx -s stop
  9. Edit the Nginx configuration file (e.g., /usr/local/nginx/conf/nginx.conf) and make the following changes:
    • Under location /, set the root parameter to $APP_DIR/public_html and append index.php to the index parameter:
      location / {
      root $APP_DIR/public_html;
      index index.html index.htm index.php;
      }
    • Set the correct port number to access PHP-FPM by adding the following lines under the server section:
      location ~ \.php$ {
      root $APP_DIR/public_html ; # (e.g., /var/www/public_html );
      fastcgi_pass 127.0.0.1:9000;
      fastcgi_index index.php;
      fastcgi_param SCRIPT_FILENAME $APP_DIR/public_html/$fastcgi_script_name; # (e.g., /var/www/public_html/$fastcgi_script_name)
      include fastcgi_params;
      }
    • Turn the access log off by adding the following line under the server section:
      access_log off;
    • To saturate the web server you might need to increase the maximum number of connections. As a rough estimate, 700-1000 connections per core should be able to saturate the web server.
      The number of concurrent connections to Nginx is controlled by two parameters in the nginx configuration file: worker_processes and worker_connections. The worker_processes parameter dictates the number of child processes, which will be forked, and worker_connections dictates the maximum number of connections per child process. The total number of concurrent connections is the multiplication of these two parameters.
  10. Start the server: sudo /usr/local/nginx/sbin/nginx
Installing PHP

The installation is made to support FPM and the libraries required by Olio. You can use the PHP version which is available in our release or download it from here.

  1. Install the following dependencies:
    libxm12-dev curl libcurl3 libcurl3-dev libjpeg-dev libpng-dev
    (e.g., sudo apt-get install libxm12-dev curl libcurl3 libcurl13-dev libjpeg-dev libpng-dev)
  2. To allow the usage of the MySQL functionality, you should point to the path of an uncompressed MySQL distribution (i.e., the one used in the backend). You must untar it and use the path in the following command. No installation of MySQL is needed.
  3. tar zxvf php-5.3.9.tar.gz
  4. cd php-5.3.9
  5. ./configure --enable-fpm --with-curl --with-pdo-mysql=/path/to/mysql --with-gd --with-jpeg-dir --with-png-dir --with-config-file-path=/path/to/php.ini
    Warning: --with-config-file-path argument should only include the path ($APP_DIR/etc/), not the file name (php.ini) in the end.
  6. make
  7. sudo make install
  8. Set an environment variable named PHPRC pointing to the directory, which contains php.ini. You should put a command in your shell startup script to automatically do that:
    export PHPRC=/path/to/php.ini-directory (e.g., export PHPRC=$APP_DIR/etc/).
  9. Make the tmp/http_sessions directory which is specified by php.ini to store PHP sessions and give read/write permission to everyone:
    mkdir -p /tmp/http_sessions
    chmod 777 /tmp/http_sessions
  10. Edit the php.ini file (e.g., $APP_DIR/etc/php.ini) and make the following changes:
    • Add the following line which declares the extensions directory:
      extension_dir=/path/to/php/lib/php/extensions/no-debug-non-zts-20090626/
    • Add a timezone:
      date.timezone = "Europe/Zurich"
    • Disable error reporting for PHP notices (or all errors). These lines should already exist in the php.ini and you need to modify them:
      error_reporting = E_ALL & ~E_NOTICE ; or error_reporting = E_NONE
      display_errors = Off
  11. To run Nginx with PHP support, the PHP-FPM module must be started separately:
    • sudo cp /usr/local/etc/php-fpm.conf.default /usr/local/etc/php-fpm.conf
    • sudo addgroup nobody # (or you can modify the php-fpm.com file to set the user and group to the ones you are already using)
    • sudo /usr/local/sbin/php-fpm
  12. The number of processes created by the PHP-FPM can be controlled by setting Process Manager parameters in the php-fpm.config file. These parameters are prefixed with pm.. See the php-fpm.conf file for a description of these parameters. Modification to these parameters requires a restart of PHP-FPM to take effect.
Installing APC

You can use the APC (Alternative PHP Cache) copy in our web benchmark package or download it from link, then run the following commands for installation:

  1. Install the following dependency: autoconf (e.g., sudo apt-get install autoconf)
  2. tar xzvf APC-3.1.9.tgz
  3. cd APC-3.1.9
  4. phpize
  5. ./configure --enable-apc --enable-apc-mmap --with-php-config=/path/to/php-config # (e.g., /usr/local/bin/php-config)
  6. make
  7. sudo make install
  8. Restart PHP-FPM. To make sure APC is enabled, you can check the "php-fpm -m" output. There should be "apc" listed as a module in the output.
Setting up the Filestore

Filestore is a directory on the local file system, which is used to store the data required for the web application. The required space depends on the load_scale and might reach tens of gigabytes.

  1. We created a patch (cloudsuite.patch) to deal with Filestore large space requirement. When this patch is applied, Olio accesses a limited set of files on the disk. Use of this patch is optional.
    cd $APP_DIR
    patch -p1 < cloudsuite.patch
  2. Create a directory referred to as $FILESTORE and give read/write permissions to everyone:
    export FILESTORE=/path/where/files/will/be/stored
    mkdir -p $FILESTORE
    chmod a+rwx $FILESTORE
  3. chmod +x $FABAN_HOME/benchmarks/OlioDriver/bin/fileloader.sh
  4. Load the Filestore:
    $FABAN_HOME/benchmarks/OlioDriver/bin/fileloader.sh <load_scale> $FILESTORE
    load_scale is the same value used when loading the database.
    Note: If you used the cloudsuite.patch then this value must be set to 102 regardless of the database load_scale.
  5. Edit $APP_DIR/etc/config.php and set the value of the localfsRoot parameter to the path which is pointed by $FILESTORE.
    $olioconfig['localfsRoot'] = '$FILESTORE'; // (e.g., $olioconfig['localfsRoot'] = '/filestore';)
  6. Restart PHP-FPM.

Running the benchmark

  1. Make sure that the Faban master, the MySQL server, Nginx, and PHP-FPM are up and running.
  2. Point your browser to http://IP.address.of.client_machine:9980.
  3. Click on the 'Schedule Run' link.
  4. Enter $JAVA_HOME of the client machine in the 'Java Home' field under the Java tab.
  5. Select the 'Driver' tab and set the configuration parameters as follows:
    • Enter the driver (client) machine address(es) in the 'Hosts' field.
    • Enter the number of 'Concurrent Users'. It defines the number of concurrent users that send request to the web server. The configuration for the web and database servers must allow that many concurrent users.
    • Enter vmstat 10 in the 'Tools' field.
    • Set the proper values for 'Ramp Up', 'Steady State' and 'Ramp Down' in seconds.
  6. Select the 'Web Server' tab.
    • Enter the web server address in the 'Host' field.
    • For the Web Server 'Type' field, select empty.
    • Clear the 'Web Server command path' field.
    • Enter the name of the monitoring tools in the 'Tools' field (separate them by semicolon): vmstat 10; mpstat 10; nicstat 10; iostat -x 10
    • Enter the log (/usr/local/nginx/logs) and config (/usr/local/nginx/conf/nginx.conf) paths, and the path containing the php.ini file ($APP_DIR/etc/php.ini) in the respective fields.
  7. Select the 'Data Servers' tab.
    Under 'Database Servers':
    • Enter the database server address in the 'Host' field.
    • In 'JDBC connection URL' replace dbHost with the database server address and set the correct user name (olio) and password (olio) to connect to the database.
    • Select No in the 'Reload Database' drop-down list.
    • In the 'Load for Concurrent Users' field, enter the number of users you populated the database for (i.e., load_scale).

    Under 'Data Storage Server':
    • Enter the frontend server address in the 'Host' field.
    • Enter the $FILESTORE path in the 'Media Directory' field.

    Under 'Memcached Servers':
    • Clear the 'Host:Port Pairs' field.
  8. Click OK to start the run.
  9. You can monitor the run and see the results by clicking on the 'View Results' link.


© EPFL PARSA 1015 Lausanne, Switzerland tel. +41 21 693 1395 all rights reserved       webmaster