ISPASS-2014 Tutorial: Rigorous and Practical Server Design Evaluation

  • Where? In conjuction with ISPASS'14 in Monterey, CA, USA.
  • When? Sunday afternoon, March 23rd, 2014 (a half-day tutorial).
  • Who? Academic/industrial researchers interested in scale-out datacenter applications, server design, and their rigorous and practical performance evaluation via full-system cycle-accurate simulation.
  • Keywords: Full-system simulation, sampling, measurement methodology, scale-out workloads, CloudSuite.
  • Registration: Please follow the registration link on the conference web page.

Tutorial In Brief

The emergence of cloud computing as a dominant computing platform highlights the need for practical and rigorous architectural evaluation of server systems. Such evaluation mandates the use of a variety of real-world server workloads, all of which are radically different from traditional desktop and scientific benchmarks. Unfortunately, deep and complex software stacks of both conventional (e.g., OLTP, DSS) and emerging scale-out (e.g., Media Streaming, Web Search) server workloads make the evaluation process even harder and slower, postponing the adoption of realistic server benchmarks within the architectural community.

CloudSuite is an on-going effort towards a common architectural evaluation basis, aimed at providing an up-to-date suite of benchmarks that represent popular scale-out cloud applications commonly found in today�s datacenters. The latest version, CloudSuite 2.0, comprises of eight scale-out applications that feature real-world server software stacks and datasets that can be used as realistic benchmarks for the scale-out applications class. These applications are data analytics, data caching, data serving, graph processing, media streaming, software testing, web search, and web serving. CloudSuite provides the means for throughput and quality of service (QoS) measurement for each benchmark.

In this tutorial, we first introduce CloudSuite 2.0 and demonstrate how to run the benchmarks on real hardware. As evaluating server software stacks via full-system simulation is prohibitively slow, a need for statistical sampling emerges. A rigorous sampling methodology is of utmost importance, because improper sampling can lead to highly misleading results. Devising the right sampling methodology for server workloads is not trivial. For that reason, our tutorial covers the SMARTS sampling methodology, a widely applicable framework that enables fast cycle-accurate simulation of both conventional and emerging server workloads on various full-system architectural simulators. We demonstrate how the SMARTS methodology can be applied in today�s simulators through a case study; we describe the complete simulation process using Flexus, a full-system multiprocessor simulator that leverages the statistical sampling methodology for fast, yet rigorous evaluation of CloudSuite 2.0 benchmarks.

Detailed Program

The emergence of cloud computing as a dominant computing platform highlights the need for architectural evaluation of cloud server systems. Moreover, the fast emergence of new applications and techniques to manipulate the massive amounts of data generated by the popular cloud applications mandates a continuous update to existing benchmarks and architectural evaluation infrastructure. In this tutorial, we will present CloudSuite 2.0, the latest release of CloudSuite that includes popular open-source benchmarks. We will highlight the key differences between cloud applications and traditional benchmarks used for architectural evaluation. Because evaluating the full software stack of modern cloud applications using existing cycle-accurate software simulators is prohibitively slow, sampling the execution is essential. We will introduce the SMARTS statistical sampling methodology, which enables fast and rigorous simulation of complex systems running real-world server software, such as CloudSuite. We will further demonstrate how the SMARTS methodology can be seamlessly integrated into today's simulators, using the Flexus simulation infrastructure as an example.

CloudSuite is an on-going effort that aims to provide an up-to-date suite of benchmarks that represent popular scale-out applications. CloudSuite 2.0 is the second release of CloudSuite and includes new benchmarks that represent existing trends in manipulating massive data, such as data caching and graph analytics. CloudSuite 2.0 is based on real-world full-system software stacks and datasets and takes into account quality of service (QoS) performance metrics.

We provide support for running CloudSuite 2.0 both on real hardware and within architectural simulators, such as Flexus. Simics images that are compatible with the Flexus infrastructure are available for download. The Flexus component-based design allows for the composition of complex multicore systems, enabling rapid evaluation of the architectural behavior of cloud applications on a wide range of processor organizations and microarchitectures. Flexus achieves fast simulation turnaround while ensuring representative results by leveraging the SMARTS simulation sampling methodology, which will be thoroughly covered in this tutorial.

Our tutorial is organized as follows. First, we present an overview of CloudSuite 2.0, providing details on the software stacks constituting each benchmark, and steps to run the benchmarks on real hardware. The second session of our tutorial makes the case for careful sampling-based simulation of CloudSuite applications, which is essential for making detailed simulation of reasonable execution windows feasible. We present the relevant sampling background and explain how the SMARTS methodology can be applied to enable fast and accurate evaluation of systems running CloudSuite regardless of the underlying simulation infrastructure. We proceed with an overview of Flexus as a case study of a cycle-accurate full-system simulator that applies statistical sampling to allow for fast simulation of complex multicore systems running real server benchmarks. We demonstrate Flexus' anatomy, how to use the preconfigured multicore organizations and microarchitectures, and how to add and simulate new components.

(Tentative) Agenda

Time SlotTopicMaterial
1:30pm - 2:15pmCloudSuite 2.0Overview of CloudSuite 2.0 benchmarks and running them on real hardware
2:15pm - 2:45pm CloudSuite Full-System Simulation Introduction to Flexus and its interaction with the Simics full-system simulator
2:45pm - 3:15pmCoffee Break
3:15pm - 3:45pmFast and Detailed Simulation Using FlexusDemonstrating Flexus internals and how to perform detailed microarchitectural simulation
3:45pm - 4:30pmStatistical SamplingMaking detailed simulation fast using SMARTS sampling methodology
4:30pm Wrap-up


    Onur Kocberber is a fifth-year PhD candidate in the Parallel Systems Architecture Laboratory at EPFL, advised by Prof. Babak Falsafi. Onur's research focuses on design for dark silicon and efficient server architectures for large-scale datacenters.
© 2014 EPFL PARSA 1015 Lausanne, Switzerland tel. +41 21 693 1395 all rights reserved       webmaster