Picks and Shovels: AI Data Pipelines in the Real World

In the last decade, large-scale voice and image applications have driven phenomenal breakthroughs in deep learning algorithms that we all use in our daily interactions with internet service providers. More recently, several different industry verticals, including HPC and Science, have started to rapidly adopt similar AI approaches, but their needs extend well beyond the standard datacenter consumer applications. These new uses of AI and ML involve complex and noisy multi-sensor data, sparsely labeled ground truth, hard constraints, and complex deployment environments that span from the edge to the cloud. The relationship of ML with the real world infrastructure is intricate and goes both ways, and this talk presents two illustrative examples. In one direction, we need to design the right data pipelines to feed the data to the ML process. So, the first example discusses “infrastructure for AI/ML”: an architecture blueprint for ML in Autonomous Driving applications that spans from instrumented test cars (at the edge) to the training core (in the cloud). In the other direction, ML can help to optimize the infrastructure itself. The second example discusses “AI/ML for infrastructure”: the use of ML for operational intelligence (AI-Ops) to automate the predictive maintenance in a high-performance datacenter where AI-Ops takes a holistic view which includes IT telemetry and facility sensors at the edge of the datacenter. The talks concludes with a discussion on the challenges than next-generation AI accelerators are going to pose to the data piepeline that feeds them, and points to some research directions to explore.

Person 1

Paolo Faraboschi

Paolo Faraboschi is Vice President and HPE Fellow, and currently leads the “AI Lab” group in Hewlett Packard Labs, developing novel end-to-end AI technology and advanced solutions. He previously led HPE’s Exascale research, as the co-PI of the “PathForward” program in collaboration with the US DoE (2017-2019). He also researched how to build better memory-driven systems, leading The Machine hardware architecture (2014-2017). Previously (2009-2014). he worked on low-energy servers (project Moonshot), led research on scalable simulation (2004-2008), and was the architect of the Lx/ST200 family of VLIW cores for consumer SoCs (1995-2003). He is an IEEE Fellow (2014) and an active member of the computer architecture community. He holds 45 patents, over 100 publications, and a Ph.D. in EECS from University of Genoa (Italy).


Computing for Neuroscience – Challenges & Opportunities for Brain Tissue Simulations

Computational science aims to extend our ability to study models where no analytical solution exists or which are complex due to a diversity of their components and their non-linear interactions. Not only is the brain a highly complex system but also are we faced with a difficulty to measure holistically and simultaneously, our experimental account is thus incomplete. On the specific example of the Blue Brain Project, which is building biologically detailed digital reconstructions and simulations of parts of therodent brain, this talk will give an introduction to the integrated eco-system of model building, simulation and analysis workflows. It will highlight some recent modifications to these model building workflows using machine learning methods and present the quantitative understanding we have gained on the computational costs of brain tissue simulations. The talk will close with an outlook where this simulation technology has to go to enable full brain simulations and whether machine learning will be able to help with this challenge.

Person 1

Felix Schürmann

Felix Schürmann is adjunct professor at the Ecole polytechnique fédérale de Lausanne (EPFL), co-director of the Blue Brain Project and affiliated with the Brain Mind Institute. He studied physics at the University of Heidelberg, Germany, supported by the German National Academic Foundation. Later, as a Fulbright Scholar, he obtained his Master’s degree in Physics from SUNY at Buffalo, USA, on simulating quantum computers. He received his Ph.D. at the University of Heidelberg, Germany, under the supervision of the late Karlheinz Meier. For his thesis he co-designed an efficient implementation of a neural network in hardware. Since 2005 he is involved in EPFL’s Blue Brain Project, where he oversees all computer science research and engineering to enable reconstruction and simulation of brain tissue models at unprecedented scale and detail. Since he strongly believes that the futures of neuroscience and computing are entangled, he also directs his own research group to rethink today’s simulation capabilities and leverage neuroscience for future computing.


ML <-> HPC: Optimizing Optimizers for Optimization

Deep Neural Networks (DNNs) are becoming ubiquitous in modern scientific and data-intensive computing. Among their applications is automating the understanding of programs and their semantics, owing to the rapidly growing open repositories of online code. On one hand, accelerating DNN training and inference is a major challenge, and techniques range from distributed algorithms to low-level circuit design. On the other hand, DNNs that comprehend code present opportunities to aid this performance optimization process. The talk outlines approaches for deep learning parallelization and distributed computing, complemented by deep learning techniques for comprehending code in various forms. We review and model the different types of concurrency in current state-of-the-art DNNs: from the single operator to distributed training. We then review neural representations and embedding spaces of programs, both as source code and in multiple intermediate representations, followed by how they are used for performance optimization. We discuss the challenges of the intersections between Machine Learning and High-Performance Computing, current approaches taken, and their results. Based on current trends, we extrapolate potential future directions for combining ML techniques with HPC, static analysis, and compilation.

Person 1

Tal Ben-Nun

https://people.inf.ethz.ch/tbennun/


NVIDIA to accelerate the HPC-AI convergence

NVIDIA has early identified the promising HPC – AI convergence trend and has been working on enabling it. The growing adoption of NVIDIA Volta GPU by the Top 500 Supercomputers highlights the need of computing acceleration for this HPC & AI convergence. Many projects today demonstrate the benefit of AI for HPC, in terms of accuracy and time to solution, in many domains such as Computational Mechanics (Computational Fluid Mechanics, Solid Mechanics…), Earth Sciences (Climate, Weather and Ocean Modeling), Life Sciences (Genomics, Proteomics…), Computational Chemistry (Quantum Chemistry, Molecular Dynamics…), Computational Physics. NVIDIA today for instance, uses Physics Informed Neural Networks for the heat sink design in our DGX system.

Person 1

Gunter Roeth

Gunter Roeth joined NVIDIA as a Solution Architect in October 2014 having previously worked at Cray, HP, Sun Microsystems and most recently BULL. He has a Master in geophysics from the Institut de Physique du Globe (IPG) in Paris and has completed a PhD in seismology on the use of neural networks (artificial intelligence) for interpreting geophysical data.


Classical Machine Learning At Scale

Deep learning reigns supreme in fields like image classification and natural language processing. However, in industries such as finance and retail, where structured, tabular data is abundant, so-called “classical” machine learning techniques remain widely used in production. Dataset sizes in these industries may exceed many Terabytes, creating the need for data-parallel training algorithms that are capable of scaling-out across multiple servers. Additionally, the models may have a large number of hyper-parameters that require tuning, further increasing the computational load. In this talk we will review the most popular “classical” machine learning techniques today and discuss the unique challenges associated with trying to deploy them at scale on a large HPC-like cluster. Techniques covered will include: generalized linear models, random forests, gradient boosting machines as well as hyper-parameter tuning methods like random search, successive halving and Hyperband.

Person 1

Thomas Parnell

Thomas received his B.Sc. and Ph.D. degrees in mathematics from the University of Warwick. U.K., in 2006 and 2011, respectively. He joined Arithmatica, Warwick, U.K., in 2005, where he was involved in FPGA design and electronic design automation. In 2007, he co-founded Siglead Europe, a U.K.-limited subsidiary of Yokohama-based Siglead Inc., where he was involved in developing signal processing and error-correction algorithms for HDD, flash, and emerging storage technologies. In 2013, he joined IBM Research in Zürich, Switzerland, where he is actively involved in the research and development of machine learning, compression and error-correction algorithms for IBM’s storage and AI products. His research interests include signal processing, information theory, machine learning and recommender systems.