Hardware (PDF)

ExaScale Computing Study:
Technology Challenges in
Achieving Exascale Systems
Peter Kogge, Editor & Study Lead
Keren Bergman
Shekhar Borkar
Dan Campbell
William Carlson
William Dally
Monty Denneau
Paul Franzon
William Harrod
Kerry Hill
Jon Hiller
Sherman Karp
Stephen Keckler
Dean Klein
Robert Lucas
Mark Richards
Al Scarpelli
Steven Scott
Allan Snavely
Thomas Sterling
R. Stanley Williams
Katherine Yelick
September 28, 2008
This work was sponsored by DARPA IPTO in the ExaScale Computing Study with Dr. William Harrod
as Program Manager; AFRL contract number FA8650-07-C-7724. This report is published in the
interest of scientific and technical information exchange and its publication does not constitute the
Government’s approval or disapproval of its ideas or findings

NOTICE
Using Government drawings, specifications, or other data included in this document for any
purpose other than Government procurement does not in any way obligate the U.S. Government.
The fact that the Government formulated or supplied the drawings, specifications, or other data
does not license the holder or any other person or corporation; or convey any rights or permission to
manufacture, use, or sell any patented invention that may relate to them.
APPROVED FOR PUBLIC RELEASE, DISTRIBUTION UNLIMITED.

This page intentionally left blank.

DISCLAIMER

The following disclaimer was signed by all members of the Exascale Study Group (listed below):
I agree that the material in this document reflects the collective views, ideas, opinions
and findings of the study participants only, and not those of any of the universities,
corporations, or other institutions with which they are affiliated. Furthermore, the
material in this document does not reflect the official views, ideas, opinions and/or
findings of DARPA, the Department of Defense, or of the United States government.
Keren Bergman
Shekhar Borkar
Dan Campbell
William Carlson
William Dally
Monty Denneau
Paul Franzon
William Harrod
Kerry Hill
Jon Hiller
Sherman Karp
Stephen Keckler
Dean Klein
Peter Kogge
Robert Lucas
Mark Richards
Al Scarpelli
Steven Scott
Allan Snavely
Thomas Sterling
R. Stanley Williams
Katherine Yelick

i

This page intentionally left blank.

ii

FOREWORD

This document reflects the thoughts of a group of highly talented individuals from universities,
industry, and research labs on what might be the challenges in advancing computing by a thousandfold by 2015. The work was sponsored by DARPA IPTO with Dr. William Harrod as Program
Manager, under AFRL contract #FA8650-07-C-7724. The report itself was drawn from the results
of a series of meetings over the second half of 2007, and as such reflects a snapshot in time.
The goal of the study was to assay the state of the art, and not to either propose a potential
system or prepare and propose a detailed roadmap for its development. Further, the report itself
was assembled in just a few months at the beginning of 2008 from input by the participants. As
such, all inconsistencies reflect either areas where there really are significant open research questions,
or misunderstandings by the editor. There was, however, virtually complete agreement about the
key challenges that surfaced from the study, and the potential value that solving them may have
towards advancing the field of high performance computing.
I am honored to have been part of this study, and wish to thank the study members for their
passion for the subject, and for contributing far more of their precious time than they expected.

Peter M. Kogge, Editor and Study Lead
University of Notre Dame
May 1, 2008.

iii

This page intentionally left blank.

iv

Contents
1 Executive Overview
2 Defining an Exascale System
2.1 Attributes . . . . . . . . . . . . . . . . . . . . . . .
2.1.1 Functional Metrics . . . . . . . . . . . . . .
2.1.2 Physical Attributes . . . . . . . . . . . . . .
2.1.3 Balanced Designs . . . . . . . . . . . . . . .
2.1.4 Application Performance . . . . . . . . . . .
2.2 Classes of Exascale Systems . . . . . . . . . . . . .
2.2.1 Data Center System . . . . . . . . . . . . .
2.2.2 Exascale and HPC . . . . . . . . . . . . . .
2.2.3 Departmental Systems . . . . . . . . . . . .
2.2.4 Embedded Systems . . . . . . . . . . . . . .
2.2.5 Cross-class Applications . . . . . . . . . . .
2.3 Systems Classes and Matching Attributes . . . . .
2.3.1 Capacity Data Center-sized Exa Systems .
2.3.2 Capability Data Center-sized Exa Systems .
2.3.3 Departmental Peta Systems . . . . . . . . .
2.3.4 Embedded Tera Systems . . . . . . . . . . .
2.4 Prioritizing the Attributes . . . . . . . . . . . . . .

1

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

5
5
5
6
6
7
8
8
9
9
10
11
12
12
13
14
14
14

3 Background
3.1 Prehistory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Trends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Overall Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 This Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5 Target Timeframes and Tipping Points . . . . . . . . . . . . . . . . . . . . .
3.6 Companion Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.7 Prior Relevant Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.7.1 1999 PITAC Report to the President . . . . . . . . . . . . . . . . . .
3.7.2 2000 DSB Report on DoD Supercomputing Needs . . . . . . . . . .
3.7.3 2001 Survey of National Security HPC Architectural Requirements .
3.7.4 2001 DoD R&D Agenda For High Productivity Computing Systems
3.7.5 2002 HPC for the National Security Community . . . . . . . . . . .
3.7.6 2003 Jason Study on Requirements for ASCI . . . . . . . . . . . . .
3.7.7 2003 Roadmap for the Revitalization of High-End Computing . . . .
3.7.8 2004 Getting Up to Speed: The Future of Supercomputing . . . . .
v

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

17
17
18
19
19
20
20
21
21
21
21
22
22
23
23
24

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

3.7.9 2005 Revitalizing Computer Architecture Research . . . . . . . . . . . . . . . 24
3.7.10 2006 DSB Task Force on Defense Critical Technologies . . . . . . . . . . . . . 25
3.7.11 2006 The Landscape of Parallel Computing Research . . . . . . . . . . . . . . 25
4 Computing as We Know It
4.1 Today’s Architectures and Execution Models .
4.1.1 Today’s Microarchitectural Trends . . .
4.1.1.1 Conventional Microprocessors
4.1.1.2 Graphics Processors . . . . . .
4.1.1.3 Multi-core Microprocessors . .
4.1.2 Today’s Memory Systems . . . . . . . .
4.1.3 Unconventional Architectures . . . . . .
4.1.4 Data Center/Supercomputing Systems .
4.1.4.1 Data Center Architectures . .
4.1.4.2 Data Center Power . . . . . .
4.1.4.2.1 Mitigation . . . . . .
4.1.4.3 Other Data Center Challenges
4.1.5 Departmental Systems . . . . . . . . . .
4.1.6 Embedded Systems . . . . . . . . . . . .
4.1.7 Summary of the State of the Art . . . .
4.2 Today’s Operating Environments . . . . . . . .
4.2.1 Unix . . . . . . . . . . . . . . . . . . . .
4.2.2 Windows NT Kernel . . . . . . . . . . .
4.2.3 Microkernels . . . . . . . . . . . . . . .
4.2.4 Middleware . . . . . . . . . . . . . . . .
4.2.5 Summary of the State of the Art . . . .
4.3 Today’s Programming Models . . . . . . . . . .
4.3.1 Automatic Parallelization . . . . . . . .
4.3.2 Data Parallel Languages . . . . . . . . .
4.3.3 Shared Memory . . . . . . . . . . . . . .
4.3.3.1 OpenMP . . . . . . . . . . . .
4.3.3.2 Threads . . . . . . . . . . . . .
4.3.4 Message Passing . . . . . . . . . . . . .
4.3.5 PGAS Languages . . . . . . . . . . . . .
4.3.6 The HPCS Languages . . . . . . . . . .
4.4 Today’s Microprocessors . . . . . . . . . . . . .
4.4.1 Basic Technology Parameters . . . . . .
4.4.2 Overall Chip Parameters . . . . . . . .
4.4.3 Summary of the State of the Art . . . .
4.5 Today’s Top 500 Supercomputers . . . . . . . .
4.5.1 Aggregate Performance . . . . . . . . .
4.5.2 Efficiency . . . . . . . . . . . . . . . . .
4.5.3 Performance Components . . . . . . . .
4.5.3.1 Processor Parallelism . . . . .
4.5.3.2 Clock . . . . . . . . . . . . . .
4.5.3.3 Thread Level Concurrency . .
4.5.3.4 Total Concurrency . . . . . . .
4.5.4 Main Memory Capacity . . . . . . . . .
vi

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

27
27
27
28
28
28
29
30
31
31
32
33
33
34
34
35
35
36
37
37
38
38
38
40
40
41
42
43
44
45
46
47
47
49
53
53
53
54
54
55
56
56
57
59

5 Exascale Application Characteristics
5.1 Kiviat Diagrams . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Balance and the von Neumann Bottleneck . . . . . . . . . . .
5.3 A Typical Application . . . . . . . . . . . . . . . . . . . . . .
5.4 Exascale Application Characteristics . . . . . . . . . . . . . .
5.5 Memory Intensive Applications of Today . . . . . . . . . . . .
5.5.1 Latency-Sensitive Applications . . . . . . . . . . . . .
5.5.2 Locality Sensitive Applications . . . . . . . . . . . . .
5.5.3 Communication Costs - Bisection Bandwidth . . . . .
5.6 Exascale Applications Scaling . . . . . . . . . . . . . . . . . .
5.6.1 Application Categories . . . . . . . . . . . . . . . . . .
5.6.2 Memory Requirements . . . . . . . . . . . . . . . . . .
5.6.3 Increasing Non-Main Memory Storage Capacity . . . .
5.6.3.1 Scratch Storage . . . . . . . . . . . . . . . .
5.6.3.2 File Storage . . . . . . . . . . . . . . . . . .
5.6.3.3 Archival Storage . . . . . . . . . . . . . . . .
5.6.4 Increasing Memory Bandwidth . . . . . . . . . . . . .
5.6.5 Increasing Bisection Bandwidth . . . . . . . . . . . . .
5.6.6 Increasing Processor Count . . . . . . . . . . . . . . .
5.7 Application Concurrency Growth and Scalability . . . . . . .
5.7.1 Projections Based on Current Implementations . . . .
5.7.2 Projections Based on Theoretical Algorithm Analysis
5.7.3 Scaling to Departmental or Embedded Systems . . . .
5.8 Applications Assessments . . . . . . . . . . . . . . . . . . . .
5.8.1 Summary Observations . . . . . . . . . . . . . . . . .
5.8.2 Implications for Future Research . . . . . . . . . . . .
6 Technology Roadmaps
6.1 Technological Maturity . . . . . . . . . . . . . . . . . . . . .
6.2 Logic Today . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.1 ITRS Logic Projections . . . . . . . . . . . . . . . .
6.2.1.1 Power and Energy . . . . . . . . . . . . . .
6.2.1.2 Area . . . . . . . . . . . . . . . . . . . . . .
6.2.1.3 High Performance Devices . . . . . . . . .
6.2.1.4 Low Operating Voltage Devices . . . . . .
6.2.1.5 Limitations of Power Density and Its Effect
6.2.2 Silicon Logic Technology . . . . . . . . . . . . . . . .
6.2.2.1 Technology Scaling Challenges . . . . . . .
6.2.2.2 Silicon on Insulator . . . . . . . . . . . . .
6.2.2.3 Supply Voltage Scaling . . . . . . . . . . .
6.2.2.4 Interaction with Key Circuits . . . . . . . .
6.2.3 Hybrid Logic . . . . . . . . . . . . . . . . . . . . . .
6.2.4 Superconducting Logic . . . . . . . . . . . . . . . . .
6.2.4.1 Logic Power and Density Comparison . . .
6.2.4.1.1 Cooling Costs . . . . . . . . . . .
6.2.4.2 The Memory Challenge . . . . . . . . . . .
6.2.4.3 The Latency Challenge . . . . . . . . . . .
6.2.4.4 The Cross-Cryo Bandwidth Challenge . . .
vii

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
on Operating
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

61
61
62
63
65
66
66
68
69
71
71
72
73
73
73
73
74
74
74
75
75
78
80
81
81
82

85
. . . . . . 86
. . . . . . 87
. . . . . . 87
. . . . . . 88
. . . . . . 88
. . . . . . 89
. . . . . . 89
Frequency 90
. . . . . . 92
. . . . . . 92
. . . . . . 93
. . . . . . 94
. . . . . . 96
. . . . . . 97
. . . . . . 100
. . . . . . 101
. . . . . . 102
. . . . . . 102
. . . . . . 102
. . . . . . 102