Testimony of Dr. Raymond L. Orbach
Director, Office of Science, U.S. Department of Energy
before the U.S. House of Representatives Committee on Science
July 16, 2003

Mr. Chairman and members of the Committee, I commend you for holding this hearing – and I appreciate the opportunity to testify on behalf of the Department of Energy’s (DOE) Office of Science–on a subject of central importance to this Nation: our need for advanced supercomputing capability. Through the efforts of DOE’s Office of Science and other federal agencies, we are working to develop the next generation of advanced scientific computational capability, a capability that supports economic competitiveness and America’s scientific enterprise.

As will become abundantly clear in my testimony, the Bush Administration has forged an integrated and unified interagency roadmap to the critical problems you have asked us to address today. No one agency can – or should – carry all the weight of ensuring that our scientists have the computational tools they need to do their job, yet duplication of effort must be avoided. The President, and John Marburger, Office of Science and Technology Policy Director, understand this. That is why all of us here are working as a team on this problem.

Mr. Chairman, for more than half a century, every President and each Congress has recognized the vital role of science in sustaining this Nation’s leadership in the world. According to some estimates, fully half of the growth in the U.S. economy in the last 50 years stems from federal funding of scientific and technological innovation. American taxpayers have received great value for their investment in the basic research sponsored by the Office of Science and other agencies in our government.

Ever since its inception as part of the Atomic Energy Commission immediately following World War II, the Office of Science has blended cutting edge-research and innovative problem solving to keep the U.S. at the forefront of scientific discovery. In fact, since the mid-1940’s, the Office of Science has supported the work of more than 40 Nobel Prize winners, testimony to the high quality and importance of the work it underwrites.

Office of Science research investments historically have yielded a wealth of dividends including: significant technological innovations; medical and health advances; new intellectual capital; enhanced economic competitiveness; and improved quality of life for the American people.

Mr. Chairman and members of this Committee, virtually all of the many discoveries, advances, and accomplishments achieved by the Office of Science in the last decade have been underpinned by advanced scientific computing and networking tools developed by the Office of Advanced Scientific Computing Research (ASCR).

The ASCR program mission is to discover, develop, and deploy the computational and networking tools that enable scientific researchers to analyze, model, simulate, and predict complex phenomena important to the Department of Energy – and to the U.S. and the world.

In fact, by fulfilling this mission over the years, the Office of Science has played a leading role in maintaining U.S. leadership in scientific computation worldwide. Consider some of the innovations and contributions made by DOE’s Office of Science:

· helped develop the Internet;
· pioneered the transition to massively parallel supercomputing in the civilian sector;
· began the computational analysis of global climate change;
· developed many of the DNA sequencing and computational technologies that have made possible the unraveling of the human genetic code; and
· opened the door for major advances in nanotechnology and protein crystallography.

Computational modeling and simulation are among the most significant developments in the practice of scientific inquiry in the latter half of the 20th Century. In the past century, scientific research has been extraordinarily successful in identifying the fundamental physical laws that govern our material world. At the same time, the advances promised by these discoveries have not been fully realized, in part because the real-world systems governed by these physical laws are extraordinarily complex. Computers help us to visualize, to test hypotheses, to guide experimental design, and most importantly to determine if there is consistency between theoretical models and experiment. Computer-based simulation provides a means for predicting the behavior of complex systems that can only be described empirically at present. Since the development of digital computers in mid-century, scientific computing has greatly advanced our understanding of the fundamental processes of nature, e.g., fluid flow and turbulence in physics, molecular structure and reactivity in chemistry, and drug-receptor interactions in biology. Computational simulation has even been used to explain, and sometimes predict, the behavior of such complex natural and engineered systems as weather patterns and aircraft performance.

Within the past two decades, scientific computing has become a contributor to essentially all scientific research programs. It is particularly important to the solution of research problems that are (i) insoluble by traditional theoretical and experimental approaches, e.g., prediction of future climates or the fate of underground contaminants; (ii) hazardous to study in the laboratory, e.g., characterization of the chemistry of radionuclides or other toxic chemicals; or (iii) time-consuming or expensive to solve by traditional means, e.g., development of new materials, determination of the structure of proteins, understanding plasma instabilities, or exploring the limitations of the “Standard Model” of particle physics. In many cases, theoretical and experimental approaches do not provide sufficient information to understand and predict the behavior of the systems being studied. Computational modeling and simulation, which allows a description of the system to be constructed from basic theoretical principles and the available experimental data, are keys to solving such problems.

Advanced scientific computing is indispensable to DOE’s missions. It is essential to simulate and predict the behavior of nuclear weapons, accelerate the development of new energy technologies, and the aid in discovery of new scientific knowledge.

As the lead government funding agency for basic research in the physical sciences, the Office of Science has a special responsibility to ensure that its research programs continue to advance the frontiers of science. All of the research programs in DOE’s Office of Science—in Basic Energy Sciences, Biological and Environmental Research, Fusion Energy Sciences, and High-Energy and Nuclear Physics—have identified major scientific questions that can only be addressed through advances in scientific computing. This will require significant enhancements to the Office of Science’s scientific computing programs. These include both more capable computing platforms and the development of the sophisticated mathematical and software tools required for large scale simulations.

Existing highly parallel computer architectures, while extremely effective for many applications, including solution of some important scientific problems, are only able to operate at 5-10% of their theoretical maximum capability on other applications. Therefore, we have initiated a Next Generation Architecture program to evaluate the effectiveness of various different computer architectures in cooperation with the National Nuclear Security Administration (NNSA) and the Defense Advanced Research Project Agency to identify those architectures which are most effective in addressing specific types of simulations.

To address the need for mathematical and software tools, and to develop highly efficient simulation codes for scientific discovery, the Office of Science launched the Scientific Discovery through Advanced Computing (SciDAC) program. We have assembled interdisciplinary teams and collaborations to develop the necessary state-of-the-art mathematical algorithms and software, supported by appropriate hardware and middleware infrastructure to use terascale computers effectively to advance fundamental scientific research essential to the DOE mission

These activities are central to the future of our mission. Advanced scientific computing will continue to be a key contributor to scientific research as we enter the twenty-first century. Major scientific challenges exist in all Office of Science research programs that can be addressed by advanced scientific supercomputing. Designing materials atom-by-atom, revealing the functions of proteins, understanding and controlling plasma turbulence, designing new particle accelerators, and modeling global climate change, are just a few examples.

Today, high-end scientific computation has reached a threshold which we were all made keenly aware of when the Japanese Earth Simulator was turned on. The Earth Simulator worked remarkably well on real physical problems at sustained speeds that have never been achieved before. The ability to get over 25 teraFLOPS in geophysical science problems was not only an achievement, but it truly opened a new world.
So the question before us at today’s hearing – “Supercomputing: Is the U.S. on the Right Path” – is very timely. There is general recognition of the opportunities that high-end computation provides, and this Administration has a path forward to meet this challenge.

The tools for scientific discovery have changed. Previously, science had been limited to experiment and theory as the two pillars for investigation of the laws of nature. With the advent of what many refer to as “Ultra-Scale” computation,” a third pillar—simulation—has been added to the foundation of scientific discovery. Modern computational methods are developing at such a rapid rate that computational simulation is possible on a scale that is comparable in importance with experiment and theory. The remarkable power of these facilities is opening new vistas for science and technology.

Tradition has it that scientific discovery is based on experiment, buttressed by theory. Sometimes the order is reversed, theory leads to concepts that are tested and sometimes confirmed by experiment. But more often, experiment provides evidence that drives theoretical reasoning. Thus, Dr. Samuel Johnson, in his Preface to Shakespeare, writes: “Every cold empirick, when his heart is expanded by a successful experiment,
swells into a theorist.”

Many times, scientific discovery is counter-intuitive, running against conventional wisdom. Probably the most vivid current example is the experiment that demonstrated that the expansion of our Universe is accelerating, rather than in steady state or contracting. We have yet to understand the theoretical origins for this surprise.
During my scientific career, computers have developed from the now “creaky” IBM 701, upon which I did my thesis research, to the so-called massively parallel processors or MPP machines, that fill rooms the size of football fields, and use as much power as a small city.

The astonishing speeds of these machines, especially the Earth Simulator, allow Ultra-Scale computation to inform our approach to science, and I believe social sciences and the humanities. We are now able to contemplate exploration of worlds never before accessible to mankind. Previously, we used computers to solve sets of equations representing physical laws too complicated to solve analytically. Now we can simulate systems to discover physical laws for which there are no known predictive equations. We can model physical or social structures with hundreds of thousands, or maybe even millions, of “actors,” interacting with one another in a complex fashion. The speed of our new computational environment allows us to test different inter-actor (or inter-personal) relations to see what macroscopic behaviors can ensue. Simulations can determine the nature of the fundamental “forces” or interactions between “actors.”

Computer simulation is now a major force for discovery in its own right.
We have moved beyond using computers to solve very complicated sets of equations to a new regime in which scientific simulation enables us to obtain scientific results and to perform discovery in the same way that experiment and theory have traditionally been used to accomplish those ends. We must think of high-end computation as the third of the three pillars that support scientific discovery, and indeed there are areas where the only approach to a solution is through high-end computation – and that has consequences.

American industry certainly is fully conversant with the past, present and prospective benefits of high-end computation. The Office of Science has received accolades for our research accomplishments from corporations such as General Electric and General Motors. We have met with the vice-presidents for research of these and other member companies of the Industrial Research Institute. We learned, for example, that GE is using simulation very effectively to detect flaws in jet engines. What’s more, we were told that, if the engine flaws identified by simulation were to go undetected, the life cycle of those GE machines would be reduced by a factor of two – and that would cause GE a loss of over $100,000,000.00.

The market for high-end computation extends beyond science, into applications, creating a commercial market for ultra-scale computers. The science and technology important to industry can generate opportunities measured in hundreds of million, and perhaps billions of dollars.

Here are just a few examples:

From General Motors:

“General Motors currently saves hundreds of millions of dollars by using its in-house high performance computing capability of more than 3.5 teraFLOPS in several areas of its new vehicle design and development processes. These include vehicle crash simulation, safety models, vehicle aerodynamics, thermal and combustion analyses, and new materials research. The savings are realized through reductions in the costs of prototyping and materials used.

However, the growing need to meet higher safety standards, greater fuel efficiency, and lighter but stronger materials, demands a steady yearly growth rate of 30 to 50% in computational capabilities but will not be met by existing architectures and technologies…A computing architecture and capability on the order of 100 teraFLOPS for example would have quite an economic impact, on the order of billions of dollars, in the commercial sector in its product design, development, and marketing.”

And from General Electric:

“Our ability to model, analyze and validate complex systems is a critical part of the creation of many of our products and design. Today we make extensive use of high-performance computing based technologies to design and develop products ranging from power systems and aircraft engines to medical imaging equipment. Much of what we would like to achieve with these predictive models is out of reach due to limitations in current generation computing capabilities. Increasing the fidelity of these models demands substantial increases in high-performance computing system performance. We have a vital interest in seeing such improvements in the enabling high-performance computing technologies…In order to stay competitive in the global marketplace, it is of vital importance that GE can leverage advances in high-performance computing capability in the design of its product lines. Leadership in high-performance computing technologies and enabling infrastructure is vital to GE if we wish to maintain our technology leadership.”

Consider the comparison between simulations and prototyping for GE jet engines.

For evaluation of a design alternative for the purpose of optimization of a compressor for a jet engine design, GE would require 3.1 x 1018 floating point operations, or over a month at a sustained speed of one teraFLOP, which is today’s state-of-the-art. To do this for the entire engine would require sustained computing power of 50 teraFLOPS for the same period. This is to be compared with millions of dollars, several years, and designs and re-designs for physical prototyping.

Opportunities abound in other fields such as pharmaceuticals, oil and gas exploration, and aircraft design.

The power of advance scientific computation is just beginning to be realized. One reason that I have emphasized this so much is that some seem to think that advanced scientific computation is the province of the Office of Science and other federal science agencies and therefore is not attractive to the vendors in this field. I believe that’s incorrect. I believe instead that our leading researchers are laying out a direction and an understanding of available opportunities. These opportunities spur markets for high-end computation quite comparable to the commercial market which we have seen in the past but requiring the efficiencies and the speeds which high-end computation can provide.

Discovery through simulation requires sustained speeds starting at 50 to 100 teraFLOPS to examine problems in accelerator science and technology, astrophysics, biology, chemistry and catalysis, climate prediction, combustion, computational fluid dynamics, computational structural and systems biology, environmental molecular science, fusion energy science, geosciences, groundwater protection, high energy physics, materials science and nanoscience, nuclear physics, soot formation and growth, and more (see http://www.ultrasim.info/doe_docs/).

Physicists in Berkeley, California, trying to determine whether our universe will continue to expand or eventually collapse, gather data from dozens of distant supernovae. By analyzing the data and simulating another 10,000 supernovae on supercomputers (at the National Energy Research Scientific Computing Center or NERSC) the scientists conclude that the universe is expanding — and at an accelerating rate.

I just returned from Vienna, where I was privileged to lead the U.S. delegation in negotiations on the future direction for ITER, an international collaboration that hopes to build a burning plasma fusion reactor, which holds out promise for the realization of fusion power. The United States pulled out of ITER in 1998. We’re back in it this year. What changed were simulations that showed that the new ITER design will in fact be capable of achieving and sustaining burning plasma. We haven’t created a stable burning plasma yet, but the simulations give us confidence that the experiments which we performed at laboratory scales could be realized in larger machines at higher temperatures and densities.

Looking to the future, we are beginning a Fusion Simulation Project to build a computer model that will fully simulate a burning plasma to both predict and interpret ITER performance and, eventually, assist in the design of a commercially feasible fusion power reactor. Our best estimate, however is that success in this effort will require at least 50 teraFLOPS of sustained computing power.

Advances in scientific computation are also vital to the success of the Office of Science’s Genomes to Life program.

The Genomes to Life program will develop new knowledge about how micro-organisms grow and function and will marry this to a national infrastructure in computational biology to build a fundamental understanding of living systems. Ultimately this approach will offer scientists insights into how to use or replicate microbiological processes to benefit the Nation.

In particular, the thrust of the Genomes to Life program is aimed directly at Department of Energy concerns: developing new sources of energy; mitigating the long-term effects of climate change through carbon sequestration; cleaning up the environment; and protecting people from adverse effects of exposure to environmental toxins and radiation.

All these benefits – and more – will be possible as long as the Genomes to Life program achieves a basic understanding of thousands of microbes and microbial systems in their native environments over the next 10 to 20 years. To meet this challenge, however, we must address huge gaps not only in knowledge but also in technology, computing, data storage and manipulation, and systems-level integration.

The Office of Science also is a leader in research efforts to capitalize on the promise of nanoscale science.

In an address to the American Association for the Advancement of Science in February 2002, Dr. John Marburger, Director of the Office of Science and Technology Policy, noted, “…[W]e are in the early stages of a revolution in science nearly as profound as the one that occurred early in the last century with the birth of quantum mechanics,” a revolution spurred in part by “the availability of powerful computing and information technology.”

“The atom-by-atom understanding of functional matter,” Dr. Marburger continued, “requires not only exquisite instrumentation, but also the capacity to capture, store and manipulate vast amounts of data. The result is an unprecedented ability to design and construct new materials with properties that are not found in nature…. [W]e are now beginning to unravel the structures of life, atom-by-atom using sensitive machinery under the capacious purview of powerful computing.”

In both nanotechnology and biotechnology, this revolution in science promises a revolution in industry. In order to exploit that promise, however, we will need both new instruments and more powerful computers, and the Office of Science has instituted initiatives to develop both.

We have begun construction at Oak Ridge National Laboratory on the first of five Nanoscale Science Research Centers located to take advantage of the complementary capabilities of other large scientific facilities, such as the Spallation Neutron Source at Oak Ridge, our synchrotron light sources at Argonne, Brookhaven and Lawrence Berkeley, and semiconductor, microelectronics and combustion research facilities at Sandia and Los Alamos. When complete, these five Office of Science nanocenters will provide the Nation with resources unmatched anywhere else in the world.

To determine the level of computing resources that will be required, the Office of Science sponsored a scientific workshop on Theory and Modeling in Nanoscience, which found that simulation will be critical to progress, and that new computer resources are required. As a first step to meeting that need, our Next Generation Architecture initiative is evaluating different computer architectures to determine which are most effective for specific scientific applications, including nanoscience simulations.

There are many other examples where high-end computation has changed and will change the nature of the field. My own field is complex systems. I work in a somewhat arcane area called spin glasses, where we can examine the dynamic properties of these very complex systems, which in fact are related to a number of very practical applications. Through scientific simulation, a correlation length was predicted for a completely random material, a concept unknown before. Simulation led to the discovery that there was a definable correlation length in this completely random system. Our experiments confirmed this hypothesis. Again, insights were created that simply were not possible from a set of physical equations that needed solutions, with observable consequences. There are countless opportunities and examples where similar advances could be made.

As the Chairman and members of this Committee know, the Bush Administration shares Congress’ keen interest in high-end computation for both scientific discovery and economic development. A senior official of the Office of Science is co-chairing the interagency High-End Computing Revitalization Task Force, which includes representatives from across the government and the private sector. We are working very closely with the NNSA, the Department of Defense, the National Aeronautics and Space Administration, the National Science Foundation, and the National Institutes of Health to assess how best to coordinate and leverage our agencies’ high-end computation investments now and in the future.

DOE is playing a major role in the task force through the Office of Science and the NNSA, and many of the task force’s findings and plans are based on Office of Science practices in advanced computing and simulation.

One of the major challenges in this area is one of metrics. How do we know what we are trying to accomplish, and how can we measure how we’re getting there? What are the opportunities? What are the barriers? What should we be addressing as we begin to explore this new world?

Our problem in the past has been that, where we have large computational facilities, we have cut them up in little pieces and the large-scale scientific programs that some researchers are interested in have never really had a chance to develop. There’s nothing wrong with our process; it is driven by a peer review system. But for some promising research efforts, there simply have not been enough cycles or there wasn’t an infrastructure which would allow large-scale simulations to truly develop and produce the kind of discoveries we hope to achieve.

Recognizing this, the Office of Science has announced that ten percent of our National Energy Research Scientific Computing Center at Lawrence Berkeley National Laboratory – now at ten teraFLOP peak speed – is going to be made available for grand challenge calculations. We are literally going to carve out 4.5 million processor hours and 100 terabytes of disk space for perhaps four or five scientific problems of major importance. We are calling this initiative INCITE -- the Innovative and Novel Computational Impact on Theory and Experiment – and we expect to be ready to proceed with it around August 1, 2003. At that time, we will open the competition to all, whether or not they are affiliated with or funded by DOE.

We are launching the INCITE initiative for two reasons. For one, it’s the right thing to do: there are opportunities for major accomplishments in this field of science. In addition, there is also a “sociology” that we need to develop.

Given the size and complexity of the machines required for sustained speeds in the 50 to 100 teraFLOPS regime, the sociology of high-end computation will probably have to change. One can think of the usage of ultra-scale computers as akin to that of our current light sources: large machines used by groups of users on a shared basis. Following the leadership of our SciDAC program, interdisciplinary teams and collaborators will develop the necessary state-of-the-art mathematical algorithms and software, supported by appropriate hardware and middleware infrastructure, to use terascale computers effectively to advance fundamental research in science. These teams will associate on the basis of the mathematical infrastructure of problems of mutual interest, working with efficient, balanced computational architectures.

The large amount of data, the high sustained speeds, and the cost will probably lead to concentration of computing power in only a few sites, with networking useful for communication and data processing, but not for core computation at terascale speeds. Peer review of proposals will be used to allocate machine time. Industry will be welcome to participate, as has happened in our light sources. Teams will make use of the facilities as user groups, using significant portions (or all) of the machine, depending on the nature of their computational requirements. Large blocks of time will enable scientific discovery of major magnitude, justifying the large investment ultra-scale computation will require.

We will open our computational facilities to everyone. Ten percent of NERSC’s capability will be available to the entire world. Prospective users will not have to have a DOE contract, or grant, or connection. The applications will be peer reviewed, and will be judged solely on their scientific merit. We need to learn how to develop the sociology that can encourage and then support computation of this magnitude; this is a lot of computer time. It may be the case that teams rather than individuals will be involved. It even is possible that one research proposal will be so compelling that the entire ten percent of NERSC will be allocated to that one research question.

The network that may be required to handle that amount of data has to be developed. There is an ES network which we are involved in, and we are studying whether or not it will be able to handle the massive amounts of data that could be produced under this program.

We need to get scientific teams – the people who are involved in algorithms, the computer scientists, and the mathematicians – together to make the most efficient use of these facilities. That’s what this opening up at NERSC is meant to do. We want to develop the community of researchers within the United States – and frankly around the world – that can take advantage of these machines and produce the results that will invigorate and revolutionize their fields of study.

But this is just the beginning.

As we develop the future high-end computational facilities for this nation and world, it is clearly our charge and our responsibility to develop scientific opportunities for everyone. This has been the U.S tradition. It has certainly been an Office of Science tradition, and we intend to see that this tradition continues, and not just in the physical sciences.

We are now seeing other fields recognizing that opportunities are available to them. In biology, we are aware that protein folding is a very difficult but crucial issue for cellular function. The timescales that biologists work with can scale from a femto-second to seconds–a huge span of time which our current simulation capabilities are unable to accommodate.

High-performance computing provides a new window for researchers to observe the natural world with a fidelity that could only be imagined a few years ago. Research investments in advanced scientific computing will equip researchers with premier computational tools to advance knowledge and to solve the most challenging scientific problems facing the Nation.

With vital support from this Committee, the Congress and the Administration, we in the Office of Science will help lead the U.S. further into the new world of supercomputing.

We are truly talking about scientific discovery. We are talking about a third pillar of support. We are talking about opportunities to understand properties of nature that have never before been explored. That’s the concept, and it explains the enormous excitement that we feel about this most promising field.

We are very gratified that this Committee is so intent on enabling us to pursue so many important opportunities, for the sake of scientific discovery, technological innovation, and economic competitiveness.

Thank you very much.