Researchers Wiring Together PCs for Supercomputer-Like Performance

Besides saving money, users of these collectives of high end but off-the-shelf PCs -- often called "Beowulf clusters - can avoid the negative side of relying on supercomputing centers from Research Triangle Park to San Diego.

"In the past we used a supercomputer," said Roni Avissar, the chairman of the Pratt School of Engineering's civil and environmental engineering department. "The problem is you had to share the supercomputer with a lot of other people."

He now uses two Beowulf clusters, one with the equivalent of 12 and the other with 24 individual desktops, to run a new complex global climate model his research group is creating at Duke.

"Now I get basically much more than I was able to get from a supercomputer when you factor in the waiting time," said Avissar, his department's W.H. Gardner Jr. Professor. Moreover, the 24 unit cluster he just bought at Duke cost Avissar only $43,000, complete with two high speed processors per computer.

"The beauty is cluster computing is you can do it on a shoestring," added Robert Brown, a longtime visiting physics professor who built an early Beowulf cluster at Duke in 1995. Brown estimates the cheapest clusters can cost as little as $700 per workstation, often called a "node," with the "head node" costing a bit more.

"It’s the kind of thing an individual investigator can own and control 100 percent of the time. That’s a lot of computing power," said John Board, Pratt's Anne T. and Robert M. Bass Associate Professor of electrical engineering and computer science, another early Beowulf cluster user who now has access to four of them.

However, John Harer, Duke's Vice Provost for Academic Affairs, sounded a note of caution for would-be cluster aspirants who might not be as experienced as researchers like Avissar, Brown and Board.

"People might think they can just go buy 12 PCs and wire them together, but it doesn't work that way," said Harer, a mathematician. "What happens is people who don't know what they're doing buy machines that end up sitting in the closet. They're forgetting there's a big startup cost and software cost and annual maintenance cost, all of which come free at a supercomputing center."

That's why Harer is spearheading plans to build a more-centralized "cluster farm" to accommodate researchers who lack others' expertise or funding base, or who may only need occasional access to supercomputer-like power. "Our goal is to provide as much sharing of expertise and support as we can," he added.

Beowulf clusters got their name from a NASA-Goddard project that required heavy duty computational power, said Brown., who has written what he thinks is the only online book on cluster engineering.

NASA investigators "decided to do some research on how cheaply they could put together a supercomputer using common off-the-shelf components," added William Rankin, who did some early work on them while getting his Ph.D. at Duke under Board. "Since then the name has come to represent in general the kind of cluster computing research that's done today."

When the university hired Rankin last December as a research scientist to help install and maintain the growing numbers of computer clusters, he first tried to find out exactly how many clusters there are at Duke.

"When I got close to two dozen, planned and current, I realized how big this job was going to be," he said. "They're now in use in almost every department across campus."

Rankin is still not sure of the total number. But, according to the minutes of an October, 2002 meeting of Duke's Information Technology Advisory Council, there are currently 32 clusters at Duke in a variety of locations.

It may be more realistic to count the total number of nodes devoted to cluster computing than to count the number of clusters, said Harer. "We have over 1,000 nodes now all around campus if you add them up."

Brown thinks Duke is "very well known, and well recognized nationally and internationally" in its use of cluster computing. "My sense is that we are not leaders in this, but fairly aggressive early adopters," added Board. Tracy Futhey, Duke's Vice President for Information Technology, agreed. "We're very respectably ahead of the average, though not bleeding edge leaders," she estimated.

Brown said he assembled the first cluster of PCs at Duke dedicated to supercomputer scale computational tasks after previously running his theoretical physics calculations behind the scenes on the same ACPUB public cluster that students used to send e-mail and write papers.

Board, who brought another early Beowulf cluster to Duke via an Intel Corp equipment donation, seconded Harer's admonitions that, while such clusters are both cheap and fast, they can be challenging to use.

The processing units themselves are not the problem, Board said. Today's commercially available processors are fast enough for scientific usage. In fact, a recent Oak Ridge National Laboratory study rates Intel's Pentium IV processor as faster than individual processors on stand-alone supercomputers made by Cray, Fujitsu and NEC, Board said.

The challenge is "to coax really high performance out of this collection of hardware, mainly because networking is the weak link in the chain," he added.

Networking is a problem because the same commercial Ethernet connections typically used in Beowulf-style clusters also link PCs in a usual office array. And Ethernet links simply do not transmit information as fast as the computers to which they are attached, in effect lowering the whole operation's speed limit.

Some Duke cluster users have responded by improving their Ethernet networks, said Board. "You can buy better networks than stock Ethernet. And there are higher-speed, priority networks that dramatically improve the performances of these clusters. But they may double the cost as well."

Another challenge with Beowulf clusters is programming the computers to communicate with one another to share a problem. Supercomputers are built around the idea that their many processors will share information. But individual desktops "have no fundamental notion of working with other computers," Board continued.

So experts like Board and their graduate students must write extensions to programming languages that add the capability of sending and receiving data, he explained. In essence, Processor 5 must be told it is different from processor 7. Processor 5 must also "inform" processor 7 that it has some information to send it.

Imbedded in that information is both a task arriving from 5, and a request that 7 perform it. Only in this way can 5 and 7 participate in "parallel processing" of parts of the same task, a hallmark of supercomputing.

One innovation that has helped is a programming standard called Message Passing Interface, or MPI, noted John Pormann, who also got his Ph.D. under Board and now works with several research teams on cluster computing. With MPI, "the exact same code will compile and run on IBMs Crays and SGI supercomputers and on Beowulf clusters," Pormann said.

Even with MPI, Rankin cautions that this kind of programming "is still complex enough that's it's not 'cookbook' by any stretch of the imagination. It still falls under the realm of research."

Desktop units used in PC-like clusters differ from a typical office models in several ways. For example, many have two processors instead of the normal one. And most lack the usual keyboard, mouse and monitor.

In some cases, such stripped-down collections of towers and interconnecting wires, still retaining a desktop-like look, sit in small rooms amid enough air conditioning to keep their circuitry from overheating. But increasingly their components are being installed in neat and compact racks of electronics that look more high tech and occupy far less space.

Board uses his departmental clusters in a joint project with biophysicists to simulate the dynamics of biological molecules such as proteins. He is also collaborating with Larry Carin, another professor of electrical and computer engineering, on radar techniques to detect hidden targets such as tanks.

Meanwhile Craig Henriquez, W.H. Gardner Jr. Associate Professor of biomedical engineering and computer science, is using a 32-PC cluster together with Miguel Nicolelis, a Duke neurobiology professor, and other researchers to analyze the way the brain processes sensory signals and generates motor output.

In Avissar’s department an additional 32-machine Beowulf cluster is being used to simulate computation-intensive problems ranging from the actions of water to structural failure. In the Department of Mathematics, Professors William Allard and John Trangenstein are using a cluster to simulate complexities like the flow of pollutants and electrical activity in the heart.

There are now more than 200 processors operating in the basement of the Physics Building, said Brown, who has helped various researchers assemble clusters there. He uses more than 50 processors himself on problems in condensed matter physics. Another 64 processors help Duke theoreticians simulate quark-gluon plasmas that may have existed at the dawn of creation. A third cluster in the Physics Building basement is used by Duke's Institute for Statistics and Decision Sciences, and a fourth by physicists and chemists involved in basic nanoscience research.

The term Beowulf cluster has now become "a generic terms for any sort of high-performance computing engine that’s built out of off-the-shelf PC-grade components," Board noted. But "the original Beowulf was built to replace supercomputers," emphasized Brown , who acknowledged that some researchers use PC clusters in other ways. "There are purists that care," Brown said.

"I would say that what we do meets the Beowulf definition but is not constrained by it," said Jeffrey Chase, an associate professor at Duke's computer science department with a giant collection at his disposal for experimental research.

"Beowolf makes a group of computers look the same, like a single computer, with a homogenous software environment," Chase added. "Our model allows different computers to play different roles and serve different purposes within the cluster."

Computer Science's collaborative computer cluster first began building in 1995 with a grant from the National Science Foundation. The cluster has since expanded to 220 interconnected computers with another 100 on the way courtesy of a National Science Foundation grant and gifts from high-tech firms like IBM.

So big it is actually located in two different buildings, this megacluster is mainly being used as a laboratory test bed for large scale distributed services such as those provided by the Internet.

Functioning more like an experimental collection of centralized computers called servers than a supercomputer equivalent, it still has enough computing power to "place it among the top 200 clusters world-wide," Chase said.

"We use our cluster to experiment with future Web services by emulating the Internet itself, like a ship in a bottle" he added. "We are also researching how to share very large clusters for many diverse purposes at once."

Harer said he is using Chase's array as a model for the centralized cluster farm he envisions to help more researchers do high end computing.

The North Carolina Biogrid Project further advances the concept of collective computing. "Basically, it's a 'grid,' which is a really big, misused buzz word in cluster computing these days," said Rankin.

The idea is to set up enough shared clusters and supporting software at area universities, with additional links to the N.C. Supercomputing Center in Research Triangle Park, so that researchers who need it "can log in on their own desktops and have a very large virtual computer at their fingertips," Rankin added.

This particular grid is being set up to help investigators deal with the vast array of biological information that is now available, a field known as "bioinformatics."

Rankin, who works for the Provost's office, said the grid concept actually represents another retreat from the idea of more autonomous PC cluster computing as it has proliferated within the last half decade.

"People have realized that maintaining that sort of horsepower - the power, the cooling, the administration, the support and everything - really does, in some cases, require centralized facilities," he said.

Efforts like the Biogrid "are attempts to find a happy medium," he added. "The computational biologist sits there on his desktop and logs in. His information goes out to some distributed, but more centralized, facility to be run. Then it comes back to him. It makes sense that we can offer the researchers more computational bang for their bucks that way."

Rankin acknowledged that "it's going to require a lot of people to buy into this idea of sharing resources, but I think it's going to happen."