KIT high-performance computer HP XC3000 (hc3)
From 2010 until the end of January 2017, the SCC operated a parallel computer system HP XC3000, which consisted of many SMP nodes with 64-bit Xeon processors from Intel. The computer fulfilled both the tasks of a parallel high-performance computer and the tasks of a serial, throughput-oriented computer. The parallel computer could be used free of charge by all KIT employees.
Configuration of the HP XC3000 system
The HP XC3000 KIT computer contained
- 2 login nodes, each with 8 cores with a theoretical peak performance of 81.0 GFLOPS and 48 GB of main memory per node,
- 312 compute nodes, each with 8 cores with a theoretical peak performance of 81.0 GFLOPS and 24 GB of main memory per node,
- 32 computing nodes, each with 8 cores with a theoretical peak performance of 81.0 GFLOPS and 48 GB of main memory per node,
- 12 computing nodes, each with 8 cores with a theoretical peak performance of 81.0 GFLOPS and 144 GB of main memory per node
- and an InfiniBand 4X QDR interconnect with ConnectX Dual Port QDR HCAs as the connection network.
The KIT computer was a massively parallel parallel computer with a total of 366 nodes, 10 of which are service nodes. All nodes - including the service nodes - had a clock frequency of 2.53 GHz and had local memory, local disks and network adapters. A single compute node had a theoretical peak performance of 81.0 GFLOPS, resulting in a theoretical peak performance of 30.8 TFLOPS for the entire system. The main memory across all computing nodes amounted to 10.8 TB-->. All nodes were interconnected via an InfiniBand 4X QDR interconnect.
The basic operating system on each node was a Suse Linux Enterprise (SLES) 11. KITE served as the management software for the cluster; KITE is an open environment for the operation of heterogeneous computing clusters.-->
The scalable, parallel Lustre file system was connected as the global file system via a separate InfiniBand network. By using several Lustre Object Storage Target (OST) servers and Meta Data Servers (MDS), both high scalability and redundancy in the event of individual server failures were achieved. Approx. 469 TB of disk space is available in the HOME directory, which is the same as the HOME directory of the InstitusClusterII. Approx. 224 TB of disk space was available in the WORK directory. In addition, each node of the cluster was equipped with local disks for temporary data.
Detailed brief description of the nodes and the connection network:
10 Proliant DL170h service nodes with 2 eight-way (login) nodes, each with 2 quad-core Intel Xeon E5540 with a clock frequency of 2.53 GHz, 48 GB main memory and 6 146 GB local SAS disks, 2 eight-way (head) nodes, each with 2 quad-core Intel Xeon E5540 with a clock frequency of 2.53 GHz, 24 GB main memory and 6 300 GB local SAS disks,2 eight-way (NAT) nodes, each with 2 quad-core Intel Xeon E5540 with a clock frequency of 2.53 GHz, 24 GB main memory and 2 250 GB local SATA disks and 4 eight-way ("Resource Management") nodes, each with 2 quad-core Intel Xeon E5540 with a clock frequency of 2.53 GHz, 24 GB main memory and 2 250 GB local SATA disks;
312 8-way (computing) nodes, each with 2 quad-core Intel Xeon E5540 with a clock frequency of 2.53 GHz, 24 GB main memory and a 250 GB local SATA disk;
32 8-way (computing) nodes, each with 2 quad-core Intel Xeon E5540 with a clock frequency of 2.53 GHz, 48 GB main memory and a 250 GB local SATA disk;
12 8-way (compute) nodes, each with 2 quad-core Intel Xeon E5540 with a clock speed of 2.53 GHz, 144 GB main memory and 8 146 GB local SAS disks.
A single quad-core processor had 4x256 KB L2 cache and 8 MB "shared" L3 cache and offered a throughput of 5.86 GT/s via the QPI links, whereby the memory modules were directly connected at a frequency of 1333 MHz.
An InfiniBand 4X QDR switch (2x324 ports) from Voltaire with a total throughput rate of 342 x 40 Gb/s = 13.7 Tb/s served as the connection network. ConnectX IB HCAs (dual port QDR, PCIe2.0x8 with 5GT/s) were used as adapters.
Access to the HP XC3000
Only secure procedures such as secure shell (ssh) and the associated secure copy (scp) were permitted when logging in or copying data from and to the HP XC3000. The telnet and rsh mechanisms and other r commands were disabled for security reasons.