Organization:National Computational Infrastructure

Short description: HPC facility in Canberra, Australia

The National Computational Infrastructure building at the Australian National University in 2013

The National Computational Infrastructure (also known as NCI or NCI Australia) is a high-performance computing and data services facility, located at the Australian National University (ANU) in Canberra, Australian Capital Territory. The NCI is supported by the Australian Government's National Collaborative Research Infrastructure Strategy (NCRIS), with operational funding provided through a formal collaboration incorporating CSIRO, the Bureau of Meteorology, the Australian National University, Geoscience Australia, the Australian Research Council, and a number of research intensive universities and medical research institutes.

Access to computational resources is provided to funding partners as well as researchers awarded grants under the National Computing Merit Allocation Scheme (NCMAS).

The current director is Sean Smith.

Notable staff

• Lindsay Botten – former director
• Chris Pigram – former CEO of Geoscience Australia and acting director after the retirement of Lindsay Botten.
• Sean Smith – current director

Facility

The NCI building is located on the ANU campus in Canberra and uses hot aisle containment and free cooling to cool their computers.

Computer systems

As of June 2020, NCI operates two main high-performance computing installations, including:

• Gadi, meaning 'to search for' in the local Ngunnawal language. a 9.26 PetaFLOP high-performance distributed memory cluster consisting of:
  • 145,152 cores (Intel Xeon Scalable 'Cascade Lake' processors) across 3024 nodes
  • 160 nodes containing four Nvidia V100 GPUs
  • 567 Terabytes of main memory
  • 20 Petabytes of fast storage
  • 47 Petabytes of storage for large data files
  • 50 Petabytes of tape storage for archival
  • HDR Mellanox Infiniband in Dragonfly+ topology (up to 200Gbit/s transfer)
• Tenjin, a 67 TeraFLOP bespoke high-performance partner cloud, consisting of:
  • 1600 Intel Xeon Sandy Bridge cores
  • 25 Terabytes main memory
  • 160 Terabytes State Disk

Data services and storage

NCI operates the fastest filesystems in the Southern Hemisphere. 20 Petabytes of storage is available for fast I/O, 47 Petabytes is available for large data and research files, and 50 Petabytes is available on tape for archival.

Datasets

NCI hosts multiple data sets that can be used on their computation systems including:

• Aboriginal and Torres Strait Islander Data Archive (ATSIDA) which provides Australian Indigenous research data
• Australian Astronomy Optical Data Repository (ODR) including:
  • Anglo-Australian Telescope (AAT) current and selected historical datasets
  • Southern Sky Survey, using the ANU's robotic SkyMapper telescope at Mount Stromlo Observatory
• Australian National Geophysical Collection (300TB in 2015) including:
  • Airborne geophysics data
  • Gravity data set
  • Seismic survey
• High resolution 'raw' Indian Ocean sea floor data generated as part of the search for Malaysia Airlines Flight 370.

Research

Research conducted or under way includes:

• Southern Sky Survey, using the ANU's robotic SkyMapper telescope at Mount Stromlo Observatory
• The Australian Community Climate and Earth System Simulator (ACCESS)
• COVID-19 mitigation research
• Medical and materials research

History

NCI Australia is a direct descendant of the ANU Supercomputing Facility ANUSF, which existed from 1987 through to 1999. At the turn of the new millennium, the Australian Government pushed ahead with a process to form the Australian Partnership for Advanced Computing (APAC), the foundation of which would be built around a new national computational infrastructure. With its heritage in supercomputing, it was decided that the APAC National Facility would be located at The Australian National University, with the facility ultimately commissioned in 2001.

In 2007, APAC began its evolution into the present NCI collaboration.

The below table is a comprehensive history of supercomputer specifications present at the NCI and its antecedents.

System (name)	Processor	Memory	Storage	Peak Perf.	Sustained Perf. (SPEC)	Initial Top500 Rank
1987–92 Fujitsu VP100	Vector	64 MByte		0.15 GFlops
1992–96 Fujitsu VP2200	Vector	512 MByte	27 GByte	1.25 GFlops
1996–2001 Fujitsu VPP	Vector/ Scalar	14 GByte		28 GFlops		59
SGI Power Challenge XL	20 MIPS R10000	2 GByte	77 GByte	6.4 GFlops
2001–05 Compaq/HP Alphaserver (sc)	512 DEC Alpha	0.5 TByte	12TByte	1 TFlop	2,000	31
2005–09 SGI Altix 3700 (ac)	1920 Intel Itanium	5.5 TByte	100 TByte	14 TFlop	21,000	26
2009–13 SGI Altix XE (xe)	1248 Intel Xeon (Nehalem)	2.5 TByte	90 TByte	14 TFlop	12,000	—
2009–13 Sun/Oracle Constellation (Vayu)	11,936 Intel Xeon (Nehalem)	37 TByte	800 TByte	140 TFlop	240,000	35
2013–2019 Fujitsu Primergy (Raijin)	57,472 Intel Xeon (Sandy Bridge)	160 TByte	12.5 PByte	1195 TFlop	1,600,000	24
2020–Present Fujitsu Primergy CX2570 (Gadi)	145152 Intel Xeon (Cascade Lake)	576 TByte	20 PByte	9260 TFlop		24

Vayu

The Vayu computer cluster, the predecessor of Raijin, was based on a Sun Microsystems Sun Constellation System. The Vayu system was taken from Sun's code name for the compute blade within the system. Vayu is a Hindu god, the name meaning "wind". The cluster was officially launched on 2009-11-16 by the Government of Australia's Minister for Innovation, Industry, Science and Research, Senator Kim Carr, after provisional acceptance on 2009-09-18.

Vayu was first operated in September 2009 with one eighth of the final computing power, with the full system commissioned in March 2010. Vayu had the following performance characteristics:

• Peak performance: 140 TFlops
• Sustained: 250K SPECfp rate
• Resources: 110M hrs p.a.

The system comprised:

• 11936 CPUs in 1492 nodes in Sun X6275 blades, each containing
  • two quad-core 2.93 GHz Intel Nehalem CPUs
  • 24Gbyte DDR3-1333 memory
  • 24GB Flash DIMM for swap and job scratch
• total: 36.9TB of RAM on compute nodes
• Dual socket, quad-core Sun X4170, X4270, X4275 servers for Lustre fileserving
• approx 835TBytes of global user storage

Power consumption of the full 11936 CPU system was approx 605 kW, but all the power was intended to be from green energy sources.

System software for the Vayu cluster includes:

• CentOS 5.4 Linux distribution (based on Red Hat Enterprise Linux 5.4)
• the oneSIS cluster software management system
• the Lustre cluster file system
• the National Facility's variant of the OpenPBS batch queuing system

The national government has provided around A$26m to enable the building of the centre and installation of Vayu. Other participating organisations included the Australian Bureau of Meteorology, Australian National University, and the Commonwealth Scientific and Industrial Research Organisation, cooperating using an integrated computational environment for the earth systems sciences, including investigating aspects of operational weather forecasting through to climate modelling and prediction. The ANU and CSIRO each subscribed about A$3m, thereby getting about a quarter of the machine. The ANU and CSIRO, with the support of the Australian Government made plans for funding Vayu's replacement, in about 2011-2012, with a machine about 12 times more powerful.

See also

• Pawsey Supercomputing Centre
• CSIRO
• Geoscience Australia
• TOP500

External links

• NCI Australia website
  • NCI facilities