Facilities, equipment, and other resources¶
Research Computing at CU Boulder consists of a small group of computational scientists, high-performance computing specialists, and system administrators with the mission to provide leadership in developing, deploying, and operating an integrated cyberinfrastructure. This cyberinfrastructure consists of high-performance computing, storage, and high speed networking that supports and encourages research, collaboration and discovery. The groups also contribute to the educational mission of the university by providing workshops and training on cyberinfrastructure related topics as well as 1:1 consulting.
RC operates the joint CSU-CU-Boulder Summit supercomputer, funded by NSF under Grant No. AC- 1532236. The system has peak performance of over 400 TFLOPS. The 380 general compute nodes have two Intel Haswell CPUs with 12 cores each, 128 GB of RAM and a local SSD. Additionally, the system has 10 GPU nodes containing two NVIDIA K80 GPUs, 5 high-memory nodes with about 1 TB of main memory and in a second deployment in December 2016 20 Xeon Phi (“Knight’s Landing”) nodes with 72 real cores supporting 288 threads for development and benchmarking. All nodes are connected through a high-performance network based on Intel Omni-Path with a bandwidth of 100 Gb/s and a latency of 0.4 microseconds. 1 PB of high-performance storage is provided using the IBM GPFS file system. This system is available to CU-Boulder researchers and collaborators, as well as 10% of cycles are provided to members of the Rocky Mountain Advanced Computing Consortium.
The RC Condo Computing service offers researchers the opportunity to purchase and own compute nodes that will be operated as part of a cluster, named “Blanca.” The aggregate cluster is made available to all condo partners while maintaining priority for the owner of each node.
The current CU Boulder network is a 40 Gbps fiber core with Cat 5 or higher wiring throughout campus. RC has created an 80 Gbps Science-DMZ to connect the Summit supercomputer to storage and to bring individual dedicated 10 Gbps circuits to various locations as needed. CU Boulder participates in I2 (the Internet 2 higher education, government, and vendor research computing consortium) and is an active member of the Front-Range gigapop and other networks. RC has started to provide campus researchers with a leading-edge network that meets their needs and facilitates collaboration, high performance data exchange, access to co-location facilities, remote mounts to storage, and real-time communications.
For moving large amounts of data Research Computing has several nodes dedicated to GridFTP file transfer. RC’s GridFTP servers support both the Globus Connect web environment and basic GridFTP via the command line.
OIT also offers a file transfer service with a web interface, which provides a good way to transfer files to collaborators. Files are uploaded to a server and a link to download the file can be emailed to an on or off-campus user.
Each researcher using the computational resources at CU Boulder has a home directory with 2GB and a project space consisting of 250 GB of storage. Additional storage is provided as part of a storage condominium at a cost of $65 per TB for single copy storage. Tape and HSM are additional storage options that are available for archive data.
The two main categories of service offered to customers of the PetaLibrary are Active storage for data that needs to be accessed frequently and Archive storage for data that is accessed infrequently. Active data is always stored on disk and is accessible to researchers on compute resources managed by RC. Archive storage consists of a two-level hierarchical storage management (HSM) solution, with disk storage for data that is more likely to be accessed and tape for data that is less likely to be accessed frequently. The cost for the research is $65/TB/year for disk and $35/TB/year for archival storage.