Alpine Hardware

Alpine Hardware#

Hardware Summary#

Important

All Alpine nodes are available to all users. For full details about node access, please read the Alpine node access and FairShare policy.

University of Colorado Boulder contribution#

Count & Type

Partition

Processor

Sockets

Cores (total)

Threads per Core

RAM per Core (GB)

GPU type

GPU count

Local Disk Capacity & Type

Fabric

284 Milan General CPU

amilan

x86_64 AMD Milan

1 or 2

64

1

3.8

N/A

0

416G SSD

HDR-100 InfiniBand (200Gb inter-node fabric)

16 Milan CPU

amilan

x86_64 AMD Milan

2

128

1

3.8

N/A

0

416G SSD

HDR-100 InfiniBand (200Gb inter-node fabric)

12 Milan High-Memory

amem

x86_64 AMD Milan

2

48

1

21.5

N/A

0

416G SSD

2x25 Gb Ethernet +RoCE

8 Milan High-Memory

amem

x86_64 AMD Milan

1

64

1

16

N/A

0

416G SSD

2x25 Gb Ethernet +RoCE

7 Milan AMD GPU

ami100

x86_64 AMD Milan

2

64

1

3.8

AMD MI100

3

416G SSD

2x25 Gb Ethernet +RoCE

7 Milan NVIDIA GPU

aa100

x86_64 AMD Milan

2

64

1

3.8

NVIDIA A100

3

416G SSD

2x25 Gb Ethernet +RoCE

2 Grace CPU NVIDIA Hopper GPU

gh200

Note: these nodes are only available upon request, please submit a support request form.

ARM Neoverse V2

1

72

1

6.6

NVIDIA Hopper GPU

1

1.8 T SSD

2x25 Gb Ethernet +RoCE

2 Milan CPU compile nodes

acompile

x86_64 AMD Milan

1 or 2

64

1

3.8

N/A

0

416G SSD

HDR-100 InfiniBand (200Gb inter-node fabric)

2 Milan CPU test nodes; pulls from CU amilan pool

atesting

x86_64 AMD Milan

1 or 2

64

1

3.8

N/A

0

416G SSD

HDR-100 InfiniBand (200Gb inter-node fabric)

1 Milan NVIDIA GPU testing node

aa100 (requested using the gpu-testing QoS)

x86_64 AMD Milan

2

64

1

3.8

NVIDIA A100

3 (each split by MIG)

416G SSD

2x25 Gb Ethernet +RoCE

1 Milan AMD GPU testing nodes; pulls from ami100 pool

ami100 (requested using the gpu-testing QoS)

x86_64 AMD Milan

2

64

1

3.8

AMD MI100

3

416G SSD

2x25 Gb Ethernet +RoCE

CU Anschutz Medical Campus contribution#

Count & Type

Partition

Processor

Sockets

Cores (total)

Threads per Core

RAM per Core (GB)

GPU type

GPU count

Local Disk Capacity & Type

Fabric

26 Milan General CPU

amilan

x86_64 AMD Milan

1

64

1

3.8

N/A

0

416G SSD

2x25 Gb Ethernet +RoCE

2 Milan High-Memory

amem

x86_64 AMD Milan

1

64

1

16

N/A

0

416G SSD

2x25 Gb Ethernet +RoCE

2 Milan High-Memory

amem

x86_64 AMD Milan

2

128

1

16

N/A

0

70G SSD

HDR-100 InfiniBand (200Gb inter-node fabric)

4 Milan NVIDIA GPU

aa100

x86_64 AMD Milan

1

64

1

3.8

NVIDIA A100

3

416G SSD

2x25 Gb Ethernet +RoCE

3 Milan NVIDIA GPU

al40

x86_64 AMD Milan

2

64

1

3.8

NVIDIA L40

3

416G SSD

2x25 Gb Ethernet +RoCE

Note

CU Anschutz job submission limit: CU Anschutz users on Alpine are subject to a campus-wide hard limit of 200 concurrent jobs across all QoS types (including normal, which system-wide allows up to 1000 jobs/user). This limit applies to all job types and partitions. It was implemented to reduce queue wait times for all Anschutz users given limited core-hour availability. If you need to run large numbers of jobs, consider using GNU Parallel as a workaround.

Colorado State University contribution#

Count & Type

Partition

Processor

Sockets

Cores (total)

Threads per Core

RAM per Core (GB)

GPU type

GPU count

Local Disk Capacity & Type

Fabric

28 Milan General CPU

amilan

x86_64 AMD Milan

2

48

1

3.8

N/A

0

416G SSD

HDR-100 InfiniBand (200Gb inter-node fabric)

49 Milan General CPU

amilan

x86_64 AMD Milan

2

32

1

3.8

N/A

0

416G SSD

2x25 Gb Ethernet +RoCE

Requesting Hardware Resources#

Resources are requested within jobs by passing in SLURM directives, or resource flags, to either a job script (most common) or to the command line when submitting a job. Below are some common resource directives for Alpine (summarized then detailed):

Partitions#

Nodes with the same hardware configuration are grouped into partitions. You specify a partition using the --partition SLURM directive in your job script (or at the command line when submitting an interactive job) in order for your job to run on the appropriate type of node.

Note

Partitions available on Alpine:#

Partition

Description

# of nodes

cores/node

RAM/core (GB)

Billing_weight/core

amilan

AMD Milan (default)

403

32 or 48 or 64 or 128

3.8

1

ami100

GPU-enabled (3x AMD MI100)

8

64

3.8

6.13

aa100

GPU-enabled (3x NVIDIA A100)4. For select nodes, MIG has been enabled providing 6x 20 GB NVIDIA A100 MIG instances.

12

64

3.8

6.13

al40

GPU-enabled (3x NVIDIA L40)4

3

64

3.8

6.13

amem1

High-memory

24

48 or 64 or 128

162

4.0

acompile

AMD Milan compile nodes

2

64

3.8

N/A

atesting

AMD Milan test nodes

2; Pulls from CU amilan pool

64

3.8

0.025

gh200

NVIDIA Grace-Hopper (GH200) nodes

Note: this partition is only available upon request, please submit a support request form.

2

72

6.65

Billed at roughly twice the rate of our A100s

Important

Partition table footnotes:

1The amem partition requires the use of either the mem-normal or mem-long QOS. These QOS require that each job request 256GB of RAM or more.

2The amem partition has a mixture of nodes with 48, 64, and 128 cores. Nodes with 48 and 64 cores have 1 TB of RAM; nodes with 128 cores have 2 TB of RAM. The default RAM-per-requested core on the amem partition is 15,927 MB, which is configured such that if you request all 64 (128) cores on a 64-core (128-core) amem node, you will receive roughly 1,000,000 MB of RAM (i.e., the full ~1 TB available). If you request all 48 cores on a 48-core node, by default you will receive 764,496 MB of RAM, which is less than the 1 TB available. If you require more RAM than the default of 15,927 MB per-requested-core, employ the --mem flag in your job script and specify the amount of RAM you need, in MB. For example, to request all of the RAM on a node, use “–mem=1000000M”.

3On the GPU partitions, ami100, aa100, and al40, the billing_weight value of 6.1/core is an aggregate estimate and will be smaller for MIG instances. In practice, users are billed 1.0 for each core they request and an amount for each GPU they request (which is defined by GPU type). For the amount charged per GPU type, see the Billing_weight/GPU column in the table provided in the section Available GRES on Alpine. For example, if a user requests all 64 cores and three a100-40gb GPUs for one hour, they will be billed (1.0 * 64) + (108.6 * 3)=389.8 SUs.

4NVIDIA A100 and L40 GPUs only support CUDA versions >11.x

All users, regardless of institution, should specify partitions as follows:

--partition=amilan
--partition=aa100
--partition=ami100
--partition=al40
--partition=amem

Quality of Service (qos)#

Quality of Service or QoS is used to constrain or modify the characteristics that a job can have. For example, by selecting the long QoS, a user can place the job in a lower priority queue with a max wall time increased from 24 hours to 7 days.

Available QoS for Alpine:#

QOS name

Description

Max walltime

Max jobs/user

Max hardware/user

Valid Partitions

normal

Standard QoS for non-testing partitions

1 day

1000

128 nodes

amilan

long

Longer wall times

7 days

200

20 nodes

amilan

mem-normal

Standard QoS for High-memory jobs

24 hours

1000

256 CPU cores

amem

mem-long

QoS for longer running High-memory jobs

7 days

200

185 CPU cores

amem

gpu-normal

Standard QoS for GPU jobs

24 hours

1000

see Available GRES on Alpine

aa100,ami100,al40

gpu-long

QoS for longer running GPU jobs

7 days

200

see Available GRES on Alpine

aa100,ami100,al40

gpu-testing

Testing QoS for GPU jobs

1 hour

5

see Available GRES on Alpine

aa100,ami100

testing

Used for all testing partitions

1 hour

5

2 nodes

atesting

compile

Used for acompile jobs

12 hours

4

1 node

acompile

gh200

Used for GH200 jobs

Note: this QoS is only available upon request, please submit a support request form.

7 days

1

1 node

gh200

QoS examples#

--qos=normal

--qos=long

General Resources (gres)#

General resources allows for fine-grain hardware specifications. On Alpine, the gres directive is required to use GPU accelerators on GPU nodes. The general form for specifying gres is as follows: --gres=gpu:<GRES type>:N. In this general form, <GRES type> specifies the type of GPU you want to run on within a given partition, and N is the number of GPUs you want to request. In the table below, we specify the available GRES types for each partition and common constraints associated with them.

Note

Alpine GPU resources and configurations can be viewed as follows on a login node (with the slurm/alpine module loaded):

$ sinfo --Format Partition,Gres |grep gpu

Available GRES on Alpine:#

GRES type

Description

Partition

gpu-normal GPU Resources

gpu-long GPU Resources

gpu-testing GPU Resources

Max cores/GPU

Billing_weight/GPU

a100_3g.20gb

NVIDIA A100 GPU with 20 GB of VRAM made possible by NVIDIA’s Multi-Instance GPU (MIG) feature

aa100

N/A

N/A

  • Total: 6
  • Max/user: 1

10

54.3

a100-40gb

NVIDIA A100 GPU with 40 GB of VRAM

aa100

  • Total: 18
  • Max/user: 6

  • Total: 6
  • Max/user: 3

N/A

21

108.6

a100_80gb

NVIDIA A100 GPU with 80 GB of VRAM

aa100

  • Total: 10
  • Max/user: 3

  • Total: 3
  • Max/user: 1

N/A

21

108.6

l40

NVIDIA L40 GPU with 48 GB of VRAM

al40

  • Total: 7
  • Max/user: 3

  • Total: 3
  • Max/user: 3

N/A

21

108.6

mi100

AMD MI100 GPU with 34 GB of VRAM

ami100

  • Total: 18
  • Max/user: 5

  • Total: 6
  • Max/user: 3

  • Total: 3
  • Max/user: 1

21

108.6

gh200

NVIDIA GH200 GPU with 102 GB of VRAM

gh200

N/A

N/A

N/A

72

260.64

Important

  • The Max/user value is the concurrent GPU limit per user for a given GRES type. Jobs exceeding this limit are held in the queue and will only be eligible to run once the user’s active GPU consumption falls below the threshold.

  • Resources belonging to gpu-testing are for verifying GPU workflows and building GPU-accelerated applications. Established workflows should be submitted to gpu-normal or gpu-long.

  • Resources requested via gpu-testing are currently only charged 10% of the provided CPU and GPU billing weights.

  • GH200 resources are part of the gh200 QoS, which is only available to users upon request. To request access, please submit a support request form.

Examples of GRES Usage#

Request a single A100 GPU with 40 GB of VRAM.

--gres=gpu:a100-40gb:1

Request multiple (in this case 3) MI100 GPUs.

--gres=gpu:mi100:3

Special-Purpose Resources#

To help users test out their workflows, CURC provides several special-purpose resources on Alpine. These resources enable users to quickly test or compile code on CPU and GPU compute nodes. To ensure equal access to these resources, the amount of resources (such as CPUs, GPUs, and runtime) are limited.

Important

Compiling and testing resources are, as their name implies, only meant for compiling code and testing workflows. They are not to be used outside of compiling or testing. Please utilize the appropriate resources when running code.

Special-Purpose CPU-only Resources#

CURC currently provides two types of special-purpose CPU-only resources on Alpine that are made available through the partitions atesting and acompile.

acompile usage#

acompile provides near-immediate access to limited resources for the purpose of viewing the module stack, verifying non-MPI jobs, and compiling software. Users can request up to 4 CPU cores (but no GPUs) for a maximum runtime of 12 hours. The partition can be quickly accessed with the acompile command, which launches an interactive compute session.

Get usage information for acompile.

acompile --help

Request 2 CPU cores for 2 hours.

acompile --ntasks=2 --time=02:00:00

atesting usage#

The atesting partition provides access to limited resources for the purpose of verifying workflows and MPI jobs. Users are able to request up to 2 CPU nodes (8 cores per node) for a maximum runtime of 1 hour and 16 CPUs.

Request one core per node for 10 minutes.

sinteractive --partition=atesting --ntasks=2 --ntasks-per-node=1 --nodes=2 --qos=testing --time=00:10:00

Request 4 cores for 30 minutes.

sinteractive --partition=atesting --ntasks=4 --nodes=1 --qos=testing --time=00:30:00

Request 2 nodes with 2 cores per node for 10 minutes - a good option for testing MPI jobs.

sinteractive --partition=atesting --ntasks=4 --ntasks-per-node=2 --nodes=2 --qos=testing --time=00:10:00

Special-Purpose GPU Resources#

The gpu-testing QoS provides access to limited GPU resources for the purpose of verifying GPU workflows and building GPU-accelerated applications. Please note that the gpu-testing QoS must be used in conjunction with a chosen GPU partition and GPU type. For a list of resources that are available via gpu-testing as well as limitations of the QoS, see the sections Quality of Service (qos) and General Resources (gres).

Important

  • The a100_3g.20gb GPU type made available on the gpu-testing QoS is a NVIDIA Multi-Instance GPU (MIG). MIG is a feature that can “slice” GPUs into multiple GPU instances. These GPU instances can be treated as a single GPU. The increase in available GPUs, and in effect increase in GPU access, provided by MIG does come with certain limitations. One important limitation is that MIG does not allow for multiple GPU instances to communicate with each other. This is the reason we limit users to just 1 GPU under the gpu-testing QoS for each GPU type. For more information on the limitations of MIG, please see NVIDIA’s MIG Application Considerations documentation.

  • Currently there are no testing resources for the L40 GPUs, however most workflows that successfully run on the aa100 resources will work on the al40 partition.

Request one 20 GB A100 MIG slice with 10 CPU cores for 30 minutes.

sinteractive --partition=aa100 --gres=gpu:a100_3g.20gb:1 --ntasks=10 --nodes=1 --qos=gpu-testing --time=00:30:00

Request one MI100 GPU with 21 CPU cores for one hour.

sinteractive --partition=ami100 --gres=gpu:mi100:1 --ntasks=21 --nodes=1 --qos=gpu-testing --time=00:60:00

Alpine is jointly funded by the University of Colorado Boulder, the University of Colorado Anschutz, Colorado State University, and the National Science Foundation (award 2201538).

Contents