Spruce Knob (2018-2023) ======================= Spruce Knob is a 3,376 core cluster spread over 176 nodes (155 active nodes with 3090 cores on February 2022) using a shared FDR Infiniband Network. The system is a heterogeneous cluster in which there are different node types. In addition, each year a new addition is added to the cluster, which is known as a new phase. The cluster has five phase. New hardware is no longer being added to this cluster. Hardware that is still under warranty will be serviced. Hardware that is out of warranty will be removed from the cluster upon failure. Overview -------- Resource manager and system scheduler ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Torque v. 6.1.x * Moab Cluster Manager v. 9.1.x Total Compute Resources ~~~~~~~~~~~~~~~~~~~~~~~ * 3,376 Cores * 176 Compute Nodes * 4 Management Nodes * 18 GPUs Shared Interconnect ~~~~~~~~~~~~~~~~~~~ * Mellanox SX6512 Infiniband Switch * 216 ports all non-blocking * FDR (56 Gb/s) speed * All Fibre Connections Hardware -------- Spruce Knob is a heterogeneous cluster with compute nodes with a variety of Intel Microarchitectures The table below shows the differences between the 4 architectures: .. csv-table:: Microarchitectures present on *Spruce Knob* :header: "Microarchitecture", "Launched", "Extensions" :widths: 10, 10, 80 :width: 90% Sandy Bridge, 2011, "MMX, SSE, SSE2, SSE3, SSSE3, SSE4, SSE4.1, SSE4.2, **AVX**, VT-x, VT-d, AES-NI, CLMUL, TXT" Ivy Bridge, 2012, "MMX, SSE, SSE2, SSE3, SSSE3, SSE4, SSE4.1, SSE4.2, **AVX, F16C**, AES-NI, CLMUL, RDRAND, TXT, VT-x, VT-d" Haswell, 2013, "MMX, SSE, SSE2, SSE3, SSSE3, SSE4, SSE4.1, SSE4.2, **AVX, AVX2**, FMA3, VT-x, VT-d, AES-NI, CLMUL, RDRAND, TXT" Broadwell, 2014, "MMX, SSE, SSE2, SSE3, SSSE3, SSE4, SSE4.1, SSE4.2, **AVX, AVX2**, TSX, FMA3, AES-NI, CLMUL, RDRAND, TXT, VT-x, VT-d" The Advanced Vector Extensions (AVX, also known as Sandy Bridge New Extensions) are extensions to the x86 instruction set architecture for microprocessors from Intel and Advanced Micro Devices (AMD) proposed by Intel in March 2008 and first supported by Intel with the Sandy Bridge processor shipping in Q1 2011. AVX provides new features, new instructions and a new coding scheme. AVX2 (also known as Haswell New Instructions) expands most integer commands to 256 bits and introduces fused multiply–accumulate operations (FMA). They were first supported by Intel with the Haswell processor, which shipped in 2013. The AVX and AVX2 are important for Scientific Computing because AVX uses sixteen YMM registers to perform a Single Instruction on Multiple pieces of Data. Each YMM register can hold and do simultaneous operations (math) on eight 32-bit single-precision floating point numbers or four 64-bit double-precision floating point numbers. Notice that all nodes on Spruce support AVX extensions but not all nodes support AVX2. Phase 0/1 Hardware ~~~~~~~~~~~~~~~~~~ * `Intel(R) Xeon(R) CPU E5-4620 `_ * `Intel(R) Xeon(R) CPU E5-2650 v2 `_ +---------------+--------------------------------------------------------+---------------------------+ | Node Type | **Description** | **Compute Nodes** | | | | | +===============+========================================================+===========================+ | Sandy Bridge | - 4 X 8 Intel(R) Xeon(R) CPU E5-4620 0 @ 2.20GHz | 14 | | | - 64GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ | Ivy Bridge | - 2 X 8 Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz | 1 | | | - 32GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ | Ivy Bridge | - 2 X 8 Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz | 95 | | | - 64GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ | Ivy Bridge | - 2 X 8 Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz | 1 | | | - 256GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ | Ivy Bridge | - 2 X 8 Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz | 3 | | | - 512GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ Phase 2 Hardware ~~~~~~~~~~~~~~~~ * `Intel(R) Xeon(R) CPU E5-2650 v3 `_ +---------------+--------------------------------------------------------+---------------------------+ | Node Type | **Description** | **Compute Nodes** | | | | | +===============+========================================================+===========================+ | Haswell | - 2 X 10 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz | 14 | | | - 64GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ | Haswell | - 2 X 10 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz | 3 | | | - 128GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ | Haswell | - 2 X 10 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz | 1 | | | - 512GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ Phase 3 Hardware ~~~~~~~~~~~~~~~~ * `Intel(R) Xeon(R) CPU E5-2650 v3 `_ +---------------+--------------------------------------------------------+---------------------------+ | Node Type | **Description** | **Compute Nodes** | | | | | +===============+========================================================+===========================+ | Haswell | - 2 X 10 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz | 1 | | | - 64GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ | Haswell | - 2 X 10 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz | 10 | | | - 128GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ | Haswell | - 2 X 10 Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz | 1 | | | - 512GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ Phase 4 Hardware ~~~~~~~~~~~~~~~~ * `Intel(R) Xeon(R) CPU E5-2650 v4 `_ +---------------+--------------------------------------------------------+---------------------------+ | Node Type | **Description** | **Compute Nodes** | | | | | +===============+========================================================+===========================+ | Broadwell | - 2 X 12 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz | 3 | | | - 64GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ | Broadwell | - 2 X 12 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz | 7 | | | - 128GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ | Broadwell | - 2 X 12 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz | 1 | | | - 512GB memory | | | | - 1TB HDD | | +---------------+--------------------------------------------------------+---------------------------+ Queues ------ +-------------------+-----------------+ | **Queue** | **Walltime** | +===================+=================+ | debug | 15:00 | +-------------------+-----------------+ | standby | 4:00:00 | +-------------------+-----------------+ | comm_mmem_week | 168:00:00 | +-------------------+-----------------+ | comm_256g_mem | 168:00:00 | +-------------------+-----------------+ | comm_mmem_day | 24:00:00 | +-------------------+-----------------+ | comm_gpu | 168:00:00 | +-------------------+-----------------+ | comm_smp | 168:00:00 | +-------------------+-----------------+ | comm_large_mem | 168:00:00 | +-------------------+-----------------+ Research Team Queues -------------------- Research teams that have bought their own compute nodes have private queues that link all their compute nodes together. Only users given permission from the research team's buyer (Usually the labs PI) will have permission to directly submit jobs to these queues. While these are private queues - unused resources/compute nodes from these queues will be available to the standby queue (see below). However, per the system-wide policies, all research team's compute nodes must be available to the research team's users within 4 hours of job submission. By default, these queues are regulated by first come, first serve queuing. However, individual research teams can ask for different settings for their respective queue, and should `contact `__ the RC HPC team with these requests. Standby Queue ------------- The standy queue is for using resources from research teams queues that are not currently being used. Priority on the standby queue is set by fair share queuing. This means that user priority is assigned based on a combination of the size of the job and how much system resources the user have used during the given week, with higher priority assigned to larger jobs and/or user jobs that have used fewer system resources in the week. Further, the standby queue has a 4 hour wall time. Community Node Queues --------------------- Spruce Knob has several queues that start with the word 'comm'. These queues are linked to the 51 compute nodes bought using NSF funding sources, and as such is open for Statewide Academic use. These queues are separated by node type (i.e. large memory, gpu, smp) and can be used by all users. Currently, these nodes are regulated by fair share queuing. This means that user priority is assigned based on a combination of the size of the job and how much system resources the user have used during the given week, with higher priority assigned to larger jobs and/or user jobs that have used less system resources in the week. Further, all community queues have a 24 hour wall time, except for the week long medium memory queue (comm\_mmem\_week). comm\_mmem\_week allows jobs up to a week (168 hours); however, this queue class also limits the maximum number of nodes to 11, and a single user can not exceed 80 CPUs total within this queue. These restrictions are set to prevent a single user occupying a large number of the community resources for an excessively long time.