Updating NVIDIA Driver and CUDA Toolkit

Steps to install new NVIDIA Driver and CUDA Toolkit This procedure must be executed as root on a machine that is not running jobs.

Draining the nodes

The compute node must be drained in advance on SLURM

On SLURM, the command scontrol can modify the state of the machine by specifying the node and the new state. You must provide a reason when disabling a node.

• Disable:

  [trcis001 ~]# scontrol update NodeName=tcogq[001-006] State=DRAIN Reason="CUDA UPDATE"
  

• Enable:

  [trcis001 ~]# scontrol update NodeName=tcogq[001-006] State=RESUME
  

Updating the NVIDIA Driver and CUDA Toolkit

Connect via SSH to the compute node and execute these steps:

1. Stop daemon that uses nvidia kernel module

#> systemctl stop nvidia-dcgm

2. Remove all modules depending on nvidia

#> rmmod nvidia_drm nvidia_modeset nvidia_uvm nvidia

3. Install NVIDIA Driver and CUDA Toolkit

#> sh /shared/src/cuda_12.2.1_535.86.10_linux.run --silent --driver
#> sh /shared/src/cuda_12.2.1_535.86.10_linux.run --silent --toolkit

4. Restart the machine

#> reboot

Verify NVIDIA Driver and CUDA Toolkit versions

On the infraestructure machine verify the versions using Ansible

[trcis009 ~]# ansible thorny-gpu -m shell -a "nvidia-smi | grep CUDA"
tcogq001.hpc.wvu.edu | CHANGED | rc=0 >>
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |

tcogq002.hpc.wvu.edu | CHANGED | rc=0 >>
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |

tcogq003.hpc.wvu.edu | CHANGED | rc=0 >>
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |

tcogq006.hpc.wvu.edu | CHANGED | rc=0 >>
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |

tcogq004.hpc.wvu.edu | CHANGED | rc=0 >>
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |

tcogq005.hpc.wvu.edu | CHANGED | rc=0 >>
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |

tbmgq001.hpc.wvu.edu | CHANGED | rc=0 >>
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |

tbegq201.hpc.wvu.edu | CHANGED | rc=0 >>
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |

tbegq202.hpc.wvu.edu | CHANGED | rc=0 >>
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |

tbmgq100.hpc.wvu.edu | CHANGED | rc=0 >>
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |

tbegq200.hpc.wvu.edu | CHANGED | rc=0 >>
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |