-
Notifications
You must be signed in to change notification settings - Fork 0
Deployment 2
We have done our best to document all the commands we ran during deployment. The purpose of this document is mainly to assist with troubleshooting when something goes wrong with the server. Besides the following commands, you may also want to review the configuration files in this repository.
SSH Access is provided for pre-approved users on Vanderbilt networks only. The IP address is 10.33.2.13.
Make sure secure boot is disabled.
Recommended by Dell for data science workstations.
/usr/bin/gsettings set org.gnome.settings-daemon.plugins.power sleep-inactive-ac-timeout 0
sudo systemctl mask hibernate.target
sudo swapoff -v /swapfile
sudo sed -i ′/^\/swapfile/d′ /etc/fstab
sudo rm /swapfile
Nvidia drivers are finicky and often require a reboot. Let's make sure nothing is updated without some supervision:
sudo dpkg-reconfigure unattended-upgrades
NVIDIA's directions at time of deployment.
sudo apt-get install linux-headers-$(uname -r)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get updat
esudo apt-get -y install cuda-toolkit-12-3
sudo apt-get install -y nvidia-kernel-open-545
sudo apt-get install -y cuda-drivers-545
We installed the debian package over the network.
Add the following to PATH:
export PATH=/usr/local/cuda-12.3/bin${PATH:+:${PATH}}
We followed the getting started guide.
sudo snap install microk8s --classic --channel=1.29
sudo usermod -a -G microk8s $USER
su $USER
Move the Kube config to a shared location:
sudo mkdir /srv/jupyter/.kube
microk8s kubectl config view --raw > /srv/jupyter/.kube/config
sudo chown -R $USER:microk8s /srv/jupyter
Make sure to add that to your PATH:
export KUBECONFIG=/srv/jupyter/.kube/config
microk8s enable dns storage helm3 gpu ingress
sudo reboot
type -p curl >/dev/null || sudo apt install curl -y
curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \
&& sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
&& sudo apt update \
&& sudo apt install gh -y
I ran the following command, which should permanently add the KUBECONFIG location to PATH for all users... (export is temporary)
echo "KUBECONFIG=/srv/jupyter/.kube/config">>/etc/environment
If you need to a run a new service on the machine, you can request a route from Vanderbilt IT here: https://it.vanderbilt.edu/services/catalog/infrastructure/network/load_balancing.php