CUDA setup on Ubuntu 24.04 can be frustrating when things do not work as expected. Common problems include nvidia-smi returning nothing despite driver installation, nvcc failing despite CUDA being present, and PATH configurations that appear correct but don't resolve binaries.

These are the points where most setups break. This guide walks through the full installation step by step, so you do not run into those gaps. You will install the drivers, properly configure CUDA, and verify each stage to ensure the system works end to end.

#What is CUDA, and why do you need it?

CUDA (Compute Unified Device Architecture) is NVIDIA's platform that lets your code talk directly to GPU cores. It's the bridge that moves heavy computation from your CPU to the thousands of parallel cores on your GPU. In a typical system, the CPU handles sequential tasks. CUDA enables running large numerical computations on the GPU, where thousands of cores execute many operations simultaneously.

CUDA plays a central role in GPU-accelerated computing. It moves compute-intensive tasks from the CPU to thousands of more efficient GPU cores, resulting in significant performance gains. These tasks include matrix multiplication, tensor calculations, and batch data processing, where the same computation is applied to large amounts of data. Instead of processing each item one by one, CUDA runs these operations in parallel across many threads using the SIMT model.

Here's where you'll see CUDA powering real-world workloads:

• LLM training using frameworks such as NVIDIA NeMo and NeMo Curator
• LLM inference to deploy AI models with low latency using frameworks such as NVIDIA Dynamo
• AI-powered CAE simulations for accelerating your CAE simulations
• ataFrame and SQL acceleration using cuDF to speed up data processing in tools like pandas and Apache Spark
• Accelerated quantum computing using CUDA-Q for research and quantum simulation workloads
• Robotics and physical AI development for autonomous vehicles, robotics, and video analytics using platforms such as NVIDIA Cosmos
• Video Rendering and Encoding for hardware-accelerated video processing

#Why Ubuntu 24.04 is a popular choice for CUDA development

Ubuntu has become the de facto standard for CUDA development for several reasons:

• NVIDIA officially supports Ubuntu in its CUDA toolkit releases. This provides dedicated .deb packages along with repository configurations for each Ubuntu LTS version.
• Ubuntu 24.04 LTS (Noble Numbat) ships with Linux kernel 6.8 by default. This provides improved hardware support, which has been tested with recent NVIDIA driver versions.
• The vast majority of deep learning tutorials, cloud VM images, and Docker containers target Ubuntu.
• Ubuntu's long-term support model provides 5 years of security updates. This makes it suitable for both production and research environments.

#Prerequisites and system requirements

Before installing CUDA, confirm that your system meets the required hardware, software, and firmware conditions. Missing any of these can lead to driver load failures, toolkit errors, or runtime issues.

#Additional checks

1. Verify whether you have a CUDA-capable GPU model

Confirm that the system detects your GPU.

lspci | grep -i nvidia  

Example output

Output01:00.0 VGA compatible controller: NVIDIA Corporation GB206 [GeForce RTX 5060] (rev a1)
06:00.0 Audio device: NVIDIA Corporation GB206 High Definition Audio Controller (rev a1)

If no output appears, the GPU may not be installed correctly or may not be detected by the PCI bus.

2. Verify whether GCC is installed

The GCC compiler is required when you compile CUDA code on your system. If you are only running precompiled CUDA applications, gcc is not required. Most Linux systems already include gcc, so in many cases, no extra step is needed.

To check if GCC is available, run the command below.

gcc --version

Example Output

Outputgcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0

If the command fails, install the development tools package for your Linux distribution, which includes gcc and the required toolchain.

3. Check whether the system is up to date

Update packages to avoid dependency issues during installation.

sudo apt update && sudo apt upgrade -y

4. Disable Secure Boot (if applicable)
Secure Boot can block unsigned kernel modules such as NVIDIA drivers. Since CUDA depends on these drivers, installation may fail if Secure Boot is active. Disable it from BIOS/UEFI before proceeding.

Check status using the command below.

mokutil --sb-state

Example Output

OutputSecureBoot disabled  

Manual module signing is possible, though it adds complexity and increases failure risk.

#Removing previous NVIDIA drivers and CUDA versions

If NVIDIA drivers/CUDA were already installed, it is better to remove them first. Different install paths, such as Ubuntu repositories, .run files, or older CUDA versions, can cause failures during a new installation because those files do not always match the new installation.

This does not affect the following cases:

• Systems where CUDA has not been installed before
• Systems where the same installation method is used (RPM/deb > RPM/deb or runfile > runfile)

The table below shows how different installation methods interact.

#Remove existing NVIDIA drivers

Remove any NVIDIA driver packages.

sudo apt remove --purge '^nvidia-.*' -y
sudo apt remove --purge '^libnvidia-.*' -y

#Remove old CUDA toolkit versions

Remove previously installed CUDA components.

sudo apt remove --purge '^cuda.*' -y
sudo apt remove --purge '^libcuda.*' -y
sudo apt remove --purge '^cudnn.*' -y

#Clean up residual configuration files and repositories

Some configuration files remain even after uninstalling packages. Remove them manually.

sudo rm -f /etc/apt/sources.list.d/cuda*.list
sudo rm -f /etc/apt/sources.list.d/nvidia*.list
sudo rm -f /usr/share/keyrings/cuda-*

Then run autoremove to remove unused packages.

sudo apt autoremove -y && sudo apt autoclean

#Reboot the system after cleanup

After removing the drivers and packages, reboot the system to clear any previously loaded kernel modules.

sudo reboot

After the restart, the system is ready for the new CUDA installation.

#Installing NVIDIA drivers on Ubuntu 24.04

Let’s start by installing the correct NVIDIA driver, since CUDA depends on a working driver before anything else. There are two main ways to install NVIDIA drivers on Ubuntu.

#Method 1: Using the Ubuntu-drivers utility (recommended for most users)

Ubuntu already comes with a helper called ubuntu-drivers. You don’t have to guess driver versions. It checks your GPU and recommends the best option.

If it’s missing, just install it.

sudo apt install ubuntu-drivers-common -y

Now, list the recommended drivers.

ubuntu-drivers devices | grep nvidia

Example output

Outputdriver : nvidia-driver-595-server-open - distro non-free
driver : nvidia-driver-580-server-open - distro non-free
driver : nvidia-driver-595 - distro non-free recommended
driver : nvidia-driver-580 - distro non-free
driver : nvidia-driver-595-open - distro non-free
driver : nvidia-driver-595-server - distro non-free
driver : nvidia-driver-580-open - distro non-free
driver : nvidia-driver-580-server - distro non-free

You’ll get a list. One of them will be marked as recommended. That’s usually the safe pick.

To install that driver, use the command below.

sudo ubuntu-drivers autoinstall

Note - There is an important caveat. Starting with the NVIDIA GeForce RTX 50-series (Blackwell architecture, e.g., RTX 5060, 5070, 5080, and 5090), NVIDIA requires the open kernel module variant of the driver. The proprietary kernel module will fail to initialize the GPU.

As of mid-2026, Ubuntu’s recommended driver flag does not properly account for Blackwell hardware. It may recommend the proprietary driver, which is incorrect for these GPUs.

If you are using a GeForce RTX 50-series GPU, run the following command.

sudo apt install nvidia-driver-595 -y

Example output

The following NEW packages will be installed:

Outputadwaita-icon-theme at-spi2-common at-spi2-core dconf-gsettings-backend dconf-service dkms fontconfig gsettings-desktop-schemas gtk-update-icon-cache hicolor-icon-theme
humanity-icon-theme libatk-bridge2.0-0t64 libatk1.0-0t64 libatspi2.0-0t64 libavahi-client3 libavahi-common-data libavahi-common3 libcairo-gobject2 libcairo2 libcolord2 libcups2t64
libdatrie1 libdconf1 libepoxy0 libfontenc1 libgdk-pixbuf-2.0-0 libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgraphite2-3 libgtk-3-0t64 libgtk-3-bin libgtk-3-common libharfbuzz0b
libice6 liblcms2-2 libnvidia-cfg1-595 libnvidia-common-595 libnvidia-compute-595 libnvidia-decode-595 libnvidia-egl-wayland1 libnvidia-encode-595 libnvidia-extra-595
libnvidia-fbc1-595 libnvidia-gl-595 libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libpixman-1-0 librsvg2-2 librsvg2-common libsm6 libthai-data libthai0 libvdpau1
libwayland-cursor0 libwayland-egl1 libwayland-server0 libxaw7 libxcb-render0 libxcomposite1 libxcursor1 libxcvt0 libxdamage1 libxfixes3 libxfont2 libxi6 libxinerama1 libxkbfile1
libxmu6 libxnvctrl0 libxrandr2 libxrender1 libxt6t64 libxtst6 mesa-vdpau-drivers nvidia-compute-utils-595 nvidia-dkms-595-open nvidia-driver-595-open nvidia-firmware-595-595.58.03
nvidia-kernel-common-595 nvidia-kernel-source-595-open nvidia-prime nvidia-settings nvidia-utils-595 ocl-icd-libopencl1 screen-resolution-extra session-migration ubuntu-mono
vdpau-driver-all x11-common x11-xkb-utils xcvt xfonts-base xfonts-encodings xfonts-utils xserver-common xserver-xorg-core xserver-xorg-video-nvidia-595

#Rebooting and verifying driver installation

Once the install finishes, reboot.

sudo reboot

After the system is back, check if everything has loaded correctly by running the following command

nvidia-smi

If things are fine, you’ll see a table output. It shows your GPU, driver version, CUDA version reported by the driver, memory usage, and any running processes.

Example output.

OutputFri May 8 03:55:17 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.58.03 Driver Version: 595.58.03 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5060 Off | 00000000:01:00.0 Off | N/A |
| 31% 37C P8 5W / 145W | 2MiB / 8151MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+


+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|

One small thing here. The CUDA version shown in nvidia-smi indicates the highest CUDA version supported by the installed driver. It is not the current version of the CUDA Toolkit installed on your system.

#Installing the CUDA toolkit

Now that the driver is ready, the next step is to install the CUDA toolkit, which can be done using two main methods.

#Method 1: Repository-based installation (APT)

This is the standard way most people use. Packages come through APT, so updates and removals stay simple. You can use either a network repository or a local repository. The workflow is almost the same after setup.

This approach pulls packages directly from NVIDIA’s repository.

Step 1: Add the CUDA repository keyring

NVIDIA provides a keyring package. It sets up the repository and handles the GPG key.

wget [https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb](https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb)

Example Output

Outputhttps://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
Resolving developer.download.nvidia.com (developer.download.nvidia.com)... 23.62.33.31, 23.62.33.19
Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|23.62.33.31|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4328 (4.2K) [application/x-deb]
Saving to: ‘cuda-keyring_1.1-1_all.deb’


cuda-keyring_1.1-1_all.deb 100%[=================================================================================================>] 4.23K --.-KB/s in 0s


2026-05-08 04:22:35 (174 MB/s) - ‘cuda-keyring_1.1-1_all.deb’ saved [4328/4328]

sudo dpkg -i cuda-keyring_1.1-1_all.deb

Example Output

Output(Reading database ... 140134 files and directories currently installed.)
Preparing to unpack cuda-keyring_1.1-1_all.deb ...
Unpacking cuda-keyring (1.1-1) ...
Setting up cuda-keyring (1.1-1) …

Step 2: Update package lists

sudo apt update

Step 3: Install CUDA

To install the latest toolkit

sudo apt install cuda-toolkit -y

Example Output

The following NEW packages will be installed:

Outputalsa-topology-conf alsa-ucm-conf ca-certificates-java cuda-cccl-13-2 cuda-command-line-tools-13-2 cuda-compiler-13-2 cuda-crt-13-2 cuda-cudart-13-2 cuda-cudart-dev-13-2
cuda-culibos-dev-13-2 cuda-cuobjdump-13-2 cuda-cupti-13-2 cuda-cupti-dev-13-2 cuda-cuxxfilt-13-2 cuda-documentation-13-2 cuda-driver-dev-13-2 cuda-gdb-13-2 cuda-libraries-13-2
cuda-libraries-dev-13-2 cuda-nsight-13-2 cuda-nsight-compute-13-2 cuda-nsight-systems-13-2 cuda-nvcc-13-2 cuda-nvdisasm-13-2 cuda-nvml-dev-13-2 cuda-nvprune-13-2 cuda-nvrtc-13-2
cuda-nvrtc-dev-13-2 cuda-nvtx-13-2 cuda-opencl-13-2 cuda-profiler-api-13-2 cuda-sandbox-dev-13-2 cuda-sanitizer-13-2 cuda-tileiras-13-2 cuda-toolkit cuda-toolkit-13-2
cuda-toolkit-13-2-config-common cuda-toolkit-13-config-common cuda-toolkit-config-common cuda-tools-13-2 cuda-visual-tools-13-2 default-jre default-jre-headless fonts-dejavu-extra
gds-tools-13-2 java-common libasound2-data libasound2t64 libatk-wrapper-java libatk-wrapper-java-jni libcublas-13-2 libcublas-dev-13-2 libcufft-13-2 libcufft-dev-13-2
libcufile-13-2 libcufile-dev-13-2 libcuobjclient-13-2 libcuobjclient-dev-13-2 libcurand-13-2 libcurand-dev-13-2 libcusolver-13-2 libcusolver-dev-13-2 libcusparse-13-2
libcusparse-dev-13-2 libgif7 libnpp-13-2 libnpp-dev-13-2 libnvfatbin-13-2 libnvfatbin-dev-13-2 libnvjitlink-13-2 libnvjitlink-dev-13-2 libnvjpeg-13-2 libnvjpeg-dev-13-2
libnvptxcompiler-13-2 libnvvm-13-2 libpcsclite1 libxcb-cursor0 libxcb-dri2-0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-render-util0 libxcb-shape0 libxcb-util1
libxcb-xinerama0 libxcb-xinput0 libxcb-xkb1 libxft2 libxkbcommon-x11-0 libxv1 libxxf86dga1 nsight-compute-2026.1.1 nsight-systems-2025.6.3 openjdk-21-jre openjdk-21-jre-headless
x11-utils

The terminal will show you package downloads and installations.

If you need a specific version, use the command below.

sudo apt install cuda-toolkit-13-2 -y

#Method 2: Runfile installer (.run)

This method does not use APT. Everything is handled through a single installer file.

It works well in offline environments or systems without package management. At the same time, it does not integrate with Ubuntu’s package system, so updates and removals take more effort.

Download the .run file from: https://developer.nvidia.com/cuda-downloads

Click ‘Linux’ > x86_64 > Ubuntu > 24.04 > runfile(local)

You will get instructions on how to download and run a self-contained local installer, which installs CUDA on your system. As of April 30, 2026, these are the download and setup instructions.

The download size is quite huge (4.1GB). Ensure you have a fast and stable connection.

wget https://developer.download.nvidia.com/compute/cuda/13.2.1/local_installers/cuda_13.2.1_595.58.03_linux.run
sudo sh cuda_13.2.1_595.58.03_linux.run

Alternatively, you can run:

chmod +x cuda_13.2.1_595.58.03_linux.run
sudo ./cuda_13.2.1_595.58.03_linux.run

The installer will guide you through the steps. You can install only the toolkit if the driver is already present.

Note: This method is less integrated with Ubuntu package management. Future updates or removals can be harder compared to the repository method.

#Understanding CUDA meta-packages

When you browse the repository, you will see several similar package names. Each one installs a different set of components.

• cuda
  Installs everything. This includes the toolkit, drivers, and all related libraries. This is the most complete option.
• cuda-toolkit
  Installs only the development tools. That includes the compiler (nvcc), profilers, debuggers, and math libraries. It does not install or change the NVIDIA driver.
• cuda-drivers
  Installs only the NVIDIA driver. No CUDA development tools are included. This is useful when you only need GPU support to run applications.

For most environments, a common approach is to install the driver first using ubuntu-drivers, then install cuda-toolkit. This keeps driver management separate and gives better control.

#Setting up environment variables and PATH configuration

CUDA is installed at this point. Files are already under /usr/local/cuda/. That path usually falls under /usr/local/cuda-13.x/.

But the shell does not look there by default. So, nvcc is not picked up. You run it. Nothing shows. The same thing happens when code tries to load CUDA libraries. This is not a missing install. It is simply a path issue.

#Add CUDA to PATH

Open the shell config using your nano editor.

nano ~/.bashrc

Go to the bottom of the file and add the following.

#CUDA configuration

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

If you installed a specific CUDA version, point to it directly. For example, use /usr/local/cuda-13.2. This avoids ambiguity later when multiple versions are present.

#Apply the changes

No restart needed. Just reload using the source command.

source ~/.bashrc

#Verify the PATH configuration

Check what PATH contains.

echo $PATH

You should see //usr/local/cuda-13.2/bin listed.

Then check whether the CUDA compiler is accessible.

nvcc --version

If it prints version details, you are done.

Example Output:

Outputnvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Thu_Mar_19_11:12:51_PM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.78
Build cuda_13.2.r13.2/compiler.37668154_0

If you reach this point, the shell can see CUDA. That is all this step is about.

#Verifying the CUDA installation

At this stage, you are checking whether CUDA was installed and whether the toolchain is actually usable. That means:

• The compiler should resolve correctly
• The driver stack should be live
• The GPU should respond to a kernel launch

#Compile and run a small CUDA test

If you want the most direct proof, run a small CUDA program. It confirms the compiler can build device code. It also confirms the runtime can launch work on the GPU.

Create a test file.

cat << 'EOF' > hello_cuda.cu
#include <stdio.h>
  __global__ void hello_kernel() {
  printf("Hello from GPU thread %d!\n", threadIdx.x);
}
int main() {
  hello_kernel<<<1, 4>>>();
  cudaDeviceSynchronize();
  return 0;
}
EOF

Now compile it and execute it.

nvcc hello_cuda.cu -o hello_cuda && ./hello_cuda

If everything is working correctly, you should see output like this.

OutputHello from GPU thread 0!
Hello from GPU thread 1!
Hello from GPU thread 2!
Hello from GPU thread 3!

#Where to find reliable NVIDIA GPU servers for your CUDA development?

If you plan to run CUDA workloads in production or test at scale, you need access to dedicated GPU infrastructure. Cherry Servers provides bare-metal GPU servers built for AI, ML, data processing, and high-performance computing. You can deploy in minutes or customize the stack based on your workload.

Here are some commonly used GPU options:

• NVIDIA A2 Tensor Core - 1280 CUDA cores with 16GB GDDR6. Good fit for lightweight inference tasks.

• NVIDIA A10 Tensor Core - 9216 CUDA cores with 24GB GDDR6. Handles mixed AI workloads and model serving.

• NVIDIA A16 Tensor Core - 5120 CUDA cores with 64GB GDDR6. Designed for multi-user or virtualized GPU workloads.

• NVIDIA A40 - 10752 CUDA cores with 48GB GDDR6. Suitable for large model training and rendering pipelines.

• NVIDIA A100 - 6912 CUDA cores with 80GB HBM2. Built for deep learning training and HPC workloads.

• NVIDIA Tesla P4 - 2560 CUDA cores with 8GB GDDR6. Works well for inference and media processing.

Key benefits you get:

• Full [bare-metal]https://www.cherryservers.com/bare-metal-dedicated-servers) access with no virtualization overhead
• Fast provisioning with pre-built servers ready in minutes
• API-driven management for automation and DevOps workflows
• Flexible billing based on usage or fixed periods
• Built-in DDoS protection and high network capacity
• Large monthly traffic allocation with scalable storage options

If you want to test CUDA projects without managing hardware, this setup gives you direct GPU access with full control. You can configure, deploy, and start running workloads right away.

Sign up and try a GPU instance to see how your CUDA workloads perform in a dedicated environment.