How to kill process on GPUs with PID in nvidia-smi using keyword?
Question:
How to kill running processes on GPUs for a specific program (e.g. python) in terminal?
For example two processes are running with python in the top picture and kill them to see the bottom picture in nvidia-smi
Answers:
You can grep python in the nvidia-smi and then pass the PID to
the kill -9 command, e.g.
sudo kill -9 $( nvidia-smi | grep ‘python’ | sed -n
‘s/|s*[0-9]s([0-9])s.*/1/p’ | sed ‘/^$/d’)
The accepted answer doesn’t work for me, probably because nvidia-smi has different formats across different versions/hardware.
I’m using a much cleaner command:
nvidia-smi | grep 'python' | awk '{ print $3 }' | xargs -n1 kill -9
You can replace $3 in the awk expression to fit your nvidia-smi output. It is the n-th column in which the PIDs occur.
Use nvidia-smi or top command to see processes running and to kill command:
sudo kill -9 PID
I guess the question is already answered when nvidia-smi shows processes occupying GPU mem. For me, even though nvidia-smi wasnt showing any processes, GPU memory was being used and I wanted to kill them.
The way to go in this case was to use the fuser command to find out the processes using the particular GPU device. In my case I wanted to kill all the processes using the GPU device 3.
This can be done using the command :
sudo fuser -k /dev/nvidia3
You can use -ki to kill the processes interactively.
As one of other answers suggest you can use: (replace 5 with the column number where process id exists)
nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9
If you might have to use this a lot you can create an alias for the command: to do that do this you should edit ~/.bash_aliases file:
nano ~/.bash_aliases
and add the following line to it and save the file:
alias killgpuprocess="nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9"
then (just needed this time):
source ~/.bashrc
Then if you run
killgpuprocess
it will kill the existing processes on GPU(s).
How to kill running processes on GPUs for a specific program (e.g. python) in terminal?
For example two processes are running with python in the top picture and kill them to see the bottom picture in nvidia-smi
You can grep python in the nvidia-smi and then pass the PID to
the kill -9 command, e.g.
sudo kill -9 $( nvidia-smi | grep ‘python’ | sed -n
‘s/|s*[0-9]s([0-9])s.*/1/p’ | sed ‘/^$/d’)
The accepted answer doesn’t work for me, probably because nvidia-smi has different formats across different versions/hardware.
I’m using a much cleaner command:
nvidia-smi | grep 'python' | awk '{ print $3 }' | xargs -n1 kill -9
You can replace $3 in the awk expression to fit your nvidia-smi output. It is the n-th column in which the PIDs occur.
Use nvidia-smi or top command to see processes running and to kill command:
sudo kill -9 PID
I guess the question is already answered when nvidia-smi shows processes occupying GPU mem. For me, even though nvidia-smi wasnt showing any processes, GPU memory was being used and I wanted to kill them.
The way to go in this case was to use the fuser command to find out the processes using the particular GPU device. In my case I wanted to kill all the processes using the GPU device 3.
This can be done using the command :
sudo fuser -k /dev/nvidia3
You can use -ki to kill the processes interactively.
As one of other answers suggest you can use: (replace 5 with the column number where process id exists)
nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9
If you might have to use this a lot you can create an alias for the command: to do that do this you should edit ~/.bash_aliases file:
nano ~/.bash_aliases
and add the following line to it and save the file:
alias killgpuprocess="nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9"
then (just needed this time):
source ~/.bashrc
Then if you run
killgpuprocess
it will kill the existing processes on GPU(s).