Python Module not Found from `systemd` service Script
Question:
With the dozens of previous answers, I’m surprised I couldn’t find anything…
I’m using the paho mqtt library for a very simply python program to report out some data (running on a Raspberry Pi). My import from within the python program (my_program.py) is:
import paho.mqtt.client as mqtt
If I run the program from the command line using python my_program.py it runs without error. However, I’m trying to get it setup as a system service to manage its execution. I’ve done this a dozen times with similar python programs, setting up the various bash scripts and service files. They all work except this one. I mention this because I don’t think it is related to the bash or the service per se. For completeness, here is the setup.
From a bash script/systemd that runs –
## Service (my_program.service):
ExecStart=/home/pi/my_program.sh
## and bash (my_program.sh)
python /home/pi/my/directory/my_program.py
When I go to start the service, I get:
pi@ArmstrongSE:/etc/systemd/system $ sudo systemctl status my_program
* my_program.service
Loaded: loaded (/etc/systemd/system/my_program.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2019-11-23 13:59:58 PST; 22s ago
Process: 31100 ExecStart=/home/pi/my_program.sh (code=exited, status=1/FAILURE)
Main PID: 31100 (code=exited, status=1/FAILURE)
Nov 23 13:59:54 ArmstrongSE systemd[1]: Started my_program.service.
Nov 23 13:59:54 ArmstrongSE my_program.sh[31100]: Starting MQTT Transmitter
Nov 23 13:59:55 ArmstrongSE my_program.sh[31100]: /home/pi/data/solar/20191123135605.json
Nov 23 13:59:58 ArmstrongSE my_program.sh[31100]: Traceback (most recent call last):
Nov 23 13:59:58 ArmstrongSE my_program.sh[31100]: File "/home/pi/my/directory/my_program.py", line 25, in <module>
Nov 23 13:59:58 ArmstrongSE my_program.sh[31100]: import paho.mqtt.client as mqtt
Nov 23 13:59:58 ArmstrongSE my_program.sh[31100]: ImportError: No module named paho.mqtt.client
To confirm that the path assignments are there – –
From the python interpreter I get:
>>> import paho.mqtt.client as mqtt
>>> print(mqtt.__file__)
/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.pyc
The sys.path reports:
>>> import sys
>>> print(sys.path)
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-arm-linux-gnueabihf',
'/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
'/home/pi/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/gtk-2.0']
I’m stuck… Any thoughts on why this won’t load?
Update/Clarification:
I can run the bash script from anywhere (any directory) and it works. So, it must be something in the systemd setup (?).
Answers:
Try to install module with root.
This may be a linking issue that python has. If you were to install a library via pip, it may or may not link or install properly. I would suggest re-installing the mqtt library using something like…
sudo -H pip3 install --system paho-mqtt
Also, I noticed that your import is wrong. You wrote: import pho.mqtt.client as mqtt.
It should be :
import paho.mqtt.client as mqtt
I think I figured out what the issue is… systemd runs its services by default as root. Which would/should, I would think (see below), allow for the equivalent of running things as root using sudo for another user or the actual root user on the cli.
Just focusing on the services file for the systemd, I found in the man pages that there is a [Service] parameter that allows you to designate who runs the service. In this case, I tried setting the parameter as:
[Service]
User=pi
ExecStart=/home/pi/my_program.sh
This allowed the service to run without the error (the error noted above).
I can also use these settings and execute without error:
[Service]
User=root
ExecStart=/home/pi/my_program.sh
Why:
For others who find themselves here, if you omit the User= parameter from the [Service] section the default is root. HOWEVER, that is different than setting the User= parameter to root (e.g., User=root) or to another user (in my case pi).
I don’t really understand the nuances, but from the documentation on environmental variables, the User= parameter is used to set the $PATH variable. if the service file doesn’t have a User= set I think it causes a problem with the path assignments. I think that’s why setting User= to either pi or root solves this problem, while using the default seems to cause the problem.
Explanations and education welcome. Thanks.
With the dozens of previous answers, I’m surprised I couldn’t find anything…
I’m using the paho mqtt library for a very simply python program to report out some data (running on a Raspberry Pi). My import from within the python program (my_program.py) is:
import paho.mqtt.client as mqtt
If I run the program from the command line using python my_program.py it runs without error. However, I’m trying to get it setup as a system service to manage its execution. I’ve done this a dozen times with similar python programs, setting up the various bash scripts and service files. They all work except this one. I mention this because I don’t think it is related to the bash or the service per se. For completeness, here is the setup.
From a bash script/systemd that runs –
## Service (my_program.service):
ExecStart=/home/pi/my_program.sh
## and bash (my_program.sh)
python /home/pi/my/directory/my_program.py
When I go to start the service, I get:
pi@ArmstrongSE:/etc/systemd/system $ sudo systemctl status my_program
* my_program.service
Loaded: loaded (/etc/systemd/system/my_program.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2019-11-23 13:59:58 PST; 22s ago
Process: 31100 ExecStart=/home/pi/my_program.sh (code=exited, status=1/FAILURE)
Main PID: 31100 (code=exited, status=1/FAILURE)
Nov 23 13:59:54 ArmstrongSE systemd[1]: Started my_program.service.
Nov 23 13:59:54 ArmstrongSE my_program.sh[31100]: Starting MQTT Transmitter
Nov 23 13:59:55 ArmstrongSE my_program.sh[31100]: /home/pi/data/solar/20191123135605.json
Nov 23 13:59:58 ArmstrongSE my_program.sh[31100]: Traceback (most recent call last):
Nov 23 13:59:58 ArmstrongSE my_program.sh[31100]: File "/home/pi/my/directory/my_program.py", line 25, in <module>
Nov 23 13:59:58 ArmstrongSE my_program.sh[31100]: import paho.mqtt.client as mqtt
Nov 23 13:59:58 ArmstrongSE my_program.sh[31100]: ImportError: No module named paho.mqtt.client
To confirm that the path assignments are there – –
From the python interpreter I get:
>>> import paho.mqtt.client as mqtt
>>> print(mqtt.__file__)
/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.pyc
The sys.path reports:
>>> import sys
>>> print(sys.path)
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-arm-linux-gnueabihf',
'/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
'/home/pi/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/gtk-2.0']
I’m stuck… Any thoughts on why this won’t load?
Update/Clarification:
I can run the bash script from anywhere (any directory) and it works. So, it must be something in the systemd setup (?).
Try to install module with root.
This may be a linking issue that python has. If you were to install a library via pip, it may or may not link or install properly. I would suggest re-installing the mqtt library using something like…
sudo -H pip3 install --system paho-mqtt
Also, I noticed that your import is wrong. You wrote: import pho.mqtt.client as mqtt.
It should be :
import paho.mqtt.client as mqtt
I think I figured out what the issue is… systemd runs its services by default as root. Which would/should, I would think (see below), allow for the equivalent of running things as root using sudo for another user or the actual root user on the cli.
Just focusing on the services file for the systemd, I found in the man pages that there is a [Service] parameter that allows you to designate who runs the service. In this case, I tried setting the parameter as:
[Service]
User=pi
ExecStart=/home/pi/my_program.sh
This allowed the service to run without the error (the error noted above).
I can also use these settings and execute without error:
[Service]
User=root
ExecStart=/home/pi/my_program.sh
Why:
For others who find themselves here, if you omit the User= parameter from the [Service] section the default is root. HOWEVER, that is different than setting the User= parameter to root (e.g., User=root) or to another user (in my case pi).
I don’t really understand the nuances, but from the documentation on environmental variables, the User= parameter is used to set the $PATH variable. if the service file doesn’t have a User= set I think it causes a problem with the path assignments. I think that’s why setting User= to either pi or root solves this problem, while using the default seems to cause the problem.
Explanations and education welcome. Thanks.