Best practice for reusing python code
Question:
I have write a python library app(which contains several *.py files). And several of my python projects need to reuse the code in the library app. What’s the recommended best practice for reusing python code? Currently I have thought out three options:
- Copy and paste. This is far away from best practice. It violates the
DRY principle.(Don’t repeat yourself.)
- Add the folder of the library app to the environment variable PYTHONPATH:
export PYTHONPATH=/path/to/library/app. Then every projects on the same computer can reference the code in the library app.
- And the folder of the library app to sys.path in python code:
sys.path.append('/path/to/library/app')
Among the three options above which one do you prefer? What advantage does it have compared to the other two options? Do you have any other better options? It is much appreciated that if some one with years of python development experiences could answer this question.
Answers:
If its a shared library you should package it up and add it to site-packages and then you wont have to worry about setting anything up. This is the best option.
If you dont want to use site-packages, then use PYTHONPATH. Its why it exists, and it is the way to do what you want.
You might want to look into using site.addsitedir, path.append does not prevent duplicates. It will also allow you to leverage .pth files.
Dynamically setting/adding things to PYTHONPATH via sys.path achieves the same result, but it can complicate things if you choose to reuse your new library. It also causes issues if your paths change, you have to change code versus an environment variable. Unless it is completely necessary, you should not dynamically setup your python path.
Copy & Paste is not a re-use pattern. You aren’t reusing anything you are duplicating and increasing maintenance.
The first way is, as you noted yourself, hardly acceptable as it has countless well-known problems.
The other two have their own problems. For starters, they require manual work when files move (particular bad if you mix it with the application logic, i.e. put it in *.py files instead of leaving it to the computer it runs on) and requires either fixed installation locations (absolute paths) or at least a certain directory structure (relative paths). IMHO, these ways are only acceptable if the applications aren’t ever going to move anywhere else. As soon as that becomes required, you should give up and use the existing solution as the one reason not to use it, being overkill for small and local scripts, doesn’t apply any more:
Make the common parts, which you apparently already treat as a free-standing library (good!), a fully-fledged project in its own right, with a setup.py that allows installing and adding to PYTHONPATH in a cross-platform way with a single command. You don’t need to actually publish it at PyPI, but it makes doing so easier if you should change your mind in the future. Should you do so and also publish some of your projects on PyPI, you also made installing the project in question and its dependencies easier for every potential user.
Allow me to propose a fourth alternative: take the time to learn how to package your library and install it in your site-packages; it’s easier than one may think and I’m convinced it’s time well spent. This is a very good starting point: https://packaging.python.org/en/latest/
Of your three options, PYTHONPATH is the way to go. Copy & paste is clearly out, and adding code to your other projects to modify sys.path simply pollutes those files with knowledge about their environment.
A fourth option is, create a true installable package from your common code, and install it into your Python installation. Then you can simply import those modules like any other 3rd-party install code.
I have write a python library app(which contains several *.py files). And several of my python projects need to reuse the code in the library app. What’s the recommended best practice for reusing python code? Currently I have thought out three options:
- Copy and paste. This is far away from best practice. It violates the
DRY principle.(Don’t repeat yourself.) - Add the folder of the library app to the environment variable PYTHONPATH:
export PYTHONPATH=/path/to/library/app. Then every projects on the same computer can reference the code in the library app. - And the folder of the library app to sys.path in python code:
sys.path.append('/path/to/library/app')
Among the three options above which one do you prefer? What advantage does it have compared to the other two options? Do you have any other better options? It is much appreciated that if some one with years of python development experiences could answer this question.
If its a shared library you should package it up and add it to site-packages and then you wont have to worry about setting anything up. This is the best option.
If you dont want to use site-packages, then use PYTHONPATH. Its why it exists, and it is the way to do what you want.
You might want to look into using site.addsitedir, path.append does not prevent duplicates. It will also allow you to leverage .pth files.
Dynamically setting/adding things to PYTHONPATH via sys.path achieves the same result, but it can complicate things if you choose to reuse your new library. It also causes issues if your paths change, you have to change code versus an environment variable. Unless it is completely necessary, you should not dynamically setup your python path.
Copy & Paste is not a re-use pattern. You aren’t reusing anything you are duplicating and increasing maintenance.
The first way is, as you noted yourself, hardly acceptable as it has countless well-known problems.
The other two have their own problems. For starters, they require manual work when files move (particular bad if you mix it with the application logic, i.e. put it in *.py files instead of leaving it to the computer it runs on) and requires either fixed installation locations (absolute paths) or at least a certain directory structure (relative paths). IMHO, these ways are only acceptable if the applications aren’t ever going to move anywhere else. As soon as that becomes required, you should give up and use the existing solution as the one reason not to use it, being overkill for small and local scripts, doesn’t apply any more:
Make the common parts, which you apparently already treat as a free-standing library (good!), a fully-fledged project in its own right, with a setup.py that allows installing and adding to PYTHONPATH in a cross-platform way with a single command. You don’t need to actually publish it at PyPI, but it makes doing so easier if you should change your mind in the future. Should you do so and also publish some of your projects on PyPI, you also made installing the project in question and its dependencies easier for every potential user.
Allow me to propose a fourth alternative: take the time to learn how to package your library and install it in your site-packages; it’s easier than one may think and I’m convinced it’s time well spent. This is a very good starting point: https://packaging.python.org/en/latest/
Of your three options, PYTHONPATH is the way to go. Copy & paste is clearly out, and adding code to your other projects to modify sys.path simply pollutes those files with knowledge about their environment.
A fourth option is, create a true installable package from your common code, and install it into your Python installation. Then you can simply import those modules like any other 3rd-party install code.