Generally speaking, how are (Python) projects structured?
Question:
I’m a bit lost when it comes to structuring my project(s). I try to structure things in ways that make sense, but always end up restructuring the whole thing at least twice per day. Granted, my projects aren’t very big, but I would love to not have to restructure everything and just settle on something for once.
I’ll describe my current program to try to make sense of things. It’s a graphical program with a database backend for calculating the price of sails. Not everything is written yet, but the user will be able to select a sail category and model from two dropdown menus. Depending on the category-model combination, the program will present checkboxes and spinboxes. These checkboxes and spinboxes, when changed, draw information from a database and present the price of having that checkbox checked or having a certain number (e.g., area in square metres) in the spinbox.
In its current form, the project looks like this:
COPYING
README.md
SailQt.pyw (Should program be called from here ...)
sailqt/
__init__.py (This holds a __version__ string)
SailQt.pyw (... or here?)
gui/
__init__.py
MainWindow.py (This needs access to a __version__ string)
MainWindow_rc.py
OptionsWidget.py
ui_MainWindow.py
ui_OptionsWidget.py
resources/
__init__.py
database.db
generate_gui.py
MainWindow.ui
MainWindow.qrc
OptionsWidget.ui
icons/
logo.png
To further clarify. resources holds all .ui files made in Qt Designer. They are XML files that describe the GUI. They can be converted to Python scripts with a terminal tool, which I’ve embedded into generate_gui.py. The same goes for .qrc files. generate_gui.py places the autogenerated files in the gui folder with either prefix ui_ or suffix _rc. database.db is currently empty, but will eventually be used to hold prices and everything.
MainWindow.py and OptionsWidget.py are Python files that hold objects of the same name, minus the .py suffix. MainWindow holds OptionsWidget in its display surface. Both objects use their corresponding ui and rc files.
SailQt.pyw is the file that makes a MainWindow instance, tells it to show itself, and then tells (Py)Qt to enter its loop and take over from there. It’s basically much like a .exe file of a lot of graphical applications in that it’s a small file that gets the program running.
My initial guess was to place SailQt.pyw inside the sailqt folder. But then MainWindow.py suddenly needed access to a __version__ string. The only way I could figure out how to achieve that was to move SailQt.pyw to the root folder of my project, and to let MainWindow.py import sailqt.__version__. But considering that was the nth time I had to shuffle things around and redo lines in most files to account for that tiny shuffle, I decided to just ask here.
My questions are fairly clear:
- How are, in general, Python projects structured? This pydoc link was helpful, but that seems more like a module to me than something that is actually executed by a user.
- Did I get the above structuring right?
- Bonus points for answering this, as it’s a bit off-topic. How come I can do
import os and then do stuff like os.system("sudo rm -rf /"), but I can’t do stuff like import sailqt and then do sailqt.gui.generate_gui.generate()?
Answers:
Let’s deal with your last question first, because it’s the most important as far as structuring python projects is concerned. Once you’ve sorted out how to get the imports working correctly within your project, the rest becomes much easier to deal with.
The key thing to understand is that the directory of the currently running script is automatically added to the start of sys.path. So if you put your main.py script (what you’re currently calling SailQt.pyw) outside of your package in a top-level container directory, it will guarantee that package imports will always work, no matter where the script is executed from.
So a minimal starting structure might look like this:
project/
main.py
package/
__init__.py
app.py
mainwindow.py
Now, because main.py must be outside of the top-level python package directory, it should contain only a minimal amout of code (just enough to get the program started). Given the above structure, that would mean not much more than this:
if __name__ == '__main__':
import sys
from package import app
sys.exit(app.run())
The app module would contain most of the actual code necessary to initialize the program and set up the gui, which would be imported like this:
from package.mainwindow import MainWindow
and this same form of fully qualified import statement can be used from anywhere with the package. So, for example, with this slightly more complicated structure:
project/
main.py
package/
__init__.py
app.py
mainwindow.py
utils.py
dialogs/
search.py
the search module could import a function from the utils module like this:
from package.utils import myfunc
On the specific issue of accessing the __version__ string: for a PyQt program, you could put the following at the top of the app module:
QtGui.QApplication.setApplicationName('progname')
QtGui.QApplication.setApplicationVersion('0.1')
and then access the name/version later like this:
name = QtGui.qApp.applicationName()
version = QtGui.qApp.applicationVersion()
The other issues with your current structure are mainly to do with maintaining separation between code files and resource files.
Firstly: the package tree should only contain code files (i.e. python modules). The resource files belong in the project directory (i.e. outside the package). Secondly: files containing code generated from resources (e.g. by pyuic or pyrcc) should probably go in a separate sub-package (this also makes it easy for your version control tool to exclude them). This would result in an overall project structure like this:
project/
db/
database.db
designer/
mainwindow.ui
icons/
logo.png
LICENSE
Makefile
resources.qrc
main.py
package/
__init__.py
app.py
mainwindow.py
ui/
__init__.py
mainwindow_ui.py
resources_rc.py
Here, the Makefile (or equivalent) is responsible for generating the ui/rc files, compiling the python modules, installing/uninstalling the program, etc. Resources needed by the program at runtime (such as the database file), will need to be installed in a standard location that your program knows how to find (e.g. something like /usr/share/progname/database.db on Linux). At installation-time, the Makefile will also need to generate an executable bash script (or equivalent) that knows where your program is and how to start it. That is, something like:
#!/bin/sh
exec 'python' '/usr/share/progname/main.py' "$@"
which would obviously need to be installed as /usr/bin/progname (or whatever).
This may seem like quite a lot to deal with at first, but of course the major benefit of finding a project structure that works well, is that you can re-use it for all future projects (and start to develop your own templates and tools for setting up and managing those projects).
I’m a bit lost when it comes to structuring my project(s). I try to structure things in ways that make sense, but always end up restructuring the whole thing at least twice per day. Granted, my projects aren’t very big, but I would love to not have to restructure everything and just settle on something for once.
I’ll describe my current program to try to make sense of things. It’s a graphical program with a database backend for calculating the price of sails. Not everything is written yet, but the user will be able to select a sail category and model from two dropdown menus. Depending on the category-model combination, the program will present checkboxes and spinboxes. These checkboxes and spinboxes, when changed, draw information from a database and present the price of having that checkbox checked or having a certain number (e.g., area in square metres) in the spinbox.
In its current form, the project looks like this:
COPYING
README.md
SailQt.pyw (Should program be called from here ...)
sailqt/
__init__.py (This holds a __version__ string)
SailQt.pyw (... or here?)
gui/
__init__.py
MainWindow.py (This needs access to a __version__ string)
MainWindow_rc.py
OptionsWidget.py
ui_MainWindow.py
ui_OptionsWidget.py
resources/
__init__.py
database.db
generate_gui.py
MainWindow.ui
MainWindow.qrc
OptionsWidget.ui
icons/
logo.png
To further clarify. resources holds all .ui files made in Qt Designer. They are XML files that describe the GUI. They can be converted to Python scripts with a terminal tool, which I’ve embedded into generate_gui.py. The same goes for .qrc files. generate_gui.py places the autogenerated files in the gui folder with either prefix ui_ or suffix _rc. database.db is currently empty, but will eventually be used to hold prices and everything.
MainWindow.py and OptionsWidget.py are Python files that hold objects of the same name, minus the .py suffix. MainWindow holds OptionsWidget in its display surface. Both objects use their corresponding ui and rc files.
SailQt.pyw is the file that makes a MainWindow instance, tells it to show itself, and then tells (Py)Qt to enter its loop and take over from there. It’s basically much like a .exe file of a lot of graphical applications in that it’s a small file that gets the program running.
My initial guess was to place SailQt.pyw inside the sailqt folder. But then MainWindow.py suddenly needed access to a __version__ string. The only way I could figure out how to achieve that was to move SailQt.pyw to the root folder of my project, and to let MainWindow.py import sailqt.__version__. But considering that was the nth time I had to shuffle things around and redo lines in most files to account for that tiny shuffle, I decided to just ask here.
My questions are fairly clear:
- How are, in general, Python projects structured? This pydoc link was helpful, but that seems more like a module to me than something that is actually executed by a user.
- Did I get the above structuring right?
- Bonus points for answering this, as it’s a bit off-topic. How come I can do
import osand then do stuff likeos.system("sudo rm -rf /"), but I can’t do stuff likeimport sailqtand then dosailqt.gui.generate_gui.generate()?
Let’s deal with your last question first, because it’s the most important as far as structuring python projects is concerned. Once you’ve sorted out how to get the imports working correctly within your project, the rest becomes much easier to deal with.
The key thing to understand is that the directory of the currently running script is automatically added to the start of sys.path. So if you put your main.py script (what you’re currently calling SailQt.pyw) outside of your package in a top-level container directory, it will guarantee that package imports will always work, no matter where the script is executed from.
So a minimal starting structure might look like this:
project/
main.py
package/
__init__.py
app.py
mainwindow.py
Now, because main.py must be outside of the top-level python package directory, it should contain only a minimal amout of code (just enough to get the program started). Given the above structure, that would mean not much more than this:
if __name__ == '__main__':
import sys
from package import app
sys.exit(app.run())
The app module would contain most of the actual code necessary to initialize the program and set up the gui, which would be imported like this:
from package.mainwindow import MainWindow
and this same form of fully qualified import statement can be used from anywhere with the package. So, for example, with this slightly more complicated structure:
project/
main.py
package/
__init__.py
app.py
mainwindow.py
utils.py
dialogs/
search.py
the search module could import a function from the utils module like this:
from package.utils import myfunc
On the specific issue of accessing the __version__ string: for a PyQt program, you could put the following at the top of the app module:
QtGui.QApplication.setApplicationName('progname')
QtGui.QApplication.setApplicationVersion('0.1')
and then access the name/version later like this:
name = QtGui.qApp.applicationName()
version = QtGui.qApp.applicationVersion()
The other issues with your current structure are mainly to do with maintaining separation between code files and resource files.
Firstly: the package tree should only contain code files (i.e. python modules). The resource files belong in the project directory (i.e. outside the package). Secondly: files containing code generated from resources (e.g. by pyuic or pyrcc) should probably go in a separate sub-package (this also makes it easy for your version control tool to exclude them). This would result in an overall project structure like this:
project/
db/
database.db
designer/
mainwindow.ui
icons/
logo.png
LICENSE
Makefile
resources.qrc
main.py
package/
__init__.py
app.py
mainwindow.py
ui/
__init__.py
mainwindow_ui.py
resources_rc.py
Here, the Makefile (or equivalent) is responsible for generating the ui/rc files, compiling the python modules, installing/uninstalling the program, etc. Resources needed by the program at runtime (such as the database file), will need to be installed in a standard location that your program knows how to find (e.g. something like /usr/share/progname/database.db on Linux). At installation-time, the Makefile will also need to generate an executable bash script (or equivalent) that knows where your program is and how to start it. That is, something like:
#!/bin/sh
exec 'python' '/usr/share/progname/main.py' "$@"
which would obviously need to be installed as /usr/bin/progname (or whatever).
This may seem like quite a lot to deal with at first, but of course the major benefit of finding a project structure that works well, is that you can re-use it for all future projects (and start to develop your own templates and tools for setting up and managing those projects).