Structure for developmental Python packages
Question:
I’m starting a brand new project. I’m splitting code of different functions (e.g., utils, database, main application) into their own respective packages such that other new projects can just add them as dependencies and import them in the future. Packages may have cross-dependencies (e.g., database depends on utils).
I know that I can build each component as a Python package and use pip to manage the dependencies. However, as I’m starting from scratch, I will be making active changes to all packages at the same time. It seems to me packaging them as a "proper package" would be quite inefficient. I envisage that I will need to add a new function in say utils, increment the version, use it database, then realise that I need another new function that belongs in utils, increment version again etc.
What would be the best way to structure the project in this scenario? I’m using conda, Python 3.10 and VSCode if that matters.
Answers:
I suggest the package approach you are thinking about. The key method you’re missing is to make editable installs locally for all your packages.
pip install -e . in package root directory
Editable installs will reflect their changes right away in your environment
Since you are using conda you probably want conda develop . like this answer suggests
I’m starting a brand new project. I’m splitting code of different functions (e.g., utils, database, main application) into their own respective packages such that other new projects can just add them as dependencies and import them in the future. Packages may have cross-dependencies (e.g., database depends on utils).
I know that I can build each component as a Python package and use pip to manage the dependencies. However, as I’m starting from scratch, I will be making active changes to all packages at the same time. It seems to me packaging them as a "proper package" would be quite inefficient. I envisage that I will need to add a new function in say utils, increment the version, use it database, then realise that I need another new function that belongs in utils, increment version again etc.
What would be the best way to structure the project in this scenario? I’m using conda, Python 3.10 and VSCode if that matters.
I suggest the package approach you are thinking about. The key method you’re missing is to make editable installs locally for all your packages.
pip install -e . in package root directory
Editable installs will reflect their changes right away in your environment
Since you are using conda you probably want conda develop . like this answer suggests