Should I commit .lock file changes separately? What should I write for the commit message?
Question:
I’m using poetry for my Python package manager but I believe this would apply to any programming practices.
I’ve been doing this without knowing exactly what I’m doing, or how I should be doing.
When you use a package manager and install a new package, there’s usually a .lock file change to keep your build deterministic.
Usually, I would commit these changes like:
$ git add poetry.lock pyproject.toml
$ git commit -m "Install packages: beautifulsoup4"
i.e, I make a commit every time I install/remove a package. I do it because I FEEL like this is what I should do, but I have 0 clue if this is actually a correct way to handle this.
Am I doing great? or is there any other specific convention & rules I should abide by to make it follow the best practices as close as possible?
Answers:
The official recommendation of the poetry maintainers is to commit the lockfile if you develop a deployable application (as opposed to a library).
That being said, my personal experience has been that it isn’t necessary to commit lockfiles to VCS. The pyproject.toml file is the reference for correct build instructions, and the lockfile is the reference for a single successful deployment. Now, I don’t know what the spec for poetry.lock is, but I had them backfire on me often enough during collaboration with colleagues in ways where only deleting them would fix the problem.
A usual problem was that using different operation systems or python versions would lead to different lockfiles, and that just doesn’t fly. I’ll gladly let our CI build and persist an authoritative reference-lockfile to enable re-builds, but it still wouldn’t be committed to the repository.
If maintaining a shared lockfile is viable given your workflow – great! You avoid a step in your pipeline, have one less artifact to worry about, and cut down dramatically on build time (even a medium-size project can take minutes to do a full dependency-resolution).
But as far as best practices go, I’d consider adding poetry.lock to the .gitignore a better practice than what you do, and only commit pyproject.toml changes when you add dependencies.
Poetry actually has a section about this on their site: https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
They sugest that it’s important to actually commit this file, since that will show which versions of the libraries included were used at the time of the commit.
Committing this file to VC is important because it will cause anyone who sets up the project to use the exact same versions of the dependencies that you are using. Your CI server, production machines, other developers in your team, everything and everyone runs on the same dependencies, which mitigates the potential for bugs affecting only some parts of the deployments. Even if you develop alone, in six months when reinstalling the project you can feel confident the dependencies installed are still working even if your dependencies released many new versions since then. (See note below about using the update command.)
For libraries it is not necessary to commit the lock file.
Commit your poetry.lock file to version control. Committing this file to VC is important because it will cause anyone
who sets up the project to use the exact same versions of the
dependencies that you are using. Your CI server, production machines,
other developers in your team, everything and everyone runs on the
same dependencies, which mitigates the potential for bugs affecting
only some parts of the deployments. Even if you develop alone, in six
months when reinstalling the project you can feel confident the
dependencies installed are still working even if your dependencies
released many new versions since then… For libraries, it is not necessary to commit the lock file.
However, if your team uses different operating systems, hardware/CPU types etc, and you do not develop with container tech such as Docker. I would advise against committing the lock file as that would cause a lot of problem for the team. It is fine if your team builds with Docker. For example, if your lock file contains information about a library built specifically for Linux, it becomes a problem [to install from the lock file] on Windows.
Until now, all of the given answers missed one important difference between Poetry’s lock file and other tools like pip freeze: Poetry’s lock file is an universal lock file.
This means that Poetry doesn’t care about the current environment, neither the Python version in use, nor the platform. Instead it makes sure that dependencies are resolvable within the given Python version range in pyproject.toml. This results in a lock file that is valid on any platform with a Python version within the range given in the pyproject.toml.
This difference to other tools, that produces lock file, is also the reason why Poetry is slower in resolving dependencies. This is also the reason why it is recommended to check in the poetry.lock in your vcs. Doing so, it speed up setting up your development environment and you make sure your environment is reproducible.
I’m using poetry for my Python package manager but I believe this would apply to any programming practices.
I’ve been doing this without knowing exactly what I’m doing, or how I should be doing.
When you use a package manager and install a new package, there’s usually a .lock file change to keep your build deterministic.
Usually, I would commit these changes like:
$ git add poetry.lock pyproject.toml
$ git commit -m "Install packages: beautifulsoup4"
i.e, I make a commit every time I install/remove a package. I do it because I FEEL like this is what I should do, but I have 0 clue if this is actually a correct way to handle this.
Am I doing great? or is there any other specific convention & rules I should abide by to make it follow the best practices as close as possible?
The official recommendation of the poetry maintainers is to commit the lockfile if you develop a deployable application (as opposed to a library).
That being said, my personal experience has been that it isn’t necessary to commit lockfiles to VCS. The pyproject.toml file is the reference for correct build instructions, and the lockfile is the reference for a single successful deployment. Now, I don’t know what the spec for poetry.lock is, but I had them backfire on me often enough during collaboration with colleagues in ways where only deleting them would fix the problem.
A usual problem was that using different operation systems or python versions would lead to different lockfiles, and that just doesn’t fly. I’ll gladly let our CI build and persist an authoritative reference-lockfile to enable re-builds, but it still wouldn’t be committed to the repository.
If maintaining a shared lockfile is viable given your workflow – great! You avoid a step in your pipeline, have one less artifact to worry about, and cut down dramatically on build time (even a medium-size project can take minutes to do a full dependency-resolution).
But as far as best practices go, I’d consider adding poetry.lock to the .gitignore a better practice than what you do, and only commit pyproject.toml changes when you add dependencies.
Poetry actually has a section about this on their site: https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
They sugest that it’s important to actually commit this file, since that will show which versions of the libraries included were used at the time of the commit.
Committing this file to VC is important because it will cause anyone who sets up the project to use the exact same versions of the dependencies that you are using. Your CI server, production machines, other developers in your team, everything and everyone runs on the same dependencies, which mitigates the potential for bugs affecting only some parts of the deployments. Even if you develop alone, in six months when reinstalling the project you can feel confident the dependencies installed are still working even if your dependencies released many new versions since then. (See note below about using the update command.)
For libraries it is not necessary to commit the lock file.
Commit your
poetry.lockfile to version control. Committing this file to VC is important because it will cause anyone
who sets up the project to use the exact same versions of the
dependencies that you are using. Your CI server, production machines,
other developers in your team, everything and everyone runs on the
same dependencies, which mitigates the potential for bugs affecting
only some parts of the deployments. Even if you develop alone, in six
months when reinstalling the project you can feel confident the
dependencies installed are still working even if your dependencies
released many new versions since then… For libraries, it is not necessary to commit the lock file.
However, if your team uses different operating systems, hardware/CPU types etc, and you do not develop with container tech such as Docker. I would advise against committing the lock file as that would cause a lot of problem for the team. It is fine if your team builds with Docker. For example, if your lock file contains information about a library built specifically for Linux, it becomes a problem [to install from the lock file] on Windows.
Until now, all of the given answers missed one important difference between Poetry’s lock file and other tools like pip freeze: Poetry’s lock file is an universal lock file.
This means that Poetry doesn’t care about the current environment, neither the Python version in use, nor the platform. Instead it makes sure that dependencies are resolvable within the given Python version range in pyproject.toml. This results in a lock file that is valid on any platform with a Python version within the range given in the pyproject.toml.
This difference to other tools, that produces lock file, is also the reason why Poetry is slower in resolving dependencies. This is also the reason why it is recommended to check in the poetry.lock in your vcs. Doing so, it speed up setting up your development environment and you make sure your environment is reproducible.