How to see pip package sizes installed?
Question:
I’m not sure this is possible. Google does not seem to have any answers.
Running Linux Debian can I list all pip packages and size (amount of disk space used) thats installed?
i.e. List all pip
packages with size on disk?
Answers:
Go to the package site to find the size e.g. https://pypi.python.org/pypi/pip/json
Then expand releases
, find the version, and look up the size
(in bytes).
Could please try this one(A bit long though, maybe there are better solutions):
$ pip list | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{print $2 "/" tolower($1)}' | xargs du -sh 2> /dev/null
the output should look like this:
80K /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/blinker
3.8M /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/docutils
296K /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/ecdsa
340K /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/execnet
564K /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/fabric
1.4M /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/flask
316K /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/httplib2
1.9M /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/jinja2
...
should works if the package is installed in Location/Name
. (location and name are from pip show <package>
)
pip show <package>
will show you the location:
---
Metadata-Version: 2.0
Name: Flask
Version: 0.10.1
Summary: A microframework based on Werkzeug, Jinja2 and good intentions
Home-page: http://github.com/mitsuhiko/flask/
Author: Armin Ronacher
Author-email: [email protected]
License: BSD
Location: /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages
Requires: itsdangerous, Werkzeug, Jinja2
we get the Name
and Location
to join them to get the location, finally use du -sh
to get the package size.
History :
There is no command or applications developed for that purpose at the moment, we need to check that manually
Manual Method I :
du /usr/lib/python3.5/ --max-depth=2 | sort -h
du /usr/lib64/python3.5/ --max-depth=2 | sort -h
This does not include packages/files installed out of that directory, thus said we will get 95% with those 2 simples command
Also if you have other version of python installed, you need to adapt the directory
Manual Method II :
pip list | sed '/Package/d' | sed '/----/d' | sed -r 's/S+//2' | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{print $2 "/" $(find $2 -maxdepth 1 -iname $1)}' | xargs du -sh | sort -h
Search the install directory with the package name with case insensitive
Manual Method II Alternative I :
pip list | sed '/Package/d' | sed '/----/d' | sed -r 's/S+//2' | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - -| awk '{print $2 "/" tolower($1)}' | xargs du -sh | sort -h
Search the install directory with the package name with lowered case
Manual Method II Alternative II :
pip list | sed '/Package/d' | sed '/----/d' | sed -r 's/S+//2' | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - -| awk '{print $2 "/" $1}' | xargs du -sh | sort -h
Search the install directory with the package name
Note :
For methods using du
, output lines starting with du: cannot access
need to be checked manually;
The command use the install directory and add to it the name of the package but some times the package name and directory name are different…
Make it simple :
- Use first method then
- Use second method and just check manually package outside python classic directory
New version for new pip list format:
pip2 list --format freeze|awk -F = {'print $1'}| xargs pip2 show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{print $2 "/" tolower($1)}' | xargs du -sh 2> /dev/null|sort -h
Modified for pip version 18 and above:
pip list | tail -n +3 | awk '{print $1}' | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{print $2 "/" tolower($1)}' | xargs du -sh 2> /dev/null | sort -hr
This command shows pip packages, sorted by descending order of sizes.
Here’s how,
pip3 show numpy | grep "Location:"
- this will return path/to/all/packages
du -h path/to/all/packages
- last line will contain size of all packages in MB
Note: You may put any package name in place of numpy
All of the above solutions do not list packages with dashes in them: PIP converts them to underscores in the folder names:
pip list --format freeze | awk -F = {'print $1'} | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{gsub("-","_",$1); print $2 "/" tolower($1)}' | xargs du -sh 2> /dev/null | sort -h
And for Mac users:
pip3 list --format freeze | awk -F = {'print $1'} | xargs pip3 show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{gsub("-","_",$1); print $2 "/" tolower($1)}' | xargs du -sh 2> /dev/null | sort -h
How
$ du -h -d 1 "$(pip -V | cut -d ' ' -f 4 | sed 's/pip//g')" | grep -vE "dist-info|_distutils_hack|__pycache__" | sort -h
Pros
No need to convert these:
case (Django:django)
hyphen (django-q:django_q)
naming (djangorestframework-gis:rest_framework_gis)
Cons
Dependencies and some unknown directories revealed as well…
There is a simple Pythonic way to find it out though.
Here is the code. Let’s call this file pipsize.py
.
import os
import pkg_resources
def calc_container(path):
total_size = 0
for dirpath, dirnames, filenames in os.walk(path):
for f in filenames:
fp = os.path.join(dirpath, f)
total_size += os.path.getsize(fp)
return total_size
dists = [d for d in pkg_resources.working_set]
for dist in dists:
try:
path = os.path.join(dist.location, dist.project_name)
size = calc_container(path)
if size/1000 > 1.0:
print (f"{dist}: {size/1000} KB")
print("-"*40)
except OSError:
'{} no longer exists'.format(dist.project_name)
When run with python pipsize.py
this will print out something like,
pip 21.1.2: 8651.906 KB
----------------------------------------
numpy 1.20.3: 25892.871 KB
----------------------------------------
numexpr 2.7.3: 1627.361 KB
----------------------------------------
zict 2.0.0: 48.54 KB
----------------------------------------
yarl 1.6.3: 1395.888 KB
----------------------------------------
widgetsnbextension 3.5.1: 4609.962 KB
----------------------------------------
webencodings 0.5.1: 54.768 KB
----------------------------------------
wcwidth 0.2.5: 452.214 KB
----------------------------------------
uvicorn 0.14.0: 257.515 KB
----------------------------------------
tzlocal 2.1: 67.11 KB
----------------------------------------
traitlets 5.0.5: 800.71 KB
----------------------------------------
tqdm 4.61.0: 289.412 KB
----------------------------------------
tornado 6.1: 2898.264 KB
You can just run part 1 by it’s self for all the current packages python tool-size.py
will total them all up for you
If you want to know the exact size of a particular pip package including all its dependencies, i’ve created a little bash and python combo to achieve this
( based off the excellent package walking code answer above https://stackoverflow.com/a/67914559/3248788 )
Steps :
- create a python script to check all currently installed pip packages
- create a shell script to create a brand new python environment and install package to test, and run the script from step 1
- run shell script
- profit 🙂
Step 1
create a python script called tool-size.py
#!/usr/bin/env python
import os
import pkg_resources
def calc_container(path):
total_size = 0
for dirpath, dirnames, filenames in os.walk(path):
for f in filenames:
fp = os.path.join(dirpath, f)
total_size += os.path.getsize(fp)
return total_size
def calc_installed_sizes():
dists = [d for d in pkg_resources.working_set]
total_size = 0
print (f"Size of Dependencies")
print("-"*40)
for dist in dists:
# ignore pre-installed pip and setuptools
if dist.project_name in ["pip", "setuptools"]:
continue
try:
path = os.path.join(dist.location, dist.project_name)
size = calc_container(path)
total_size += size
if size/1000 > 1.0:
print (f"{dist}: {size/1000} KB")
print("-"*40)
except OSError:
'{} no longer exists'.format(dist.project_name)
print (f"Total Size (including dependencies): {total_size/1000} KB")
if __name__ == "__main__":
calc_installed_sizes()
Step 2
create a bash script called tool-size.sh
#!/usr/bin/env bash
# uncomment to to debug
# set -x
rm -rf ~/.virtualenvs/tool-size-tester
python -m venv ~/.virtualenvs/tool-size-tester
source ~/.virtualenvs/tool-size-tester/Scripts/activate
pip install -q $1
python tool-size.py
deactivate
Step 3
run script with package you want to get the size of
tool-size.sh xxx
say for truffleHog3
$ ./tool-size.sh truffleHog3
Size of Dependencies
----------------------------------------
truffleHog3 2.0.6: 56.46 KB
----------------------------------------
smmap 4.0.0: 108.808 KB
----------------------------------------
MarkupSafe 2.0.1: 40.911 KB
----------------------------------------
Jinja2 3.0.1: 917.551 KB
----------------------------------------
gitdb 4.0.7: 320.08 KB
----------------------------------------
Total Size (including dependencies): 1443.81 KB
I’m not sure this is possible. Google does not seem to have any answers.
Running Linux Debian can I list all pip packages and size (amount of disk space used) thats installed?
i.e. List all pip
packages with size on disk?
Go to the package site to find the size e.g. https://pypi.python.org/pypi/pip/json
Then expand releases
, find the version, and look up the size
(in bytes).
Could please try this one(A bit long though, maybe there are better solutions):
$ pip list | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{print $2 "/" tolower($1)}' | xargs du -sh 2> /dev/null
the output should look like this:
80K /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/blinker
3.8M /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/docutils
296K /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/ecdsa
340K /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/execnet
564K /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/fabric
1.4M /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/flask
316K /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/httplib2
1.9M /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages/jinja2
...
should works if the package is installed in Location/Name
. (location and name are from pip show <package>
)
pip show <package>
will show you the location:
---
Metadata-Version: 2.0
Name: Flask
Version: 0.10.1
Summary: A microframework based on Werkzeug, Jinja2 and good intentions
Home-page: http://github.com/mitsuhiko/flask/
Author: Armin Ronacher
Author-email: [email protected]
License: BSD
Location: /home/lord63/.pyenv/versions/2.7.11/envs/py2/lib/python2.7/site-packages
Requires: itsdangerous, Werkzeug, Jinja2
we get the Name
and Location
to join them to get the location, finally use du -sh
to get the package size.
History :
There is no command or applications developed for that purpose at the moment, we need to check that manually
Manual Method I :
du /usr/lib/python3.5/ --max-depth=2 | sort -h
du /usr/lib64/python3.5/ --max-depth=2 | sort -h
This does not include packages/files installed out of that directory, thus said we will get 95% with those 2 simples command
Also if you have other version of python installed, you need to adapt the directory
Manual Method II :
pip list | sed '/Package/d' | sed '/----/d' | sed -r 's/S+//2' | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{print $2 "/" $(find $2 -maxdepth 1 -iname $1)}' | xargs du -sh | sort -h
Search the install directory with the package name with case insensitive
Manual Method II Alternative I :
pip list | sed '/Package/d' | sed '/----/d' | sed -r 's/S+//2' | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - -| awk '{print $2 "/" tolower($1)}' | xargs du -sh | sort -h
Search the install directory with the package name with lowered case
Manual Method II Alternative II :
pip list | sed '/Package/d' | sed '/----/d' | sed -r 's/S+//2' | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - -| awk '{print $2 "/" $1}' | xargs du -sh | sort -h
Search the install directory with the package name
Note :
For methods using du
, output lines starting with du: cannot access
need to be checked manually;
The command use the install directory and add to it the name of the package but some times the package name and directory name are different…
Make it simple :
- Use first method then
- Use second method and just check manually package outside python classic directory
New version for new pip list format:
pip2 list --format freeze|awk -F = {'print $1'}| xargs pip2 show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{print $2 "/" tolower($1)}' | xargs du -sh 2> /dev/null|sort -h
Modified for pip version 18 and above:
pip list | tail -n +3 | awk '{print $1}' | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{print $2 "/" tolower($1)}' | xargs du -sh 2> /dev/null | sort -hr
This command shows pip packages, sorted by descending order of sizes.
Here’s how,
pip3 show numpy | grep "Location:"
- this will return path/to/all/packages
du -h path/to/all/packages
- last line will contain size of all packages in MB
Note: You may put any package name in place of numpy
All of the above solutions do not list packages with dashes in them: PIP converts them to underscores in the folder names:
pip list --format freeze | awk -F = {'print $1'} | xargs pip show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{gsub("-","_",$1); print $2 "/" tolower($1)}' | xargs du -sh 2> /dev/null | sort -h
And for Mac users:
pip3 list --format freeze | awk -F = {'print $1'} | xargs pip3 show | grep -E 'Location:|Name:' | cut -d ' ' -f 2 | paste -d ' ' - - | awk '{gsub("-","_",$1); print $2 "/" tolower($1)}' | xargs du -sh 2> /dev/null | sort -h
How
$ du -h -d 1 "$(pip -V | cut -d ' ' -f 4 | sed 's/pip//g')" | grep -vE "dist-info|_distutils_hack|__pycache__" | sort -h
Pros
No need to convert these:
case (Django:django)
hyphen (django-q:django_q)
naming (djangorestframework-gis:rest_framework_gis)
Cons
Dependencies and some unknown directories revealed as well…
There is a simple Pythonic way to find it out though.
Here is the code. Let’s call this file pipsize.py
.
import os
import pkg_resources
def calc_container(path):
total_size = 0
for dirpath, dirnames, filenames in os.walk(path):
for f in filenames:
fp = os.path.join(dirpath, f)
total_size += os.path.getsize(fp)
return total_size
dists = [d for d in pkg_resources.working_set]
for dist in dists:
try:
path = os.path.join(dist.location, dist.project_name)
size = calc_container(path)
if size/1000 > 1.0:
print (f"{dist}: {size/1000} KB")
print("-"*40)
except OSError:
'{} no longer exists'.format(dist.project_name)
When run with python pipsize.py
this will print out something like,
pip 21.1.2: 8651.906 KB
----------------------------------------
numpy 1.20.3: 25892.871 KB
----------------------------------------
numexpr 2.7.3: 1627.361 KB
----------------------------------------
zict 2.0.0: 48.54 KB
----------------------------------------
yarl 1.6.3: 1395.888 KB
----------------------------------------
widgetsnbextension 3.5.1: 4609.962 KB
----------------------------------------
webencodings 0.5.1: 54.768 KB
----------------------------------------
wcwidth 0.2.5: 452.214 KB
----------------------------------------
uvicorn 0.14.0: 257.515 KB
----------------------------------------
tzlocal 2.1: 67.11 KB
----------------------------------------
traitlets 5.0.5: 800.71 KB
----------------------------------------
tqdm 4.61.0: 289.412 KB
----------------------------------------
tornado 6.1: 2898.264 KB
You can just run part 1 by it’s self for all the current packages python tool-size.py
will total them all up for you
If you want to know the exact size of a particular pip package including all its dependencies, i’ve created a little bash and python combo to achieve this
( based off the excellent package walking code answer above https://stackoverflow.com/a/67914559/3248788 )
Steps :
- create a python script to check all currently installed pip packages
- create a shell script to create a brand new python environment and install package to test, and run the script from step 1
- run shell script
- profit 🙂
Step 1
create a python script called tool-size.py
#!/usr/bin/env python
import os
import pkg_resources
def calc_container(path):
total_size = 0
for dirpath, dirnames, filenames in os.walk(path):
for f in filenames:
fp = os.path.join(dirpath, f)
total_size += os.path.getsize(fp)
return total_size
def calc_installed_sizes():
dists = [d for d in pkg_resources.working_set]
total_size = 0
print (f"Size of Dependencies")
print("-"*40)
for dist in dists:
# ignore pre-installed pip and setuptools
if dist.project_name in ["pip", "setuptools"]:
continue
try:
path = os.path.join(dist.location, dist.project_name)
size = calc_container(path)
total_size += size
if size/1000 > 1.0:
print (f"{dist}: {size/1000} KB")
print("-"*40)
except OSError:
'{} no longer exists'.format(dist.project_name)
print (f"Total Size (including dependencies): {total_size/1000} KB")
if __name__ == "__main__":
calc_installed_sizes()
Step 2
create a bash script called tool-size.sh
#!/usr/bin/env bash
# uncomment to to debug
# set -x
rm -rf ~/.virtualenvs/tool-size-tester
python -m venv ~/.virtualenvs/tool-size-tester
source ~/.virtualenvs/tool-size-tester/Scripts/activate
pip install -q $1
python tool-size.py
deactivate
Step 3
run script with package you want to get the size of
tool-size.sh xxx
say for truffleHog3
$ ./tool-size.sh truffleHog3
Size of Dependencies
----------------------------------------
truffleHog3 2.0.6: 56.46 KB
----------------------------------------
smmap 4.0.0: 108.808 KB
----------------------------------------
MarkupSafe 2.0.1: 40.911 KB
----------------------------------------
Jinja2 3.0.1: 917.551 KB
----------------------------------------
gitdb 4.0.7: 320.08 KB
----------------------------------------
Total Size (including dependencies): 1443.81 KB