MANIFEST.in ignored on "python setup.py install" – no data files installed?
Question:
Here’s my stripped-down setup.py script with non-code stuff removed:
#!/usr/bin/env python
from distutils.core import setup
from whyteboard.misc import meta
setup(
name = 'Whyteboard',
version = meta.version,
packages = ['whyteboard', 'whyteboard.gui', 'whyteboard.lib', 'whyteboard.lib.pubsub',
'whyteboard.lib.pubsub.core', 'whyteboard.lib.pubsub.utils', 'whyteboard.misc'],
py_modules = ['whyteboard'],
scripts = ['whyteboard.py'],
)
MANIFEST.in:
include *.txt
include whyteboard-help/*.*
recursive-include locale *.mo
recursive-include images *.png
When I run “python setup.py install sdist” I get a nice .tar.gz with a “whyteboard-0.41” root folder, with my locale/ images/ and whyteboard-help/ folders inside. This also has my whyteboard.py script that launches my program from inside the whyteboard source package.
So:
whyteboard/
├── locale/
├── images
├── whyteboard-help/
├── whyteboard/
│ ├── __init__.py
│ └── other packages etc
├── whyteboard.py
├── README
├── setup.py
└── CHANGELOG
This mirrors the source of my program, is how everything should be, and is correct.
However when I run “python setup.py install” none of my data files are written – only the “whyteboard” source package, and the whyteboard.py is placed in /usr/local/lib/python2.6/dist-packages/.
Ideally, I’d like the same directory structure as what’s been generated in the .tar.gz file to be created in dist-packages, as this is how my program expects to look for its resources.
How can I get “install” to create this directory structure? It seems to be ignoring my manifest file, as far as I can tell.
Answers:
MANIFEST.in tells Distutils what files to include in the source distribution but it does not directly affect what files are installed. For that you need to include the appropriate files in the setup.py file, generally either as package data or as additional files.
Some notes in addition to Ned’s answer (which hits on the core problem):
Distutils does not install Python packages and modules inside a per-project subdirectory within site-packages (or dist-packages on Debian/Ubuntu): they are installed directly into site-packages, as you’ve seen. So the containing whyteboard-xx directory in your sdist will not exist in the final installed form.
One implication of this is that you should be careful to name your data_files in a way that clarifies what project they belong to, because those files/directories are installed directly into the global site-packages directory, not inside any containing whyteboard directory.
Or you could instead make your data package_data of the whyteboard package (which means it needs to live inside that package, i.e. next to __init__.py), and then this isn’t a problem.
Lastly, it doesn’t make much sense to have both a whyteboard.py module in py_modules and a whyteboard/__init__.py package in packages. The two are mutually exclusive, and if you have both, the whyteboard.py module will be ignored by imports in favor of the package of the same name.
If whyteboard.py is just a script, and is not intended to be imported, then you should use the scripts option for it, and remove it from py_modules.
Running python 2.6.1 on Mac OSX, I had absolutely no luck except by using the data_files parameter in setup.py. Everything with MANIFEST.in simply resulted in files being included in the dist package, but never installed. I checked some other packages and they were indeed using data_files to specify additional files.
I created a short function to help enumerate all the files from a directory tree in the
(target_dir, [file list]) format that data_files expects:
def gen_data_files(*dirs):
results = []
for src_dir in dirs:
for root,dirs,files in os.walk(src_dir):
results.append((root, map(lambda f:root + "/" + f, files)))
return results
Now I can just call this inside my setup call:
setup(... data_files = gen_data_files("docs", "lib") ...
And everything in those trees gets installed.
You should use setuptools:
#!/usr/bin/env python
from setuptools import setup, find_packages
from whyteboard.misc import meta
setup(
name = 'Whyteboard',
version = meta.version,
packages = find_packages(),
include_package_data=True,
py_modules = ['whyteboard'],
scripts = ['whyteboard.py'],
)
This is not actually using the MANIFEST file to do the job, but it includes all the needed files.
I couldn’t figure out why my MANIFEST.in file was being ignored when I ran python setup.py install – turns out include_package_data=True solves the problem. The package_data option isn’t actually required.
Minimal published runnable example
- https://github.com/cirosantilli/python-sample-package-with-data
- https://pypi.org/project/python-sample-package-with-data
Key takeaway: only MANIFEST.in worked for me, package_data did not.
Tested on Ubuntu 19.10, Python 3.7.5, wheel==0.32.3, setuptools==41.1.0, twine==3.1.1.
How end users use the package from https://pypi.org/project/python-sample-package-with-data/:
python3 -m pip install --user python-sample-package-with-data
python-sample-package-with-data
Expected output:
hello data
How maintainers publish it:
# One time setup.
python3 -m pip install --user setuptools wheel twine
# Every time you want to publish.
python setup.py sdist bdist_wheel
twine upload dist/*
rm -rf build dist *.egg-info
The actual files:
MANIFEST.in
# Or else pip install cannot find README.md on the setup.py under certain conditions.
include README.md
# This actually adds the data file.
include python_sample_package_with_data/mydata.txt
python-sample-package-with-data
#!/usr/bin/env python3
import python_sample_package_with_data
print(python_sample_package_with_data.get_data(), end='')
python_sample_package_with_data/__init__.py
try:
import importlib.resources as importlib_resources
except ImportError:
# In PY<3.7 fall-back to backported `importlib_resources`.
import importlib_resources
def get_data():
return importlib_resources.read_text(__name__, 'mydata.txt')
python_sample_package_with_data/mydata.txt
hello data
setup.py
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from setuptools import setup, find_packages
from os import path
this_directory = path.abspath(path.dirname(__file__))
with open(path.join(this_directory, 'README.md')) as f:
long_description = f.read()
setup(
name='python-sample-package-with-data',
version='0.0.3',
description='My short description',
long_description=long_description,
long_description_content_type='text/markdown',
url='https://github.com/cirosantilli/python-sample-package-with-data',
author='Ciro Santilli',
author_email='[email protected]',
packages=find_packages(),
include_package_data=True,
scripts=['python-sample-package-with-data'],
)
Bibliography:
- How do I distribute my pip package with data files correctly?
- Python pkg_resources and file access in packages
- Finding a file in a Python module distribution
- Any python function to get "data_files" root directory?
- MANIFEST.in, package_data, and data_files clarification?
- setup.py not installing data files
Here’s my stripped-down setup.py script with non-code stuff removed:
#!/usr/bin/env python
from distutils.core import setup
from whyteboard.misc import meta
setup(
name = 'Whyteboard',
version = meta.version,
packages = ['whyteboard', 'whyteboard.gui', 'whyteboard.lib', 'whyteboard.lib.pubsub',
'whyteboard.lib.pubsub.core', 'whyteboard.lib.pubsub.utils', 'whyteboard.misc'],
py_modules = ['whyteboard'],
scripts = ['whyteboard.py'],
)
MANIFEST.in:
include *.txt
include whyteboard-help/*.*
recursive-include locale *.mo
recursive-include images *.png
When I run “python setup.py install sdist” I get a nice .tar.gz with a “whyteboard-0.41” root folder, with my locale/ images/ and whyteboard-help/ folders inside. This also has my whyteboard.py script that launches my program from inside the whyteboard source package.
So:
whyteboard/
├── locale/
├── images
├── whyteboard-help/
├── whyteboard/
│ ├── __init__.py
│ └── other packages etc
├── whyteboard.py
├── README
├── setup.py
└── CHANGELOG
This mirrors the source of my program, is how everything should be, and is correct.
However when I run “python setup.py install” none of my data files are written – only the “whyteboard” source package, and the whyteboard.py is placed in /usr/local/lib/python2.6/dist-packages/.
Ideally, I’d like the same directory structure as what’s been generated in the .tar.gz file to be created in dist-packages, as this is how my program expects to look for its resources.
How can I get “install” to create this directory structure? It seems to be ignoring my manifest file, as far as I can tell.
MANIFEST.in tells Distutils what files to include in the source distribution but it does not directly affect what files are installed. For that you need to include the appropriate files in the setup.py file, generally either as package data or as additional files.
Some notes in addition to Ned’s answer (which hits on the core problem):
Distutils does not install Python packages and modules inside a per-project subdirectory within site-packages (or dist-packages on Debian/Ubuntu): they are installed directly into site-packages, as you’ve seen. So the containing whyteboard-xx directory in your sdist will not exist in the final installed form.
One implication of this is that you should be careful to name your data_files in a way that clarifies what project they belong to, because those files/directories are installed directly into the global site-packages directory, not inside any containing whyteboard directory.
Or you could instead make your data package_data of the whyteboard package (which means it needs to live inside that package, i.e. next to __init__.py), and then this isn’t a problem.
Lastly, it doesn’t make much sense to have both a whyteboard.py module in py_modules and a whyteboard/__init__.py package in packages. The two are mutually exclusive, and if you have both, the whyteboard.py module will be ignored by imports in favor of the package of the same name.
If whyteboard.py is just a script, and is not intended to be imported, then you should use the scripts option for it, and remove it from py_modules.
Running python 2.6.1 on Mac OSX, I had absolutely no luck except by using the data_files parameter in setup.py. Everything with MANIFEST.in simply resulted in files being included in the dist package, but never installed. I checked some other packages and they were indeed using data_files to specify additional files.
I created a short function to help enumerate all the files from a directory tree in the
(target_dir, [file list]) format that data_files expects:
def gen_data_files(*dirs):
results = []
for src_dir in dirs:
for root,dirs,files in os.walk(src_dir):
results.append((root, map(lambda f:root + "/" + f, files)))
return results
Now I can just call this inside my setup call:
setup(... data_files = gen_data_files("docs", "lib") ...
And everything in those trees gets installed.
You should use setuptools:
#!/usr/bin/env python
from setuptools import setup, find_packages
from whyteboard.misc import meta
setup(
name = 'Whyteboard',
version = meta.version,
packages = find_packages(),
include_package_data=True,
py_modules = ['whyteboard'],
scripts = ['whyteboard.py'],
)
This is not actually using the MANIFEST file to do the job, but it includes all the needed files.
I couldn’t figure out why my MANIFEST.in file was being ignored when I ran python setup.py install – turns out include_package_data=True solves the problem. The package_data option isn’t actually required.
Minimal published runnable example
- https://github.com/cirosantilli/python-sample-package-with-data
- https://pypi.org/project/python-sample-package-with-data
Key takeaway: only MANIFEST.in worked for me, package_data did not.
Tested on Ubuntu 19.10, Python 3.7.5, wheel==0.32.3, setuptools==41.1.0, twine==3.1.1.
How end users use the package from https://pypi.org/project/python-sample-package-with-data/:
python3 -m pip install --user python-sample-package-with-data
python-sample-package-with-data
Expected output:
hello data
How maintainers publish it:
# One time setup.
python3 -m pip install --user setuptools wheel twine
# Every time you want to publish.
python setup.py sdist bdist_wheel
twine upload dist/*
rm -rf build dist *.egg-info
The actual files:
MANIFEST.in
# Or else pip install cannot find README.md on the setup.py under certain conditions.
include README.md
# This actually adds the data file.
include python_sample_package_with_data/mydata.txt
python-sample-package-with-data
#!/usr/bin/env python3
import python_sample_package_with_data
print(python_sample_package_with_data.get_data(), end='')
python_sample_package_with_data/__init__.py
try:
import importlib.resources as importlib_resources
except ImportError:
# In PY<3.7 fall-back to backported `importlib_resources`.
import importlib_resources
def get_data():
return importlib_resources.read_text(__name__, 'mydata.txt')
python_sample_package_with_data/mydata.txt
hello data
setup.py
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from setuptools import setup, find_packages
from os import path
this_directory = path.abspath(path.dirname(__file__))
with open(path.join(this_directory, 'README.md')) as f:
long_description = f.read()
setup(
name='python-sample-package-with-data',
version='0.0.3',
description='My short description',
long_description=long_description,
long_description_content_type='text/markdown',
url='https://github.com/cirosantilli/python-sample-package-with-data',
author='Ciro Santilli',
author_email='[email protected]',
packages=find_packages(),
include_package_data=True,
scripts=['python-sample-package-with-data'],
)
Bibliography:
- How do I distribute my pip package with data files correctly?
- Python pkg_resources and file access in packages
- Finding a file in a Python module distribution
- Any python function to get "data_files" root directory?
- MANIFEST.in, package_data, and data_files clarification?
- setup.py not installing data files