Python packaging with setup.py does ignore manifest specifications

Question:

I’m currently trying to pack a module that uses precompiled *.pyd files from a swig routine.

The process for the user is supposed to be:

  • install base library (C, C++); directories linked in the environment variables; here are also the *.pyd files.
  • get python package; open dir from python environment (be it conda or else) and run "pip install ."
  • enjoy module….

What I did:

  • I generated the manifest file to include the *.pyd files from setup.py based on the environmental path from the installation.
  • I checked and upon installation of the module, the files are listed in the sources text file of the egg info.

But there are no *.pyd files in the module.

This is the way I assumed was correct, but I also tried many other ways (data option in setup specification, etc.), but nothing worked so far.

What did I do wrong?
On research, it seems like C related files are deleted after installation, but I thought the manifest defined files are save.

Edit: setup.py added.

from pathlib import Path
from setuptools import setup, find_packages, Extension
import os
import shutil

name = 'mypackage'

#define requirements
REQUIREMENTS = {
    # Installation script (this file) dependencies
    'setup': [
        'setuptools_scm',
    ],
    # Installation dependencies
    # Use with pip install . to install from source
    'install': [
        'Cpython',
        'setuptools < 64',
        'numpy >= 1.23',
        'matplotlib',
        'DateTime',
        'psutil',
        'xarray',
        'PyYAML',
        'scipy',
        'PySimpleGui'
    ],
}

#check for installed C library
lib_dir = ""
if 'BASEPACKAGE_IN_C' in os.environ:
    lib_dir = os.getenv('BASEPACKAGE_IN_C')
    print('BASEPACKAGE_IN_C found at {}!'.format(lib_dir))
else:
    Exception('BASEPACKAGE_IN_C does not seem to exist on this machine! Make sure that the environment variable BASEPACKAGE_IN_C is set.')

# define function to make manifest file
def __createManifest__(subdirs):
    """inventory all files in path and create a manifest file"""
    current = os.path.dirname(__file__)
    relative_paths = [os.path.relpath(path, current) for path in subdirs]
    with open(os.path.join(current, "MANIFEST.in"), "w") as manifest:
        manifest.writelines("recursive-include {} *".format(" ".join(relative_paths)))

# check for interface layer directory 
add_il = Path(lib_dir).parents[0].joinpath("sdk", "my_package_pyd_dir")
il_dest = os.path.join(os.path.dirname(__file__), "pyil" + os.sep)
if not os.path.exists(il_dest):
    os.makedirs(il_dest)
if os.path.exists(add_il):
    print('Python SDK interface layer found at {}!'.format(add_il))
    for root, dirs, files in os.walk(add_il):
        for file in files:
            #copy files locally
            shutil.copy(os.path.join(root, file), il_dest)
else:
    Exception('Python SDK interface layer does not seem to exist on this machine! Make sure that the BASEPACKAGE_IN_C SDK is '
              'properly installed.')

# make manifest file
__createManifest__([il_dest])

#standard setup call
setup(
    name=name,
    python_requires='>= 3.9',
    version='0.1',
    packages=find_packages(),
    url='',
    license='',
    author='Ben',
    author_email='',
    description='BASEPACKAGE_IN_C Python SDK. Linked to the BASEPACKAGE_IN_C installation at {}.'.format(lib_dir),
    setup_requires=REQUIREMENTS['setup'],
    install_requires=REQUIREMENTS['install'],
    include_package_data=True,
)

The setup.py generates the manifest.in with the following line:recursive-include pyil *

I also tried include pyil * or specifying the ending recursive-include pyil *.pyd or combinations thereof.

The Sources.txt file looks like that:

MANIFEST.in
setup.py
mypackage/moduleClass1.py
mypackage/moduleClass2.py
mypackage/moduleClass3.py
mypackage/__init__.py
mypackage.egg-info/PKG-INFO
mypackage.egg-info/SOURCES.txt
mypackage.egg-info/dependency_links.txt
mypackage.egg-info/requires.txt
mypackage.egg-info/top_level.txt
pyil/_pyil.pyd

So it is working so far. I tried with different filetypes in pyil, and they all worked beside the pyd file.

Asked By: Ben Müller

||

Answers:

I made it too complicated. The dir management copying the *.pyd files into a separate directory inside source (src) did not work.

Putting it directly into src/mypackage worked like a charm. The code for setup.py is:

from pathlib import Path
from setuptools import setup, find_packages, Extension
import os
import shutil

name = 'mypackage'

REQUIREMENTS = {
    # Installation script (this file) dependencies
    'setup': [
        'setuptools_scm',
    ],
    # Installation dependencies
    # Use with pip install . to install from source
    'install': [
        'Cpython',
        'setuptools < 64',
        'numpy >= 1.23',
        'matplotlib',
        'DateTime',
        'psutil',
        'xarray',
        'PyYAML',
        'scipy',
        'PySimpleGui'
    ],
}

lib_dir = ""
if 'BASEPACKAGE_IN_C' in os.environ:
    lib_dir = os.getenv('BASEPACKAGE_IN_C')
    print('BASEPACKAGE_IN_C SDK found at {}!'.format(lib_dir))
else:
    Exception('BASEPACKAGE_IN_C SDK does not seem to exist on this machine! Make sure that the environment variable BASEPACKAGE_IN_C is set.')


def __createManifest__(subdirs):
    """inventory all files in path and create a manifest file"""
    current = os.path.dirname(__file__)
    relative_paths = [os.path.relpath(path, current) for path in subdirs]
    with open(os.path.join(current, "MANIFEST.in"), "w") as manifest:
        manifest.writelines("recursive-include {} *.pyd".format(" ".join(relative_paths)))


add_il = os.path.join(os.path.dirname(__file__), "mypackage")

__createManifest__([add_il])

setup(
    name=name,
    python_requires='>= 3.9',
    version='0.1',
    packages=find_packages(),
    url='',
    license='',
    author='Ben',
    author_email='',
    description='BASEPACKAGE_IN_C Python SDK. Linked to the BASEPACKAGE_IN_C installation at {}.'.format(lib_dir),
    setup_requires=REQUIREMENTS['setup'],
    install_requires=REQUIREMENTS['install'],
    include_package_data=True,
)
Answered By: Ben Müller
Categories: questions Tags: , , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.