Make distutils place compiled extensions in the right folder

Question

I wrote a Python C extension that I’m building with distutils. This is setup.py:

from distutils.core import setup, Extension

utilsmodule = Extension('utils', sources = ['utilsmodule.c'])

setup (name = 'myPackage',
       version = '1.0',
       description = 'My package',
       ext_modules = [utilsmodule])

When I run python setup.y build, the extension is built correctly but the .pyd file goes into the folder build/lib.win-amd64-3.7, and not into the module’s root where Python looks for modules to import. I need to move that file out of build after building to be able to import it.

I thought about adding a line after setup() that moves the file, but that seems a bit dirty, I’m guessing that distutils should do that work.

What is the right way to compile an extension in a way that it can be imported by other Python files immediately after build?

Asked By: Hey

||

Source

Answer 1

distutils should do that work.

It shouldn’t. Building is just an intermediate phase to packaging, installing and using.

What is the right way to compile an extension in a way that it can be imported by other Python files immediately after build?

There is no such a way. You can set %PYTHONPATH% to point to build/lib.win-amd64-3.7 and import the module directly from build/.

Or you may try to bend distutils:

from distutils.core import setup
from distutils.command.build_ext import build_ext as _build_ext
import os

class build_ext(_build_ext):
    def run(self):
        _build_ext.run(self)
        os.rename("build/lib.win-amd64-3.7/%s" % mypydname, dest_dir)

setup(
    …
    cmdclass={'build_ext': build_ext},
    …
)

but any way you have to decide where to move the compiled module.

Answered By: phd

Answer 2

Just to update the info in 2022~

If you want to test your module inside the project you develop your package, I think changing the extension name and then using --inplace flag is enough.

For example, if this_package is your extension, and you want your package structure to be like:

project_dir
    |-- package
    |   |-- __init__.py
    |   |-- this_package.xxxx.so
    |   |-- other.py
    |-- setup.py

You can just set the name of this extension like:

Extension("package.this_package", source_files)

And then build it by:

python setup.py build_ext --inplace

If you want to distribute and publish the package, and you want the structure of the installed package to be also like the above one.

If you’re open to sharing your source files, you could use python setup.py sdist to create the .tar.gz distribution. If all the source files are gathered in the .tar.gz file (by configuring setup.py). The package would be correctly built on the target machine and then installed in the site-packages directory with such a structure.
If not, you might need to use python setup.py bdist_wheel or something to build a binary wheel for publishing. It would build the library locally and (I think) we are able to publish only the .so library. However, the .whl is OS+Arch dependent, so you might need to build multiple distributions for different OS+Arch.

Whatever way we use, by setting the name package.this_package for the extension, you could always have the .so at the proper place.

Answered By: milliele

Answer 3

I think changing the name of the extension to <path>.<to>.<package>.<extension_name> will place it in the directory that you want. For example:

from distutils.core import setup, Extension

utilsmodule = Extension('package.utils', sources = ['utilsmodule.c'])

setup (name = 'myPackage',
       version = '1.0',
       description = 'My package',
       ext_modules = [utilsmodule])

Will place the utils extension under the package package

Answered By: Matt

Make distutils place compiled extensions in the right folder

Question:

Answers: