How to include sql files from a folder to setuptools while packaging as python egg

Question

I have a directory structure as below.

.
├── package
    ├── app
    │   ├── __init__.py
    │   ├── file1.py
    │   └── file2.py
    ├── master_sql_folder
    │   │
    │   │──sql_folder_1
    │   │      ├── world.sql
    │   │      ├── earth.sql
    │   │
    │   └──sql_folder_2
    │          ├── planet.sql
    │          ├── sun.sql
    │         
    └── wrapper_scripts
    │    ├── wrapper.py
    │    └── __init__.py
    ├── setup.py

I am trying to include all the sql files that exist in sub folders of master_sql_folder ( world.sql, earth.sql, planet.sql, sun.sql ) to setuptools while packaging as egg,I cannot use sql_folder_1 and sql_folder_2 in the path as new folders can be added in future under master_sql_folder and I need the code to read them too.I have tried adding the following lines to my setup.py but its not including the sql files in the build.

package_data={'master_sql_folder':['*']}
packages=['app', 'wrapper_scripts']

I appreciate your help in advance.

Asked By: Somu Sinhhaa

||

Source

Answer 1

Based on @S3DEV ‘s suggestion and accepted answer here How to add package data recursively in Python setup.py?. I was able to get a solution.

#sql_file_picker.py — script to feed files to setup.py

import glob
class FilePicker:
    def __init__(self):
        pass
    def sql_file_picker(self):
        sql_files = []
        directories = glob.glob('master_sql_folder\**\')
        for directory in directories:
            files = glob.glob(directory + '*.sql')
            if len(files) != 0:
                sql_files.append((directory, files))
        return sql_files

in setup.py

from wrapper_scripts.sql_file_picker import FilePicker
from setuptools import setup
setup(
    name='XXXXXXXXX',
    version='X.X.X',
    packages=['app', 'wrapper_scripts'],
    url='',
    license='',
    author='',
    author_email='',
    description='XXXXXXXXXXXXX',
    data_files=FilePicker().sql_file_picker()
)

Answered By: Somu Sinhhaa

Answer 2

I tried a few iterations of this without success.

In the end what worked was the official solution: https://python-packaging.readthedocs.io/en/latest/non-code-files.html

Often packages will need to depend on files which are not .py files: e.g. images, data tables, documentation, etc. Those files need special treatment in order for setuptools to handle them correctly.

The mechanism that provides this is the MANIFEST.in file. This is relatively quite simple: MANIFEST.in is really just a list of relative file paths specifying files or globs to include.:

include README.rst
include docs/*.txt
include funniest/data.json

In order for these files to be copied at install time to the package’s folder inside site-packages, you’ll need to supply include_package_data=True to the setup() function.

super easy to do, literally just include MANIFEST.in in the same dir as your setup.py, and as per the documentation, you’ll need to supply include_package_data=True to the setup() function.

Answered By: David Mendes

How to include sql files from a folder to setuptools while packaging as python egg

Question:

Answers:

in setup.py