Failed to import python module from different directory

Question

I have this code structure in python3:

- datalake
  __init__.py
  utils
     __init__.py
     utils.py
  lambdas
     __init__.py
     my-lambdas.py
- tests
    __init__.py
    demo.py

All init__.py files are empty.

My problem is how I can import datalake module from tests/demo.py?
I tried from datalake.utils import utils in demo.py but when I run python tests/demo.py from command line, I get this error ModuleNotFoundError: No module named 'datalake'.

If I use this code:

from ..datalake.utils import utils

I will get error ValueError: attempted relative import beyond top-level package.

I also tried to import the module utils from my-lambda.py file which also failed. The code in my-lambda.py is from datalake.utils import utils but I get ModuleNotFoundError: No module named 'datalake' error when run python datalake/lambda/my-lambda.py from command line.

How can I import the module?

Asked By: Joey Yi Zhao

||

Source

Answer 1

you can try run :

python -m datalake.lambda.my-lambda

follow: https://docs.python.org/3.7/using/cmdline.html#cmdoption-m

Answered By: Peep Peep Peep

Answer 2

You can add the module directory into your sys.path:

import sys
sys.path.append("your/own/modules/folder")  # like sys.path.append("../tests")

but this is a one-shot method, which is just valid at this time, the added path is not permanent, it will be eliminated after the code completed execution.

Answered By: Hu Xixi

Answer 3

First of all, my-lambdas.py is not importable with the import statement as hyphens are not valid in Python identifiers. Try to follow PEP-8’s naming conventions, such as mylambdas.py.

Otherwise the package structure looks good, and it should be importable as long as you are at the level above datalake/, e.g., if you were in the directory myproject/ below:

myproject
├── datalake
│   ├── __init__.py
│   ├── utils
│   │   ├── __init__.py
│   │   └── utils.py
│   └── lambdas
│       ├── __init__.py
│       └── mylambdas.py
└── tests
    ├── __init__.py
    └── demo.py

Then this should work:

~/myproject$ python -c 'from datalake import utils'

Otherwise, setting the environment variable PYTHONPATH to the path above datalake/ or modifying sys.path are both ways of changing where Python can import from. See the official tutorial on modules for more information.

Also some general advice: I’ve found it useful to stick with simple modules rather than packages (directories) until there is a need to expand. Then you can change foo.py into a foo/ directory with an __init__.py file and import foo will work as before, although you may need to add some imports to the __init__.py to maintain API compatibility. This would leave you with a simpler structure:

myproject
├── datalake
│   ├── __init__.py
│   ├── utils.py
│   └── lambdas.py
└── tests
    ├── __init__.py
    └── demo.py

Answered By: goodmami

Answer 4

When you run a command like python tests/demo.py, the folder you are in does not get added to the PYTHONPATH, the script folder does. So a top-level import like import datalake will fail. To get around this you can run your tests as a module:

Python 2:

python -m tests/demo

Python 3:

python -m tests.demo

and any datalake imports in demo.py will work.

It sounds like what you really want to do is have a folder with tests separate to your main application and run them. For this I recommend py.test, for your case you can read Tests Outside Application Code for how to do it. TL;DR is run your tests from your top level project folder with python -m py.test and it will work.

Answered By: bananafish

Answer 5

One of the ways to import the file directly instead of using from, like import util

Answered By: whiteheart

Failed to import python module from different directory

Question:

Answers: