split python source code into multiple files?
Question:
I have a code that I wish to split apart into multiple files. In matlab one can simply call a .m file, and as long as it is not defined as anything in particular it will just run as if it were part of the called code. Example (edited):
test.m (matlab)
function [] = test()
... some code using variables ...
test2
test2.m (matlab)
... some more code using same variables ...
Calling test runs the code in test as well as the code in test2.
Is there a similar way for python, to put ... some more code ..., into an external file, that will simply be read as if it is in the file that it is called from?
Answers:
Python has importing and namespacing, which are good. In Python you can import into the current namespace, like:
>>> from test import disp
>>> disp('World!')
Or with a namespace:
>>> import test
>>> test.disp('World!')
Sure!
#file -- test.py --
myvar = 42
def test_func():
print("Hello!")
Now, this file (“test.py”) is in python terminology a “module”. We can import it (as long as it can be found in our PYTHONPATH) Note that the current directory is always in PYTHONPATH, so if use_test is being run from the same directory where test.py lives, you’re all set:
#file -- use_test.py --
import test
test.test_func() #prints "Hello!"
print (test.myvar) #prints 42
from test import test_func #Only import the function directly into current namespace
test_func() #prints "Hello"
print (myvar) #Exception (NameError)
from test import *
test_func() #prints "Hello"
print(myvar) #prints 42
There’s a lot more you can do than just that through the use of special __init__.py files which allow you to treat multiple files as a single module), but this answers your question and I suppose we’ll leave the rest for another time.
You can do the same in python by simply importing the second file, code at the top level will run when imported. I’d suggest this is messy at best, and not a good programming practice. You would be better off organizing your code into modules
Example:
F1.py:
print "Hello, "
import f2
F2.py:
print "World!"
When run:
python ./f1.py
Hello,
World!
Edit to clarify: The part I was suggesting was “messy” is using the import statement only for the side effect of generating output, not the creation of separate source files.
I am researching module usage in python just now and thought I would answer the question Markus asks in the comments above (“How to import variables when they are embedded in modules?”) from two perspectives:
- variable/function, and
- class property/method.
Here is how I would rewrite the main program f1.py to demonstrate variable reuse for Markus:
import f2
myStorage = f2.useMyVars(0) # initialze class and properties
for i in range(0,10):
print "Hello, "
f2.print_world()
myStorage.setMyVar(i)
f2.inc_gMyVar()
print "Display class property myVar:", myStorage.getMyVar()
print "Display global variable gMyVar:", f2.get_gMyVar()
Here is how I would rewrite the reusable module f2.py:
# Module: f2.py
# Example 1: functions to store and retrieve global variables
gMyVar = 0
def print_world():
print "World!"
def get_gMyVar():
return gMyVar # no need for global statement
def inc_gMyVar():
global gMyVar
gMyVar += 1
# Example 2: class methods to store and retrieve properties
class useMyVars(object):
def __init__(self, myVar):
self.myVar = myVar
def getMyVar(self):
return self.myVar
def setMyVar(self, myVar):
self.myVar = myVar
def print_helloWorld(self):
print "Hello, World!"
When f1.py is executed here is what the output would look like:
%run "f1.py"
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Display class property myVar: 9
Display global variable gMyVar: 10
I think the point to Markus would be:
- To reuse a module’s code more than once, put your module’s code into
functions or classes,
- To reuse variables stored as properties in modules, initialize
properties within a class and add “getter” and “setter” methods so
variables do not have to be copied into the main program,
- To reuse variables stored in modules, initialize the variables and use
getter and setter functions. The setter functions would declare the
variables as global.
There is nothing incorrect about the answers added so far, but they will only take you so far.
You may find yourself in the same situation as I did recently if your project begins to scale up.
What is the limitation of modules?
Consider a project structure consisting of a "main script" and some library code which is organized into 1 or more modules.
What is the limitation of this? What happens if you want to have more than one binary?
Asside: Python does not have binaries as such. But you can have multiple scripts each with their own entry point. (A def main() function analagous to int main() in C, and similar main functions in other languages.) You may at some point find yourself wanting to write a project which consists of 1 or more libraries which are shared by multiple Python "binaries" (scripts). This is exactly analagous to the kind of project structure you might write in other languages including C, C++, Rust, Java, C#, and many others.
Back to the point: Using only the modules approach described above you are limited, and will be forced down a route of putting all library code in the same directory as a collection of things which are meant to be used like executables.
This is obviously not great.
A possible solution using Virtual Environments, a build backend like setuptools, and a pyproject.toml file:
In summary, do the following things:
- create a Virtual Environment with
python3 -m venv .venv
- activate it (enable the venv)
source .venv/bin/activate
Asside: I like to create a symlink to .venv/bin/activate so I can simply run . activate from the project root to activate my venvs.
- install any libraries you need with
pip3 install ...
- create a folder for your "binaries" called
bin, put your "executable" Python scripts here
- create a folder called
src for your library code
- for each library module create a directory under
src
- add an
__init__.py file to each library module
- add your module code for each library module
- create
pyproject.toml in the project root (see below)
- "build" your libraries, and "install" them
pip3 install -e . from the project root
What does all of this do?
What you are effectively doing here is moving module code out of one directory, where the python interpreter can find it, into another where it cannot. That doesn’t seem to make a lot of sense, except to say the project directory structure is now improved. You should have some minimal "executable" python scripts in a bin directory and everything else logically organized under a src directory.
In order for the Python interpreter to find it, what you are doing here is "installing" your library code into the active Virtual Environment using pip3. In order to install it, you first need to build it with setuptools. When you build it and install it you use the -e flag. This installs it using symlinks instead of the usual way, which means if you modify your code, you can use the modifications immediatly without having to run the install step again.
Hopefully that all makes sense, I only recently learned about this, so this is my best attempt to provide a straightforward explanation.
Example toml file
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "my_project_name"
version = "1.0.0"
description = "My Project Description"
[tools.setuptools.packages.find]
where = ["src"]
I have a code that I wish to split apart into multiple files. In matlab one can simply call a .m file, and as long as it is not defined as anything in particular it will just run as if it were part of the called code. Example (edited):
test.m (matlab)
function [] = test()
... some code using variables ...
test2
test2.m (matlab)
... some more code using same variables ...
Calling test runs the code in test as well as the code in test2.
Is there a similar way for python, to put ... some more code ..., into an external file, that will simply be read as if it is in the file that it is called from?
Python has importing and namespacing, which are good. In Python you can import into the current namespace, like:
>>> from test import disp
>>> disp('World!')
Or with a namespace:
>>> import test
>>> test.disp('World!')
Sure!
#file -- test.py --
myvar = 42
def test_func():
print("Hello!")
Now, this file (“test.py”) is in python terminology a “module”. We can import it (as long as it can be found in our PYTHONPATH) Note that the current directory is always in PYTHONPATH, so if use_test is being run from the same directory where test.py lives, you’re all set:
#file -- use_test.py --
import test
test.test_func() #prints "Hello!"
print (test.myvar) #prints 42
from test import test_func #Only import the function directly into current namespace
test_func() #prints "Hello"
print (myvar) #Exception (NameError)
from test import *
test_func() #prints "Hello"
print(myvar) #prints 42
There’s a lot more you can do than just that through the use of special __init__.py files which allow you to treat multiple files as a single module), but this answers your question and I suppose we’ll leave the rest for another time.
You can do the same in python by simply importing the second file, code at the top level will run when imported. I’d suggest this is messy at best, and not a good programming practice. You would be better off organizing your code into modules
Example:
F1.py:
print "Hello, "
import f2
F2.py:
print "World!"
When run:
python ./f1.py
Hello,
World!
Edit to clarify: The part I was suggesting was “messy” is using the import statement only for the side effect of generating output, not the creation of separate source files.
I am researching module usage in python just now and thought I would answer the question Markus asks in the comments above (“How to import variables when they are embedded in modules?”) from two perspectives:
- variable/function, and
- class property/method.
Here is how I would rewrite the main program f1.py to demonstrate variable reuse for Markus:
import f2
myStorage = f2.useMyVars(0) # initialze class and properties
for i in range(0,10):
print "Hello, "
f2.print_world()
myStorage.setMyVar(i)
f2.inc_gMyVar()
print "Display class property myVar:", myStorage.getMyVar()
print "Display global variable gMyVar:", f2.get_gMyVar()
Here is how I would rewrite the reusable module f2.py:
# Module: f2.py
# Example 1: functions to store and retrieve global variables
gMyVar = 0
def print_world():
print "World!"
def get_gMyVar():
return gMyVar # no need for global statement
def inc_gMyVar():
global gMyVar
gMyVar += 1
# Example 2: class methods to store and retrieve properties
class useMyVars(object):
def __init__(self, myVar):
self.myVar = myVar
def getMyVar(self):
return self.myVar
def setMyVar(self, myVar):
self.myVar = myVar
def print_helloWorld(self):
print "Hello, World!"
When f1.py is executed here is what the output would look like:
%run "f1.py"
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Hello,
World!
Display class property myVar: 9
Display global variable gMyVar: 10
I think the point to Markus would be:
- To reuse a module’s code more than once, put your module’s code into
functions or classes, - To reuse variables stored as properties in modules, initialize
properties within a class and add “getter” and “setter” methods so
variables do not have to be copied into the main program, - To reuse variables stored in modules, initialize the variables and use
getter and setter functions. The setter functions would declare the
variables as global.
There is nothing incorrect about the answers added so far, but they will only take you so far.
You may find yourself in the same situation as I did recently if your project begins to scale up.
What is the limitation of modules?
Consider a project structure consisting of a "main script" and some library code which is organized into 1 or more modules.
What is the limitation of this? What happens if you want to have more than one binary?
Asside: Python does not have binaries as such. But you can have multiple scripts each with their own entry point. (A def main() function analagous to int main() in C, and similar main functions in other languages.) You may at some point find yourself wanting to write a project which consists of 1 or more libraries which are shared by multiple Python "binaries" (scripts). This is exactly analagous to the kind of project structure you might write in other languages including C, C++, Rust, Java, C#, and many others.
Back to the point: Using only the modules approach described above you are limited, and will be forced down a route of putting all library code in the same directory as a collection of things which are meant to be used like executables.
This is obviously not great.
A possible solution using Virtual Environments, a build backend like setuptools, and a pyproject.toml file:
In summary, do the following things:
- create a Virtual Environment with
python3 -m venv .venv - activate it (enable the venv)
source .venv/bin/activate
Asside: I like to create a symlink to .venv/bin/activate so I can simply run . activate from the project root to activate my venvs.
- install any libraries you need with
pip3 install ... - create a folder for your "binaries" called
bin, put your "executable" Python scripts here - create a folder called
srcfor your library code - for each library module create a directory under
src - add an
__init__.pyfile to each library module - add your module code for each library module
- create
pyproject.tomlin the project root (see below) - "build" your libraries, and "install" them
pip3 install -e .from the project root
What does all of this do?
What you are effectively doing here is moving module code out of one directory, where the python interpreter can find it, into another where it cannot. That doesn’t seem to make a lot of sense, except to say the project directory structure is now improved. You should have some minimal "executable" python scripts in a bin directory and everything else logically organized under a src directory.
In order for the Python interpreter to find it, what you are doing here is "installing" your library code into the active Virtual Environment using pip3. In order to install it, you first need to build it with setuptools. When you build it and install it you use the -e flag. This installs it using symlinks instead of the usual way, which means if you modify your code, you can use the modifications immediatly without having to run the install step again.
Hopefully that all makes sense, I only recently learned about this, so this is my best attempt to provide a straightforward explanation.
Example toml file
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "my_project_name"
version = "1.0.0"
description = "My Project Description"
[tools.setuptools.packages.find]
where = ["src"]