Why does the presence of a token.py cause the built-in help() to break?
Question:
As a hobby/learning project, I’m writing a parser generator in Python. One of my code files is named “token.py” – which contains a couple of classes for turning plain strings into Token objects. I’ve just discovered that using the “help()” function from the console in Python raises an error for any module defined in a directory that contains a “token.py”.
Here’s a way to reproduce the error. Create a new directory with the following files:
/New Folder
main.py
token.py
Leave ‘token.py’ blank. In main.py, write a simple function – for example:
def test():
pass
Then, in your Python console, import ‘main’ and call ‘help(main.test)’ – here’s what you’ll get:
C:New Folder>python
Python 3.1.1 (r311:74483, Aug 17 2009, 17:02:12) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import main
>>> help(main.test)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:Python31libsite.py", line 428, in __call__
import pydoc
File "C:Python31libpydoc.py", line 55, in <module>
import sys, imp, os, re, inspect, builtins, pkgutil
File "C:Python31libinspect.py", line 40, in <module>
import tokenize
File "C:Python31libtokenize.py", line 37, in <module>
COMMENT = N_TOKENS
NameError: name 'N_TOKENS' is not defined
>>>
If you delete the ‘token.py’ file, help() will behave normally. Both Python 3.1 and Python 2.5 exhibit this behavior.
Is this a known issue? If not, how do I report it?
EDIT:
Several comments state that this behavior isn’t a bug. The module that defines “help” imports a module called “token” from Python’s standard library. However, Python looks in the application folder before it looks in its library to find modules. In the example above, “help” tries to use my “token.py” instead of Python’s, which causes the error.
Since Python is defined to exhibit this behavior, I suppose it isn’t a bug. But why do people think that this behavior is acceptable? It implies that adding new modules to Python’s library – even without changing existing modules – could break existing applications. It also implies that programmers are expected to have memorized the names of all modules in Python’s library – how is that any less ridiculous than expecting programmers to memorize every namespace in .NET or Java? Why don’t Python applications get their own namespaces? Why aren’t Python standard library modules in their own namespace?
Answers:
You can search for issues and report new ones here: http://bugs.python.org/.
The problem is that your local token.py is being imported by help() instead of Python’s actual token.py. This will occur for any number of .py files whose names collide with built-in modules. For example, try creating a pydoc.py file in the CWD and then try help() in Python. The help() function is just a built-in Python function, so it follows the same import path as any other Python code.
This isn’t a bug with the the help() method. There is a module in the standard library named token.py which is imported by tokenize.py (where the error you’re seeing originates from).
from tokenize.py:
from token import *
So tokenize.py expects to have a bunch of variables from the standard library token.py, but since the token.py in your working directory is actually imported they are not present which is causing the NameError (one amoungst many reasons import * shouldn’t be used).
Whenever you choose to name your modules in a way that mimics module names defined in Python’s standard library, you’re fully responsible for whatever happens as a result — other Python standard library modules are likely to rely on those that you are hiding/overriding, and if your own modules don’t carefully mimic the functionality of the modules you’re hiding, that’s not a bug with Python: it’s a bug with your code. Of course, this applies to module token as much as to any other.
If this is happening to you accidentally, and you’re looking for a way to check your code for likely bugs or iffy constructs (rather than, as you say, for a way to report a nonexisting bug with the Python standard library), I think tools like pylint may be able to help.
As a hobby/learning project, I’m writing a parser generator in Python. One of my code files is named “token.py” – which contains a couple of classes for turning plain strings into Token objects. I’ve just discovered that using the “help()” function from the console in Python raises an error for any module defined in a directory that contains a “token.py”.
Here’s a way to reproduce the error. Create a new directory with the following files:
/New Folder
main.py
token.py
Leave ‘token.py’ blank. In main.py, write a simple function – for example:
def test():
pass
Then, in your Python console, import ‘main’ and call ‘help(main.test)’ – here’s what you’ll get:
C:New Folder>python
Python 3.1.1 (r311:74483, Aug 17 2009, 17:02:12) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import main
>>> help(main.test)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:Python31libsite.py", line 428, in __call__
import pydoc
File "C:Python31libpydoc.py", line 55, in <module>
import sys, imp, os, re, inspect, builtins, pkgutil
File "C:Python31libinspect.py", line 40, in <module>
import tokenize
File "C:Python31libtokenize.py", line 37, in <module>
COMMENT = N_TOKENS
NameError: name 'N_TOKENS' is not defined
>>>
If you delete the ‘token.py’ file, help() will behave normally. Both Python 3.1 and Python 2.5 exhibit this behavior.
Is this a known issue? If not, how do I report it?
EDIT:
Several comments state that this behavior isn’t a bug. The module that defines “help” imports a module called “token” from Python’s standard library. However, Python looks in the application folder before it looks in its library to find modules. In the example above, “help” tries to use my “token.py” instead of Python’s, which causes the error.
Since Python is defined to exhibit this behavior, I suppose it isn’t a bug. But why do people think that this behavior is acceptable? It implies that adding new modules to Python’s library – even without changing existing modules – could break existing applications. It also implies that programmers are expected to have memorized the names of all modules in Python’s library – how is that any less ridiculous than expecting programmers to memorize every namespace in .NET or Java? Why don’t Python applications get their own namespaces? Why aren’t Python standard library modules in their own namespace?
You can search for issues and report new ones here: http://bugs.python.org/.
The problem is that your local token.py is being imported by help() instead of Python’s actual token.py. This will occur for any number of .py files whose names collide with built-in modules. For example, try creating a pydoc.py file in the CWD and then try help() in Python. The help() function is just a built-in Python function, so it follows the same import path as any other Python code.
This isn’t a bug with the the help() method. There is a module in the standard library named token.py which is imported by tokenize.py (where the error you’re seeing originates from).
from tokenize.py:
from token import *
So tokenize.py expects to have a bunch of variables from the standard library token.py, but since the token.py in your working directory is actually imported they are not present which is causing the NameError (one amoungst many reasons import * shouldn’t be used).
Whenever you choose to name your modules in a way that mimics module names defined in Python’s standard library, you’re fully responsible for whatever happens as a result — other Python standard library modules are likely to rely on those that you are hiding/overriding, and if your own modules don’t carefully mimic the functionality of the modules you’re hiding, that’s not a bug with Python: it’s a bug with your code. Of course, this applies to module token as much as to any other.
If this is happening to you accidentally, and you’re looking for a way to check your code for likely bugs or iffy constructs (rather than, as you say, for a way to report a nonexisting bug with the Python standard library), I think tools like pylint may be able to help.