pylint 1.4 reports E1101(no-member) on all C extensions
Question:
We’ve been long-time fans of pylint. Its static analysis has become a critical part of all our python projects and has saved tons of time chasing obscure bugs. But after upgrading from 1.3 -> 1.4, almost all compiled c extensions result in E1101(no-member) errors.
Projects that previously run perfectly clean through pylint 1.3 now complain about almost every C extension member with E1101. We’ve been forced to disable E1101 errors, but this materially detracts from the usefulness of pylint.
For example, this perfectly valid use of the lxml package
r"""valid.py: demonstrate pylint 1.4 error"""
from lxml import etree
print etree.Element('mydoc')
Run this through pylint, and it reports:
$ pylint -rn valid.py
No config file found, using default configuration
************* Module valid
E: 3, 6: Module 'lxml.etree' has no 'Element' member (no-member)
But it is perfectly valid:
$ python valid.py
<Element mydoc at 7fddf67b1ba8>
Here’s where it gets really weird. A very small handful of C extensions seem to work just fine through pylint, e.g.:
r"""valid2.py: this one works fine"""
import sqlite3
print sqlite3.version
$ pylint -rn valid2.py
No config file found, using default configuration
My question is, has anyone else witnessed this? And if so, would you be willing to share your workaround/solution?
We’ve experimented with trying to create plugins to suppress these warnings
(http://docs.pylint.org/plugins.html#enter-plugin), but we’re having difficulty making heads or tails of the docs — and the astroid base class is uber-complex and has defied our attempts to grok it.
For real bonus points (and our eternal gratitude) we’d love to understand what changed in pylint. We’d be happy to fix the code (or at least publish a best practice document for C extension authors) that would satisfy pylint.
Platform details
$ pylint --version
No config file found, using default configuration
pylint 1.4.0,
astroid 1.3.2, common 0.63.2
Python 2.7.5 (default, Jul 1 2013, 18:09:11)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
Answers:
Shortly after posting my question, I found the answer. The change was in fact done on purpose as a security measure. Pylint imports modules to effectively identify valid methods and attributes. It was decided that importing c extensions that are not part of the python stdlib is a security risk and could introduce malicious code.
This was done in the release of Astroid 1.3.1 https://mail.python.org/pipermail/code-quality/2014-November/000394.html
Only C extensions from trusted sources (the standard library) are
loaded into the examining Python process to build an AST from the live
module.
There are four solutions if you want to use pylint on projects that import non-stdlib c extensions.
1) Disable safety using the --unsafe-load-any-extension=y command line option. This feature is undocumented and classified as a hidden option (https://mail.python.org/pipermail/code-quality/2014-November/000439.html).
2) Disable safety using the pylint.rc setting unsafe-load-any-extensions=yes. This is recommended over option 1 and includes full documentation in the default pylint.rc file (created with --generate-rcfile).
3) Specifically list packages or modules names that you trust to be loaded by pylint in the pylint.rc file using the extension-pkg-whitelist= option.
4) Create a plugin to manipulate the AST (I have no idea how to effect this — but it’s regularly discussed on on the pylint mailing list).
We opted for Option 3. We added the following line to our project pylint.rc file:
extension-pkg-whitelist=lxml
@user590028, thanks a lot for your answer! I just ran into this same problem with the libraries win32api, win32evtlog, win32file, win32gui, and win32process, and your solution worked.
I used another method I think is worth posting here, which is to call pylint and pass the whitelisted packages as a parameter:
pylint --extension-pkg-whitelist=win32api,win32evtlog,win32file,win32gui,win32process myfile.py
For those of you using VS Code, it’s a bit tricky to find where to put the command as I couldn’t find my executable.
In VS Code;
- click on File > Preferences > Settings.
- Scroll down to “Python Configurations” in the left window
- scroll down to “Python Linting: Mypy Args” in the right window
- click on “Edit in settings.json” link
- edit the json to include:
“–extension-pkg-whitelist=”
I had to do all this because PyLint isn’t executable from my Windows command line…
If you’re using VS Code for Mac, this is what you need to do in order to edit the settings.json file:
- Click on Code (i.e. the Visual Studio Code tab which is on the left of the ‘File’ tab) -> Preferences – > Settings
- Scroll down to Extensions and click on Python in the list.
- Click on any of the
Edit in settings.json links. This opens up settings.json for editing.
- Add the line
"python.linting.pylintArgs": ["----extension-pkg-whitelist=1xml"].
We’ve been long-time fans of pylint. Its static analysis has become a critical part of all our python projects and has saved tons of time chasing obscure bugs. But after upgrading from 1.3 -> 1.4, almost all compiled c extensions result in E1101(no-member) errors.
Projects that previously run perfectly clean through pylint 1.3 now complain about almost every C extension member with E1101. We’ve been forced to disable E1101 errors, but this materially detracts from the usefulness of pylint.
For example, this perfectly valid use of the lxml package
r"""valid.py: demonstrate pylint 1.4 error"""
from lxml import etree
print etree.Element('mydoc')
Run this through pylint, and it reports:
$ pylint -rn valid.py
No config file found, using default configuration
************* Module valid
E: 3, 6: Module 'lxml.etree' has no 'Element' member (no-member)
But it is perfectly valid:
$ python valid.py
<Element mydoc at 7fddf67b1ba8>
Here’s where it gets really weird. A very small handful of C extensions seem to work just fine through pylint, e.g.:
r"""valid2.py: this one works fine"""
import sqlite3
print sqlite3.version
$ pylint -rn valid2.py
No config file found, using default configuration
My question is, has anyone else witnessed this? And if so, would you be willing to share your workaround/solution?
We’ve experimented with trying to create plugins to suppress these warnings
(http://docs.pylint.org/plugins.html#enter-plugin), but we’re having difficulty making heads or tails of the docs — and the astroid base class is uber-complex and has defied our attempts to grok it.
For real bonus points (and our eternal gratitude) we’d love to understand what changed in pylint. We’d be happy to fix the code (or at least publish a best practice document for C extension authors) that would satisfy pylint.
Platform details
$ pylint --version
No config file found, using default configuration
pylint 1.4.0,
astroid 1.3.2, common 0.63.2
Python 2.7.5 (default, Jul 1 2013, 18:09:11)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
Shortly after posting my question, I found the answer. The change was in fact done on purpose as a security measure. Pylint imports modules to effectively identify valid methods and attributes. It was decided that importing c extensions that are not part of the python stdlib is a security risk and could introduce malicious code.
This was done in the release of Astroid 1.3.1 https://mail.python.org/pipermail/code-quality/2014-November/000394.html
Only C extensions from trusted sources (the standard library) are
loaded into the examining Python process to build an AST from the live
module.
There are four solutions if you want to use pylint on projects that import non-stdlib c extensions.
1) Disable safety using the --unsafe-load-any-extension=y command line option. This feature is undocumented and classified as a hidden option (https://mail.python.org/pipermail/code-quality/2014-November/000439.html).
2) Disable safety using the pylint.rc setting unsafe-load-any-extensions=yes. This is recommended over option 1 and includes full documentation in the default pylint.rc file (created with --generate-rcfile).
3) Specifically list packages or modules names that you trust to be loaded by pylint in the pylint.rc file using the extension-pkg-whitelist= option.
4) Create a plugin to manipulate the AST (I have no idea how to effect this — but it’s regularly discussed on on the pylint mailing list).
We opted for Option 3. We added the following line to our project pylint.rc file:
extension-pkg-whitelist=lxml
@user590028, thanks a lot for your answer! I just ran into this same problem with the libraries win32api, win32evtlog, win32file, win32gui, and win32process, and your solution worked.
I used another method I think is worth posting here, which is to call pylint and pass the whitelisted packages as a parameter:
pylint --extension-pkg-whitelist=win32api,win32evtlog,win32file,win32gui,win32process myfile.py
For those of you using VS Code, it’s a bit tricky to find where to put the command as I couldn’t find my executable.
In VS Code;
- click on File > Preferences > Settings.
- Scroll down to “Python Configurations” in the left window
- scroll down to “Python Linting: Mypy Args” in the right window
- click on “Edit in settings.json” link
- edit the json to include:
“–extension-pkg-whitelist=”
I had to do all this because PyLint isn’t executable from my Windows command line…
If you’re using VS Code for Mac, this is what you need to do in order to edit the settings.json file:
- Click on Code (i.e. the Visual Studio Code tab which is on the left of the ‘File’ tab) -> Preferences – > Settings
- Scroll down to Extensions and click on Python in the list.
- Click on any of the
Edit in settings.jsonlinks. This opens upsettings.jsonfor editing. - Add the line
"python.linting.pylintArgs": ["----extension-pkg-whitelist=1xml"].