How to get a list of built-in modules in python?
Question:
I would like to get a list of names of built-in modules in python such that I can test the popularity of function’s naming conventions (underline, CamelCase or mixedCase).
I know there is a Global Module Index but I am wondering if there is a list of strings, which is easier to use 🙂
Update:
len(dir(__builtins__)) = 145
len(stdlib_list("2.7")) = 430
help('modules') = 508 # counting manually the output
Answers:
How about this? Though, this gets a list of built-in functions and variables rather than modules…
dir(__builtins__)
help('modules') will give you a list of all modules, according to How can I get a list of locally installed Python modules?. Not a list of strings, though.
The compiled-in module names are in sys.builtin_module_names. For all importable modules, see pkgutil.iter_modules.
Run these in a clean virtualenv to get (almost) only the modules that come with Python itself.
Note that a “popularity poll” will necessarily include modules that use old, discouraged naming conventions because they were written before today’s guidelines were put in place, and can’t change because need to be backwards compatible. It might be useful for something, but not for answering best-practice questions such as “How should I name my functions?”. For that, see the PEP8, the Python style guide, especially the “Naming Conventions” section.
>>>dir (__builtins__)
or
>>>help (__builtins__)
Now there is a 3rd party package for this. It scrapes the TOC of the Standard Library page in the official Python docs and builds a list.
You can install it using pip
pip install stdlib_list
and got get a list of libraries
In [12]: from stdlib_list import stdlib_list
In [13]: libraries = stdlib_list("3.5")
In [14]: libraries[4:12]
Out[14]: ['abc', 'aifc', 'argparse', 'array', 'ast', 'asynchat', 'asyncio', 'asyncore']
You can find source code here.
From the CPython`s docs:
All known built-in modules are listed in sys.builtin_module_names
Names of modules in sys.builtin_module_names is actual only for used a Python interpreter:
A tuple of strings giving the names of all modules that are compiled into this Python interpreter
Each built-in module use the special loader while importing: BuiltinImporter
In [65]: import itertools, sys, gc
In [66]: itertools.__loader__, sys.__loader__, gc.__loader__
Out[66]:
(_frozen_importlib.BuiltinImporter,
_frozen_importlib.BuiltinImporter,
_frozen_importlib.BuiltinImporter)
In the Python 3 the number of built-in modules has slightly increased
$ python2.7 -c "import sys; print('Count built-in modules: %d' %len(sys.builtin_module_names)); print(sys.builtin_module_names)"
Count built-in modules: 51
('__builtin__', '__main__', '_ast', '_bisect', '_codecs', '_collections', '_functools', '_heapq', '_io', '_locale', '_md5', '_random', '_sha', '_sha256', '_sha512', '_socket', '_sre', '_struct', '_symtable', '_warnings', '_weakref', 'array', 'binascii', 'cPickle', 'cStringIO', 'cmath', 'datetime', 'errno', 'exceptions', 'fcntl', 'gc', 'grp', 'imp', 'itertools', 'marshal', 'math', 'operator', 'posix', 'pwd', 'select', 'signal', 'spwd', 'strop', 'sys', 'syslog', 'thread', 'time', 'unicodedata', 'xxsubtype', 'zipimport', 'zlib')
$ python3.4 -c "import sys; print('Count built-in modules: %d' %len(sys.builtin_module_names)); print(sys.builtin_module_names)"
Count built-in modules: 54
('_ast', '_bisect', '_codecs', '_collections', '_datetime', '_elementtree', '_functools', '_heapq', '_imp', '_io', '_locale', '_md5', '_operator', '_pickle', '_posixsubprocess', '_random', '_sha1', '_sha256', '_sha512', '_socket', '_sre', '_stat', '_string', '_struct', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', 'array', 'atexit', 'binascii', 'builtins', 'errno', 'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', 'marshal', 'math', 'posix', 'pwd', 'pyexpat', 'select', 'signal', 'spwd', 'sys', 'syslog', 'time', 'unicodedata', 'xxsubtype', 'zipimport', 'zlib')
As the CPython is implemented (primary) on the C programming language, so it is not easy to find it, as example location the Python`s module sys (based on this answer):
$ locate sysmodule | grep python
/usr/include/python2.7/sysmodule.h
/usr/include/python3.4m/sysmodule.h
/usr/local/include/python3.5m/sysmodule.h
More information about getting an information about all available modules is the CPython, look in my answer here.
It can be done using the given block of code below and it is the most effective way as per me.
import sys
a = sys.builtin_module_names
print(a)
The last line to be included if you want to print them.
Here, a is a tuple and so it can access all the functionalities of a tuple.
You can have a look at sys.builtin_module_names for further help
https://docs.python.org/3/library/sys.html
I was working on a similar problem when I learned that ‘builtin’ means something like "there is no source file associated with this object".
Here’s a solution based on checking the /lib and /Dlls folders manually.
The use of "unique" may be redundant; it’s there because I’m not sure if DLLs is strictly for packages which come with python/it was useful for a different problem I was trying to solve (finding out the requirements for a source package).
from typing import Generator, Iterable
import os, re, sys
def unique(iterable:Iterable, yielded_list=None) -> Generator:
"""
Iterate over unique elements of an iterable
examples:
>>> [*unique('abbcccdddd')]
['a', 'b', 'c', 'd']
>>> [*unique('abcd')]
['a', 'b', 'c', 'd']
"""
yielded = yielded if not isinstance(yielded, type(None)) else []
for i in iterable:
if not i in yielded:
yield i
yielded.append(i)
def native_modules() -> Generator:
"""
Every module found:
under your installation's /lib and /DLLs directories; excuding any under /lib/site-packages/*
in sys.builtin_module_names
"""
omitables = 'site-packages __pycache__ .pyo .ico .dll .pdb'.split()
path = lambda folder: os.path.join(sys.exec_prefix, folder)
tidy = lambda pth: os.path.split(os.path.splitext(pth)[0])[-1] # separate the name from the path and extension, if present
discriminant = lambda pth: not any(re.match(i, pth) for i in map(re.escape, omitables))
lib = map(tidy, filter(discriminant, os.listdir(path('lib'))))
dlls = map(tidy, filter(discriminant, os.listdir(path('DLLs'))))
yielded = []
yield from yielded
yield from unique(sys.builtin_module_names, yielded)
yield from unique(lib, yielded)
yield from unique(dlls, yielded)
All the commands like dir(builtins) or help(builtins) or sys.builtin_module_names, are either missing several core inbuilt package names or giving thousands of lines of verbose output.
>>> help(‘modules’)
The best command to show all the installed modules is help(‘modules’). It gives you name of all the installed modules in python without missing anything.
Incase you need names of inbuilt modules only, create a temporary fresh venev and give the command there. It might be bit slow but is the only way by far in my experience to list every single package.
In Linux, you can get a dump of all the built-in modules that came with the fresh install with the following commands
sudo apt-get install python3-virtualenv
tmpDir=`mktemp -d`
python3 -m virtualenv $tmpDir
source /tmp/virtualenv/bin/activate
python
help('modules')
Example Execution
Here’s an example execution of the above commands in Debian 11 with python 3.9
user@disp643:~$ sudo apt-get install python3-virtualenv
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3-virtualenv is already the newest version (20.4.0+ds-2+deb11u1).
The following packages were automatically installed and are no longer required:
ethtool libbotan-2-17 libtspi1 linux-image-5.10.0-10-amd64
linux-image-5.10.0-13-amd64 linux-image-5.10.0-14-amd64
linux-image-5.10.0-15-amd64 linux-image-5.10.0-16-amd64
linux-image-5.10.0-17-amd64 net-tools sse3-support
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
user@disp643:~$
user@disp643:~$ tmpDir=`mktemp -d`
user@disp643:~$
user@disp643:~$ python3 -m virtualenv $tmpDir
created virtual environment CPython3.9.2.final.0-64 in 120ms
creator CPython3Posix(dest=/tmp/tmp.qQsKZHGZqk, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/user/.local/share/virtualenv)
added seed packages: pip==20.3.4, pkg_resources==0.0.0, setuptools==44.1.1, wheel==0.34.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
user@disp643:~$ source /tmp/virtualenv/bin/activate
(virtualenv) user@disp643:~$ python
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help('modules')
Please wait a moment while I gather a list of all available modules...
__future__ _tracemalloc graphlib retrying
_abc _uuid grp rlcompleter
_aix_support _virtualenv gzip runpy
_ast _warnings hashlib sched
_asyncio _weakref heapq secrets
_bisect _weakrefset hmac select
_blake2 _xxsubinterpreters html selectors
_bootlocale _xxtestfuzz html5lib setuptools
_bootsubprocess _zoneinfo http shelve
_bz2 abc idna shlex
_codecs aifc imaplib shutil
_codecs_cn antigravity imghdr signal
_codecs_hk appdirs imp site
_codecs_iso2022 argparse importlib sitecustomize
_codecs_jp array inspect smtpd
_codecs_kr ast io smtplib
_codecs_tw asynchat ipaddr sndhdr
_collections asyncio ipaddress socket
_collections_abc asyncore itertools socketserver
_compat_pickle atexit json spwd
_compression audioop keyword sqlite3
_contextvars base64 lib2to3 sre_compile
_crypt bdb linecache sre_constants
_csv binascii locale sre_parse
_ctypes binhex logging ssl
_ctypes_test bisect lzma stat
_curses builtins mailbox statistics
_curses_panel bz2 mailcap string
_datetime cProfile marshal stringprep
_dbm calendar math struct
_decimal certifi mimetypes subprocess
_elementtree cgi mmap sunau
_functools cgitb modulefinder symbol
_hashlib chunk msgpack symtable
_heapq cmath multiprocessing sys
_imp cmd netrc sysconfig
_io code nis syslog
_json codecs nntplib tabnanny
_locale codeop ntpath tarfile
_lsprof collections nturl2path telnetlib
_lzma colorama numbers tempfile
_markupbase colorsys opcode termios
_md5 compileall operator test
_multibytecodec concurrent optparse textwrap
_multiprocessing configparser os this
_opcode contextlib ossaudiodev threading
_operator contextlib2 packaging time
_osx_support contextvars parser timeit
_peg_parser copy pathlib tkinter
_pickle copyreg pdb token
_posixshmem crypt pep517 tokenize
_posixsubprocess csv pickle toml
_py_abc ctypes pickletools trace
_pydecimal curses pip traceback
_pyio dataclasses pipes tracemalloc
_queue datetime pkg_resources tty
_random dbm pkgutil turtle
_sha1 decimal platform types
_sha256 difflib plistlib typing
_sha3 dis poplib unicodedata
_sha512 distlib posix unittest
_signal distutils posixpath urllib
_sitebuiltins doctest pprint urllib3
_socket easy_install profile uu
_sqlite3 email progress uuid
_sre encodings pstats venv
_ssl enum pty warnings
_stat errno pwd wave
_statistics faulthandler py_compile weakref
_string fcntl pyclbr webbrowser
_strptime filecmp pydoc wheel
_struct fileinput pydoc_data wsgiref
_symtable fnmatch pyexpat xdrlib
_sysconfigdata__linux_x86_64-linux-gnu formatter pyparsing xml
_sysconfigdata__x86_64-linux-gnu fractions queue xmlrpc
_testbuffer ftplib quopri xxlimited
_testcapi functools random xxsubtype
_testimportmultiple gc re zipapp
_testinternalcapi genericpath readline zipfile
_testmultiphase getopt reprlib zipimport
_thread getpass requests zlib
_threading_local gettext resolvelib zoneinfo
_tkinter glob resource
Enter any module name to get more help. Or, type "modules spam" to search
for modules whose name or summary contain the string "spam".
>>>
I would like to get a list of names of built-in modules in python such that I can test the popularity of function’s naming conventions (underline, CamelCase or mixedCase).
I know there is a Global Module Index but I am wondering if there is a list of strings, which is easier to use 🙂
Update:
len(dir(__builtins__)) = 145
len(stdlib_list("2.7")) = 430
help('modules') = 508 # counting manually the output
How about this? Though, this gets a list of built-in functions and variables rather than modules…
dir(__builtins__)
help('modules') will give you a list of all modules, according to How can I get a list of locally installed Python modules?. Not a list of strings, though.
The compiled-in module names are in sys.builtin_module_names. For all importable modules, see pkgutil.iter_modules.
Run these in a clean virtualenv to get (almost) only the modules that come with Python itself.
Note that a “popularity poll” will necessarily include modules that use old, discouraged naming conventions because they were written before today’s guidelines were put in place, and can’t change because need to be backwards compatible. It might be useful for something, but not for answering best-practice questions such as “How should I name my functions?”. For that, see the PEP8, the Python style guide, especially the “Naming Conventions” section.
>>>dir (__builtins__)
or
>>>help (__builtins__)
Now there is a 3rd party package for this. It scrapes the TOC of the Standard Library page in the official Python docs and builds a list.
You can install it using pip
pip install stdlib_list
and got get a list of libraries
In [12]: from stdlib_list import stdlib_list
In [13]: libraries = stdlib_list("3.5")
In [14]: libraries[4:12]
Out[14]: ['abc', 'aifc', 'argparse', 'array', 'ast', 'asynchat', 'asyncio', 'asyncore']
You can find source code here.
From the CPython`s docs:
All known built-in modules are listed in sys.builtin_module_names
Names of modules in sys.builtin_module_names is actual only for used a Python interpreter:
A tuple of strings giving the names of all modules that are compiled into this Python interpreter
Each built-in module use the special loader while importing: BuiltinImporter
In [65]: import itertools, sys, gc
In [66]: itertools.__loader__, sys.__loader__, gc.__loader__
Out[66]:
(_frozen_importlib.BuiltinImporter,
_frozen_importlib.BuiltinImporter,
_frozen_importlib.BuiltinImporter)
In the Python 3 the number of built-in modules has slightly increased
$ python2.7 -c "import sys; print('Count built-in modules: %d' %len(sys.builtin_module_names)); print(sys.builtin_module_names)"
Count built-in modules: 51
('__builtin__', '__main__', '_ast', '_bisect', '_codecs', '_collections', '_functools', '_heapq', '_io', '_locale', '_md5', '_random', '_sha', '_sha256', '_sha512', '_socket', '_sre', '_struct', '_symtable', '_warnings', '_weakref', 'array', 'binascii', 'cPickle', 'cStringIO', 'cmath', 'datetime', 'errno', 'exceptions', 'fcntl', 'gc', 'grp', 'imp', 'itertools', 'marshal', 'math', 'operator', 'posix', 'pwd', 'select', 'signal', 'spwd', 'strop', 'sys', 'syslog', 'thread', 'time', 'unicodedata', 'xxsubtype', 'zipimport', 'zlib')
$ python3.4 -c "import sys; print('Count built-in modules: %d' %len(sys.builtin_module_names)); print(sys.builtin_module_names)"
Count built-in modules: 54
('_ast', '_bisect', '_codecs', '_collections', '_datetime', '_elementtree', '_functools', '_heapq', '_imp', '_io', '_locale', '_md5', '_operator', '_pickle', '_posixsubprocess', '_random', '_sha1', '_sha256', '_sha512', '_socket', '_sre', '_stat', '_string', '_struct', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', 'array', 'atexit', 'binascii', 'builtins', 'errno', 'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', 'marshal', 'math', 'posix', 'pwd', 'pyexpat', 'select', 'signal', 'spwd', 'sys', 'syslog', 'time', 'unicodedata', 'xxsubtype', 'zipimport', 'zlib')
As the CPython is implemented (primary) on the C programming language, so it is not easy to find it, as example location the Python`s module sys (based on this answer):
$ locate sysmodule | grep python
/usr/include/python2.7/sysmodule.h
/usr/include/python3.4m/sysmodule.h
/usr/local/include/python3.5m/sysmodule.h
More information about getting an information about all available modules is the CPython, look in my answer here.
It can be done using the given block of code below and it is the most effective way as per me.
import sys
a = sys.builtin_module_names
print(a)
The last line to be included if you want to print them.
Here, a is a tuple and so it can access all the functionalities of a tuple.
You can have a look at sys.builtin_module_names for further help
https://docs.python.org/3/library/sys.html
I was working on a similar problem when I learned that ‘builtin’ means something like "there is no source file associated with this object".
Here’s a solution based on checking the /lib and /Dlls folders manually.
The use of "unique" may be redundant; it’s there because I’m not sure if DLLs is strictly for packages which come with python/it was useful for a different problem I was trying to solve (finding out the requirements for a source package).
from typing import Generator, Iterable
import os, re, sys
def unique(iterable:Iterable, yielded_list=None) -> Generator:
"""
Iterate over unique elements of an iterable
examples:
>>> [*unique('abbcccdddd')]
['a', 'b', 'c', 'd']
>>> [*unique('abcd')]
['a', 'b', 'c', 'd']
"""
yielded = yielded if not isinstance(yielded, type(None)) else []
for i in iterable:
if not i in yielded:
yield i
yielded.append(i)
def native_modules() -> Generator:
"""
Every module found:
under your installation's /lib and /DLLs directories; excuding any under /lib/site-packages/*
in sys.builtin_module_names
"""
omitables = 'site-packages __pycache__ .pyo .ico .dll .pdb'.split()
path = lambda folder: os.path.join(sys.exec_prefix, folder)
tidy = lambda pth: os.path.split(os.path.splitext(pth)[0])[-1] # separate the name from the path and extension, if present
discriminant = lambda pth: not any(re.match(i, pth) for i in map(re.escape, omitables))
lib = map(tidy, filter(discriminant, os.listdir(path('lib'))))
dlls = map(tidy, filter(discriminant, os.listdir(path('DLLs'))))
yielded = []
yield from yielded
yield from unique(sys.builtin_module_names, yielded)
yield from unique(lib, yielded)
yield from unique(dlls, yielded)
All the commands like dir(builtins) or help(builtins) or sys.builtin_module_names, are either missing several core inbuilt package names or giving thousands of lines of verbose output.
>>> help(‘modules’)
The best command to show all the installed modules is help(‘modules’). It gives you name of all the installed modules in python without missing anything.
Incase you need names of inbuilt modules only, create a temporary fresh venev and give the command there. It might be bit slow but is the only way by far in my experience to list every single package.
In Linux, you can get a dump of all the built-in modules that came with the fresh install with the following commands
sudo apt-get install python3-virtualenv
tmpDir=`mktemp -d`
python3 -m virtualenv $tmpDir
source /tmp/virtualenv/bin/activate
python
help('modules')
Example Execution
Here’s an example execution of the above commands in Debian 11 with python 3.9
user@disp643:~$ sudo apt-get install python3-virtualenv
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3-virtualenv is already the newest version (20.4.0+ds-2+deb11u1).
The following packages were automatically installed and are no longer required:
ethtool libbotan-2-17 libtspi1 linux-image-5.10.0-10-amd64
linux-image-5.10.0-13-amd64 linux-image-5.10.0-14-amd64
linux-image-5.10.0-15-amd64 linux-image-5.10.0-16-amd64
linux-image-5.10.0-17-amd64 net-tools sse3-support
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
user@disp643:~$
user@disp643:~$ tmpDir=`mktemp -d`
user@disp643:~$
user@disp643:~$ python3 -m virtualenv $tmpDir
created virtual environment CPython3.9.2.final.0-64 in 120ms
creator CPython3Posix(dest=/tmp/tmp.qQsKZHGZqk, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/user/.local/share/virtualenv)
added seed packages: pip==20.3.4, pkg_resources==0.0.0, setuptools==44.1.1, wheel==0.34.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
user@disp643:~$ source /tmp/virtualenv/bin/activate
(virtualenv) user@disp643:~$ python
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help('modules')
Please wait a moment while I gather a list of all available modules...
__future__ _tracemalloc graphlib retrying
_abc _uuid grp rlcompleter
_aix_support _virtualenv gzip runpy
_ast _warnings hashlib sched
_asyncio _weakref heapq secrets
_bisect _weakrefset hmac select
_blake2 _xxsubinterpreters html selectors
_bootlocale _xxtestfuzz html5lib setuptools
_bootsubprocess _zoneinfo http shelve
_bz2 abc idna shlex
_codecs aifc imaplib shutil
_codecs_cn antigravity imghdr signal
_codecs_hk appdirs imp site
_codecs_iso2022 argparse importlib sitecustomize
_codecs_jp array inspect smtpd
_codecs_kr ast io smtplib
_codecs_tw asynchat ipaddr sndhdr
_collections asyncio ipaddress socket
_collections_abc asyncore itertools socketserver
_compat_pickle atexit json spwd
_compression audioop keyword sqlite3
_contextvars base64 lib2to3 sre_compile
_crypt bdb linecache sre_constants
_csv binascii locale sre_parse
_ctypes binhex logging ssl
_ctypes_test bisect lzma stat
_curses builtins mailbox statistics
_curses_panel bz2 mailcap string
_datetime cProfile marshal stringprep
_dbm calendar math struct
_decimal certifi mimetypes subprocess
_elementtree cgi mmap sunau
_functools cgitb modulefinder symbol
_hashlib chunk msgpack symtable
_heapq cmath multiprocessing sys
_imp cmd netrc sysconfig
_io code nis syslog
_json codecs nntplib tabnanny
_locale codeop ntpath tarfile
_lsprof collections nturl2path telnetlib
_lzma colorama numbers tempfile
_markupbase colorsys opcode termios
_md5 compileall operator test
_multibytecodec concurrent optparse textwrap
_multiprocessing configparser os this
_opcode contextlib ossaudiodev threading
_operator contextlib2 packaging time
_osx_support contextvars parser timeit
_peg_parser copy pathlib tkinter
_pickle copyreg pdb token
_posixshmem crypt pep517 tokenize
_posixsubprocess csv pickle toml
_py_abc ctypes pickletools trace
_pydecimal curses pip traceback
_pyio dataclasses pipes tracemalloc
_queue datetime pkg_resources tty
_random dbm pkgutil turtle
_sha1 decimal platform types
_sha256 difflib plistlib typing
_sha3 dis poplib unicodedata
_sha512 distlib posix unittest
_signal distutils posixpath urllib
_sitebuiltins doctest pprint urllib3
_socket easy_install profile uu
_sqlite3 email progress uuid
_sre encodings pstats venv
_ssl enum pty warnings
_stat errno pwd wave
_statistics faulthandler py_compile weakref
_string fcntl pyclbr webbrowser
_strptime filecmp pydoc wheel
_struct fileinput pydoc_data wsgiref
_symtable fnmatch pyexpat xdrlib
_sysconfigdata__linux_x86_64-linux-gnu formatter pyparsing xml
_sysconfigdata__x86_64-linux-gnu fractions queue xmlrpc
_testbuffer ftplib quopri xxlimited
_testcapi functools random xxsubtype
_testimportmultiple gc re zipapp
_testinternalcapi genericpath readline zipfile
_testmultiphase getopt reprlib zipimport
_thread getpass requests zlib
_threading_local gettext resolvelib zoneinfo
_tkinter glob resource
Enter any module name to get more help. Or, type "modules spam" to search
for modules whose name or summary contain the string "spam".
>>>