How to log source file name and line number in Python
Question:
Is it possible to decorate/extend the python standard logging system, so that when a logging method is invoked it also logs the file and the line number where it was invoked or maybe the method that invoked it?
Answers:
Sure, check formatters in logging docs. Specifically the lineno and pathname variables.
%(pathname)s Full pathname of the source file where the logging call was issued(if available).
%(filename)s Filename portion of pathname.
%(module)s Module (name portion of filename).
%(funcName)s Name of function containing the logging call.
%(lineno)d Source line number where the logging call was issued (if available).
Looks something like this:
formatter = logging.Formatter('[%(asctime)s] p%(process)s {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s','%m-%d %H:%M:%S')
On top of Seb’s very useful answer, here is a handy code snippet that demonstrates the logger usage with a reasonable format:
#!/usr/bin/env python
import logging
logging.basicConfig(format='%(asctime)s,%(msecs)03d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s',
datefmt='%Y-%m-%d:%H:%M:%S',
level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.debug("This is a debug log")
logger.info("This is an info log")
logger.critical("This is critical")
logger.error("An error occurred")
Generates this output:
2017-06-06:17:07:02,158 DEBUG [log.py:11] This is a debug log
2017-06-06:17:07:02,158 INFO [log.py:12] This is an info log
2017-06-06:17:07:02,158 CRITICAL [log.py:13] This is critical
2017-06-06:17:07:02,158 ERROR [log.py:14] An error occurred
To build on the above in a way that sends debug logging to standard out:
import logging
import sys
root = logging.getLogger()
root.setLevel(logging.DEBUG)
ch = logging.StreamHandler(sys.stdout)
ch.setLevel(logging.DEBUG)
FORMAT = "[%(filename)s:%(lineno)s - %(funcName)20s() ] %(message)s"
formatter = logging.Formatter(FORMAT)
ch.setFormatter(formatter)
root.addHandler(ch)
logging.debug("I am sent to standard out.")
Putting the above into a file called debug_logging_example.py produces the output:
[debug_logging_example.py:14 - <module>() ] I am sent to standard out.
Then if you want to turn off logging comment out root.setLevel(logging.DEBUG).
For single files (e.g. class assignments) I’ve found this a far better way of doing this as opposed to using print() statements. Where it allows you to turn the debug output off in a single place before you submit it.
For devs using PyCharm or Eclipse pydev, the following will produce a link to the source of the log statement in the console log output:
import logging, sys, os
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, format='%(message)s | '%(name)s:%(lineno)s'')
log = logging.getLogger(os.path.basename(__file__))
log.debug("hello logging linked to source")
See Pydev source file hyperlinks in Eclipse console for longer discussion and history.
import logging
# your imports above ...
logging.basicConfig(
format='%(asctime)s,%(msecs)d %(levelname)-8s [%(pathname)s:%(lineno)d in '
'function %(funcName)s] %(message)s',
datefmt='%Y-%m-%d:%H:%M:%S',
level=logging.DEBUG
)
logger = logging.getLogger(__name__)
# your classes and methods below ...
# A very naive sample of usage:
try:
logger.info('Sample of info log')
# your code here
except Exception as e:
logger.error(e)
Different from the other answers, this will log the full path of file and the function name that might have occurred an error. This is useful if you have a project with more than one module and several files with the same name distributed in these modules.
Example output:
2022-12-02:10:00:00,000 INFO [<stdin>:2 in function <module>] Sample of info log
2022-12-02:10:00:00,000 INFO [<full path>/logging_test_file.py:15 in function <module>] Sample of info log
If the logger is set with the GetLogger(name) option, where the name is a name that you specified you can also format the logger using %(name)s. You can specify a different name in every file with the GetLogger function, and when a log is produced you will know from which file comes through the name you set.
Example:
import logging
logging.getLogger("main")
logging.basicConfig(#filename=log_fpath,
level=log_level,
format='[%(asctime)s] src:%(name)s %(levelname)s:%(message)s',
handlers=[logging.FileHandler(log_fpath)])
Is it possible to decorate/extend the python standard logging system, so that when a logging method is invoked it also logs the file and the line number where it was invoked or maybe the method that invoked it?
Sure, check formatters in logging docs. Specifically the lineno and pathname variables.
%(pathname)s Full pathname of the source file where the logging call was issued(if available).
%(filename)s Filename portion of pathname.
%(module)s Module (name portion of filename).
%(funcName)s Name of function containing the logging call.
%(lineno)d Source line number where the logging call was issued (if available).
Looks something like this:
formatter = logging.Formatter('[%(asctime)s] p%(process)s {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s','%m-%d %H:%M:%S')
On top of Seb’s very useful answer, here is a handy code snippet that demonstrates the logger usage with a reasonable format:
#!/usr/bin/env python
import logging
logging.basicConfig(format='%(asctime)s,%(msecs)03d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s',
datefmt='%Y-%m-%d:%H:%M:%S',
level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.debug("This is a debug log")
logger.info("This is an info log")
logger.critical("This is critical")
logger.error("An error occurred")
Generates this output:
2017-06-06:17:07:02,158 DEBUG [log.py:11] This is a debug log
2017-06-06:17:07:02,158 INFO [log.py:12] This is an info log
2017-06-06:17:07:02,158 CRITICAL [log.py:13] This is critical
2017-06-06:17:07:02,158 ERROR [log.py:14] An error occurred
To build on the above in a way that sends debug logging to standard out:
import logging
import sys
root = logging.getLogger()
root.setLevel(logging.DEBUG)
ch = logging.StreamHandler(sys.stdout)
ch.setLevel(logging.DEBUG)
FORMAT = "[%(filename)s:%(lineno)s - %(funcName)20s() ] %(message)s"
formatter = logging.Formatter(FORMAT)
ch.setFormatter(formatter)
root.addHandler(ch)
logging.debug("I am sent to standard out.")
Putting the above into a file called debug_logging_example.py produces the output:
[debug_logging_example.py:14 - <module>() ] I am sent to standard out.
Then if you want to turn off logging comment out root.setLevel(logging.DEBUG).
For single files (e.g. class assignments) I’ve found this a far better way of doing this as opposed to using print() statements. Where it allows you to turn the debug output off in a single place before you submit it.
For devs using PyCharm or Eclipse pydev, the following will produce a link to the source of the log statement in the console log output:
import logging, sys, os
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, format='%(message)s | '%(name)s:%(lineno)s'')
log = logging.getLogger(os.path.basename(__file__))
log.debug("hello logging linked to source")
See Pydev source file hyperlinks in Eclipse console for longer discussion and history.
import logging
# your imports above ...
logging.basicConfig(
format='%(asctime)s,%(msecs)d %(levelname)-8s [%(pathname)s:%(lineno)d in '
'function %(funcName)s] %(message)s',
datefmt='%Y-%m-%d:%H:%M:%S',
level=logging.DEBUG
)
logger = logging.getLogger(__name__)
# your classes and methods below ...
# A very naive sample of usage:
try:
logger.info('Sample of info log')
# your code here
except Exception as e:
logger.error(e)
Different from the other answers, this will log the full path of file and the function name that might have occurred an error. This is useful if you have a project with more than one module and several files with the same name distributed in these modules.
Example output:
2022-12-02:10:00:00,000 INFO [<stdin>:2 in function <module>] Sample of info log
2022-12-02:10:00:00,000 INFO [<full path>/logging_test_file.py:15 in function <module>] Sample of info log
If the logger is set with the GetLogger(name) option, where the name is a name that you specified you can also format the logger using %(name)s. You can specify a different name in every file with the GetLogger function, and when a log is produced you will know from which file comes through the name you set.
Example:
import logging
logging.getLogger("main")
logging.basicConfig(#filename=log_fpath,
level=log_level,
format='[%(asctime)s] src:%(name)s %(levelname)s:%(message)s',
handlers=[logging.FileHandler(log_fpath)])