How to correctly use place holders for log file path/name within a config file?

Question

I am trying to set the log file name dynamically based on the path where the python script is running.

I have a config file that looks like this (it does contain additional configuration, what I am showing here is just the parts relevant to logging):

[loggers]
keys = root

[logger_root]
handlers = screen,file
level = NOTSET

[formatters]
keys = simple,complex

[formatter_simple]
format = %(asctime)s - %(name)s - %(levelname)s - %(message)s

[formatter_complex]
format = %(asctime)s - %(name)s - %(levelname)s - %(module)s : %(lineno)d - %(message)s

[handlers]
keys = file,screen

[handler_file]
class = handlers.TimedRotatingFileHandler
interval = midnight
backupcount = 5
formatter = complex
level = DEBUG
args = ('%(logfile)s',)

[handler_screen]
class = StreamHandler
formatter = simple
level = INFO
args = (sys.stdout,)

and my code looks like this:

    if not os.path.exists(DEFAULT_CONFIG_FILE) or not os.path.isfile(DEFAULT_CONFIG_FILE):
        msg = '%s configuration file does not exist!', config_file
        logging.getLogger(__name__).error(msg)
        raise ValueError(msg)
    try:
        logfilename = os.path.join(
            os.path.dirname(__file__), 'logs', 'camera.log')
        config.read(DEFAULT_CONFIG_FILE)
        logging.config.fileConfig(
            config, defaults={'logfile': logfilename}, disable_existing_loggers=False)
        logging.info(f'{DEFAULT_CONFIG_FILE} configuration file was loaded.')
    except Exception as e:
        logging.getLogger(__name__).error(
            'Failed to load configuration from %s!', DEFAULT_CONFIG_FILE)
        logging.getLogger(__name__).debug(str(e), exc_info=True)
        raise e

I am trying to achieve that the logs are written to a subdirectory called logs which is located in the path that the script is running.

What I actually get is a log file called %(logfile)s

I suppose that this is something really obvious but I am just going in circles!

Any help greatly appreciated.

Asked By: The Welsh Dragon

||

Source

Answer 1

Well I have found one solution.

I still do not understand why the OP did not work and would really appreciate it if someone who knows can add another answer but this achieves the same result:

[handler_file]
class = handlers.TimedRotatingFileHandler
interval = midnight
backupcount = 5
formatter = complex
level = DEBUG
args = (os.getcwd()+'/logs/camera.log',)

with this code

    if not os.path.exists(DEFAULT_CONFIG_FILE) or not os.path.isfile(DEFAULT_CONFIG_FILE):
        msg = '%s configuration file does not exist!', config_file
        logging.getLogger(__name__).error(msg)
        raise ValueError(msg)
    try:
        config.read(DEFAULT_CONFIG_FILE)

        logging.config.fileConfig(
            config, disable_existing_loggers=False)

        logging.info(f'{DEFAULT_CONFIG_FILE} configuration file was loaded.')
    except Exception as e:
        logging.getLogger(__name__).error(
            'Failed to load configuration from %s!', DEFAULT_CONFIG_FILE)
        logging.getLogger(__name__).debug(str(e), exc_info=True)
        raise e

Answered By: The Welsh Dragon

Answer 2

I did eventually get the ‘%(logfile)s’ parsing to substitute the variables correctly.

The issue was that the defaults should have been passed in to the initialisation of the ConfigParser not the logger. Like this:

    logfile = os.path.join(os.path.dirname(__file__), 'logs','camera.log')

    config = ConfigParser(defaults={'logfile': logfile})

    if not os.path.exists(DEFAULT_CONFIG_FILE) or not os.path.isfile(DEFAULT_CONFIG_FILE):
        msg = '%s configuration file does not exist!', config_file
        logging.getLogger(__name__).error(msg)
        raise ValueError(msg)
    try:
        config.read(DEFAULT_CONFIG_FILE)

        logging.config.fileConfig(
            config, disable_existing_loggers=False)

        logging.info(f'{DEFAULT_CONFIG_FILE} configuration file was loaded.')
    except Exception as e:
        logging.getLogger(__name__).error(
            'Failed to load configuration from %s!', DEFAULT_CONFIG_FILE)
        logging.getLogger(__name__).debug(str(e), exc_info=True)
        raise e

and not like this:

    logfile = os.path.join(os.path.dirname(__file__), 'logs','camera.log')
    logging.config.fileConfig(
        config,
        defaults={'logfile': logfile},
        disable_existing_loggers=False)

However, I then started running in to issues with unicode errors:

yntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated UXXXXXXXX escape

Caused, I think, by the Users in the path.

I gave up at this point and simply set the working directory = to the the directory where the script was installed

os.path.dirname(__file__)

Answered By: The Welsh Dragon

Answer 3

On Windows, I got your initial configuration to work by adding as_posix()

logging.config.fileConfig(
    config, defaults={"logfile": path.as_posix(),},
)

where path is a pathlib.Path instance (from pathlib import Path)

[handler_file]
class = FileHandler
formatter = file
args=('%(logfile)s',)

Answered By: thomast

How to correctly use place holders for log file path/name within a config file?

Question:

Answers: