Using Logging correctly in python

Question

My code looks like this. It is for image reshuffling in a couple of folders. Please assume that I have made all the required imports correctly.

logging.basicConfig(filename = 'make_folders.log',  filemode= 'w', level=logging.INFO, format='%(asctime)::%(message)s')




def get_path_list(directory: str) -> list:
    """
    Get a list of absolute paths of all the files in the directory.

    :param directory: Path to the directory from which the file paths are to be extracted.

    """
    path_list =[]

    for file in glob.glob(pathname=directory+'/*.png', recursive=True):
        path_list.append(file)

    return path_list


# Make a list of all the images contained in folder A and folder B
img_A = get_path_list(args.source_A)
img_B = get_path_list(args.source_B)

try:
    assert len(img_A) == len(img_B)

except AssertionError:
    logging.error("The number of images in source_A folder and source_B folder are not the same. Check input folders.")

# Find how many images we want in training, validation and test sets based on the length of list containing all image paths.
train_len = round(0.7 * len(img_A))
val_len = round(0.15 * len(img_A))
test_len = val_len

logging.info("Total number of training, validation and test images in destination folder A and B are {}, {} and {} respectively".format(train_len, val_len, test_len))

I am getting following error while using logging.info()

logging.info("Total number of training, validation and test images in destination folder A and B are {}, {} and {} respectively".format(train_len, val_len, test_len))
Message: 'Total number of training, validation and test images in destination folder A and B are 5265, 1128 and 1128 respectively'
Arguments: ()

can someone please tell me what am I doing wrong?

Asked By: Shubham Deshpnde

||

Source

Answer 1

Your error comes from:

format='%(asctime)::%(message)s'

Change it to:

format='%(asctime)s %(message)s'

And it works.

Additionally, newer versions of Python suggest we use f-strings instead of string formatting:

logging.info(f"Total number of training, validation and test images in destination folder A and B are {train_len}, {val_len} and {test_len} respectively")

Answered By: Laguilhoat

Answer 2

You should create a new logger instance yourself and not use the root logger as that gets confusing fast.

An example:

# Import the logging package and a file handler that can rotate your logs automatically
import logging
from logging.handlers import RotatingFileHandler

# Create logger and set a name for the logger
# You can write a name yourself, or use the __name__ variable 
logger = logging.getLogger(__name__)
# Add a consolehandler so your log is printed on your screen
consoleHandler = logging.StreamHandler()
loglevel = logging.DEBUG
formatter = logging.Formatter("%(asctime)s [%(levelname)s] - %(message)s")
# Filehandler so you can read back your logs when the script has quit
# It automatically rotates the log after it reaches a size of 1 MegaByte
# You can configure it however you want, just see the docs
fileHandler = RotatingFileHandler("your_log.log", maxBytes=1 * 1024 * 1024, backupCount=3)

# Set loglevel of all handlers and the logger itself.
consoleHandler.setLevel(loglevel)
fileHandler.setLevel(loglevel)
logger.setLevel(loglevel)  # Not entirely sure if this line is needed

# Assign the formatter you defined earlier to the two handlers
consoleHandler.setFormatter(formatter)
fileHandler.setFormatter(formatter)

# Add the consolehandler, but check if there's a handler already
# If you don't check in some situations you might get duplicate handlers added to your logger, resulting in all your log lines to be printend multiple times.
if len(logger.handlers) == 0:
    logger.debug("Adding new log handler, none exists yet.")
    logger.addHandler(consoleHandler)
    logger.addHandler(fileHandler)

Now if you want to print a line to the log you can do:

logger.debug("This message is only visible when the debug level is set")
logger.info("An informative message")
logger.warning("A warning to the user")
logger.error("An error has occurred, please check...")

Answered By: Saaru Lindestøkke

Using Logging correctly in python

Question:

Answers: