Catching the exception thrown by python script in shell script

Question

I am having a shell script which opens a file and pass it to python script for processing it. So if there is any issue with the file ( e.g. the file content is not in the format required for the successful execution for python script), the python script throws an exception. Since my objective it to process N files using python script. I need to know which is the file causing the script to break. I read on how to catch the exception in thrown by command execution. http://linuxcommand.org/wss0150.php . But in my case its the python script which throws the exception and I need to know in shell script what the exception has been thrown. Can anyone help me how can I proceed with this?

Below is the code snippet :

#!/bin/bash
yesterday=$(date --date "yesterday" "+%y%m%d")
ftoday=$(date --date "today" "+%m-%d-%Y")
year=$(date "+%Y")
fileList=$(find C:/logdata/$year/$ftoday/ -iname "*"$yesterday".log")
for var in $fileList
do
echo -e "n START Processing File : $var" >> shelltestlog.txt
cat $var| ./scriptA.py 
echo -e "n END Processing File : $var" >> shelltestlog.txt
done

Asked By: Codelearner

||

Source

Answer 1

An uncaught exception will produce a traceback printed to standard error. The best you can do is capture that, and try to parse it.

if ! ./scriptA.py < "$var" 2> stderr.txt; then
  # Parse stderr.txt to see what exception was raised.
fi

Answered By: chepner

Answer 2

If your python script returns an non-zero error level whenever it gets an exception, you can use || { } to log messages:

./scriptA.py < "$file" || {
    printf "n Python script scriptA.py failed with file "%s".n" "$file" >> shelltestlog.txt
}

I actually tried to simplify your code first:

#!/bin/bash

yesterday=$(date --date "yesterday" "+%y%m%d")
ftoday=$(date --date "today" "+%m-%d-%Y")
year=$(date "+%Y")

readarray -t filesList < <(find C:/logdata/$year/$ftoday/ -iname "*"$yesterday".log")

for file in "${filesList[@]}"; do
    printf "n START Processing File : %sn" "$file" >> shelltestlog.txt
    ./scriptA.py < "$file" || {
        printf "n Python script scriptA.py failed with file "%s".n" "$file" >> shelltestlog.txt
    }
    printf "n END Processing File : %sn" "$file" >> shelltestlog.txt
done

Answered By: konsolebox

Answer 3

You can play some redirection games:

# open file descriptor 3, pointing to the same destination as stdout
exec 3>&1

# run the script, 
# python's stderr goes to stdout so it can be captured in a variable
# python's stdout goes to file descriptor 3, displayed on the terminal
python_stderr=$( ./scriptA.py < "$var" 2>&1 1>&3 )

# close the file descriptor
exec 3>&-

Then you can examine $python_stderr for patterns

case $python_stderr in
    "") echo "nothing on stderr" ;;
    *foo*) handle_foo_error ;;
    *bar*) handle_bar_error ;;
    *baz*) handle_baz_error ;;
esac

Answered By: glenn jackman

Answer 4

You should rewrite the shell script in Python and merge it into the existing Python script.
The stuff that you are currently doing with the date command, can be done with the datetime and time modules. The stuff you are doing with find can be done with os.walk and fnmatch.fnmatch.

Here’s an outline:

#! /usr/bin/python

def scriptA(file_to_process):
    # the code currently in scriptA goes here;
    # modify it to read from `file_to_process` rather than `sys.stdin`

def get_file_list():
    # your bash logic to construct the file list goes here,
    # converted to Python as suggested above

def main():
   for filename in get_file_list():
       sys.stderr.write("Processing {}...n".format(filename))
       try:
           scriptA(open(filename, "rt"))

       except SomeExceptionType as e:
           # handle exception...

       except SomeOtherExceptionType as e:
           # handle other kind of exception...

       sys.stderr.write("Done processing {}.n".format(filename))

main()

Answered By: zwol

Answer 5

You can make use of a special $? variable in bash which contains the exit code of the previous command that was executed. Since you are running a python script inside bash, once the call ends it will generate some exit code.

Now, we need to manually access the exit code which can be found in bash $? variable which stores the exit codes. see more here

It is worth noting that successful execution of a process gives exit code 0. So any other exit code means there was some exception involved in the execution. So you can make use of if else here like this:

./scriptA.py
if [ $? -gt 0 ] ;then
    # echo "exit code $?"
    # do something with it
else
    # do something else
fi

This link is also helpful in understanding exit codes

Answered By: Mahi

Catching the exception thrown by python script in shell script

Question:

Answers: