BigQuery : Python Client not capturing errors found in audit logs when running multiple statements in a single query

Question:

BigQuery : Python Client not capturing errors found in audit logs , when running multiple statements in a single query (for eg a declare variable statement before a create table statement ).

There was a syntax issue in variable declaration in the query , The python client is not able to catch the error and it deems the status as done . However inspection of logs in the auditlog trail shows the error

BQ Client Code Snippet:

            query_job = self.client.query(query) 
            print(dir(query_job))
            # print(query_job.exception())
            with suppress(TypeError):
                ##https://github.com/googleapis/python-bigquery/issues/1459
                exc = query_job.exception()
                if exc:
                    raise exc
            while (query_job.state != 'DONE'):
                
                print("Job {} is currently in state {}".format(query_job.job_id, query_job.state))

                if query_job.errors is not None:
                    raise Exception("Bigquery job failed with error {}".format(query_job.errors)) 

                query_job = self.client.get_job(query_job.job_id)
                print(query_job.total_bytes_processed)
                print(query_job.errors)
                time.sleep(wait)

The log trail shows below error :

jobStatus: {
error: {
code: 11
message: "Query error: Unrecognized name: varname at [4:56]"
}
state: "DONE"
}}}}
serviceName: "bigquery.googleapis.com"

Is there a way to capture such errors / exceptions ?

Asked By: Nixon

||

Answers:

You can get the detailed error by using the message field of .errors() function return.

try:
    job = client.query(query)
    job.result()
except:
    for error in job.errors:
        print('Following error occurred while executing the query: {}'.format(error['message']))
    
Answered By: Sakshi Gatyan

To add to @Sakshi Gatyan‘s answer , adding parameter (create_session=True) to query job config helped in identifying the entire query containing multiple statements as a single session along with waiting for query_job.result()

query_job = self.client.query(query,job_config=bq.QueryJobConfig(create_session=True))
query_job = self.client.get_job(query_job.job_id)

query_job.result()
Answered By: Nixon