BigQuery : Python Client not capturing errors found in audit logs when running multiple statements in a single query
Question:
BigQuery : Python Client not capturing errors found in audit logs , when running multiple statements in a single query (for eg a declare variable statement before a create table statement ).
There was a syntax issue in variable declaration in the query , The python client is not able to catch the error and it deems the status as done . However inspection of logs in the auditlog trail shows the error
BQ Client Code Snippet:
query_job = self.client.query(query)
print(dir(query_job))
# print(query_job.exception())
with suppress(TypeError):
##https://github.com/googleapis/python-bigquery/issues/1459
exc = query_job.exception()
if exc:
raise exc
while (query_job.state != 'DONE'):
print("Job {} is currently in state {}".format(query_job.job_id, query_job.state))
if query_job.errors is not None:
raise Exception("Bigquery job failed with error {}".format(query_job.errors))
query_job = self.client.get_job(query_job.job_id)
print(query_job.total_bytes_processed)
print(query_job.errors)
time.sleep(wait)
The log trail shows below error :
jobStatus: {
error: {
code: 11
message: "Query error: Unrecognized name: varname at [4:56]"
}
state: "DONE"
}}}}
serviceName: "bigquery.googleapis.com"
Is there a way to capture such errors / exceptions ?
Answers:
You can get the detailed error by using the message
field of .errors() function return.
try:
job = client.query(query)
job.result()
except:
for error in job.errors:
print('Following error occurred while executing the query: {}'.format(error['message']))
To add to @Sakshi Gatyan‘s answer , adding parameter (create_session=True) to query job config helped in identifying the entire query containing multiple statements as a single session along with waiting for query_job.result()
query_job = self.client.query(query,job_config=bq.QueryJobConfig(create_session=True))
query_job = self.client.get_job(query_job.job_id)
query_job.result()
BigQuery : Python Client not capturing errors found in audit logs , when running multiple statements in a single query (for eg a declare variable statement before a create table statement ).
There was a syntax issue in variable declaration in the query , The python client is not able to catch the error and it deems the status as done . However inspection of logs in the auditlog trail shows the error
BQ Client Code Snippet:
query_job = self.client.query(query)
print(dir(query_job))
# print(query_job.exception())
with suppress(TypeError):
##https://github.com/googleapis/python-bigquery/issues/1459
exc = query_job.exception()
if exc:
raise exc
while (query_job.state != 'DONE'):
print("Job {} is currently in state {}".format(query_job.job_id, query_job.state))
if query_job.errors is not None:
raise Exception("Bigquery job failed with error {}".format(query_job.errors))
query_job = self.client.get_job(query_job.job_id)
print(query_job.total_bytes_processed)
print(query_job.errors)
time.sleep(wait)
The log trail shows below error :
jobStatus: {
error: {
code: 11
message: "Query error: Unrecognized name: varname at [4:56]"
}
state: "DONE"
}}}}
serviceName: "bigquery.googleapis.com"
Is there a way to capture such errors / exceptions ?
You can get the detailed error by using the message
field of .errors() function return.
try:
job = client.query(query)
job.result()
except:
for error in job.errors:
print('Following error occurred while executing the query: {}'.format(error['message']))
To add to @Sakshi Gatyan‘s answer , adding parameter (create_session=True) to query job config helped in identifying the entire query containing multiple statements as a single session along with waiting for query_job.result()
query_job = self.client.query(query,job_config=bq.QueryJobConfig(create_session=True))
query_job = self.client.get_job(query_job.job_id)
query_job.result()