BigQuery Python 409 Already Exists: Table

Question

I’m coding a python script that writes query results to a BQ table . After the first time running the script, it always errors out after that with the following error: google.api_core.exceptions.Conflict: 409 Already Exists: Table project-id.dataset-id. I do not understand why it is attempting to create a table everytime I run the script. Do I have specify any specific parameters?

This is from the documentation from google. I’m using this as an example and under the idea that a current table has already been created. Where do I stop the api from creating the same table ?

from google.cloud import bigquery

# TODO(developer): Construct a BigQuery client object.
client = bigquery.Client()

# TODO(developer): Set table_id to the ID of the destination table.
 table_id = "your-project.your_dataset.your_table_name"

job_config = bigquery.QueryJobConfig(destination=table_id)

sql = """
    SELECT corpus
    FROM `bigquery-public-data.samples.shakespeare`
    GROUP BY corpus;
"""

# Start the query, passing in the extra configuration.
query_job = client.query(sql, job_config=job_config)  # Make an API request.
query_job.result()  # Wait for the job to complete.

print("Query results loaded to the table {}".format(table_id))

Asked By: Malcode

||

Source

Answer 1

If you check the class QueryJobConfig you will see that there is a parameter called write_disposition. As you can see in the REST API reference here, this parameter can be set to 3 different options:

WRITE_TRUNCATE: If the table already exists, BigQuery overwrites the table data and uses the schema from the query result.
WRITE_APPEND: If the table already exists, BigQuery appends the data to the table.
WRITE_EMPTY: If the table already exists and contains data, a ‘duplicate’ error is returned in the job result.

So, adding this line after the job_config definiton will do the trick:

job_config.write_disposition = bigquery.WriteDisposition.WRITE_TRUNCATE

Answered By: itroulli

Answer 2

look into this BigQuery client documentation here look into exists_ok parameter

Answered By: malik

BigQuery Python 409 Already Exists: Table

Question:

Answers: