Exporting a BigQuery table with Airflow: "extract_table() missing 1 required positional argument: 'self'; 1764166" error

Question

I’m trying to use an airflow task to export a BigQuery table to a Google Cloud Storage, but I’m getting the following error message:

{standard_task_runner.py:93} ERROR - Failed to execute job 1819 for task export_objs_table_to_bucket (extract_table() missing 1 required positional argument: 'self'; 1764166)

This is the airflow task I created:

task2_objs = PythonOperator(
        task_id = "export_objs_table_to_bucket",
        python_callable = callables.bq_export_to_gcs,
        op_kwargs = {
            "bucket": BUCKET,
            "blob": BLOB_OBJ,
            "project": PROJECT,
            "dataset_id": DATASET_ID,
            "table_id": TABLE_ID_OBJS
        }
    )

And this is the callable Python function that I use in that task, which, by the way, I took from the GCP documentation: https://cloud.google.com/bigquery/docs/samples/bigquery-extract-table?hl=en-419

def bq_export_to_gcs(bucket: str, 
                     blob: str, 
                     project: str, 
                     dataset_id: str, 
                     table_id: str) -> None:

    client = bigquery.Client

    destination_uri = f"gs://{bucket}/{blob}"

    dataset_ref = bigquery.DatasetReference(project, dataset_id)
    table_ref = dataset_ref.table(table_id)


    extract_job = client.extract_table(
        source = table_ref,
        destination_uris=destination_uri,
        location="europe-west1",
    )
    extract_job.result()

I don’t really know if it has to do with Airflow or with BigQuery, but as I mentioned, I’m doing it as it is written in the BigQuery documentation so I’m completely lost.

Asked By: JMarcos87

||

Source

Answer 1

I was able to replicate your use case using your provided code. The problem is that you were not able to create client properly because it was not instantiated correctly.

You should update the line of code client = bigquery.Client to client = bigquery.Client()

Answered By: Scott B

Answer 2

The issue is solved but I wanted to show another solution anyway.

You can also use an existing operator to extract a table to GCS:

An example that exports a table to a CSV file with , as field delimiter.

BigQueryToGCSOperator(
        task_id='task_id',
        source_project_dataset_table='my_project.my_dataset.my_table',
        export_format='CSV',
        destination_cloud_storage_uris=[
            'my_bucket_path'
        ],
        field_delimiter=','
    )

Answered By: Mazlum Tosun

Exporting a BigQuery table with Airflow: "extract_table() missing 1 required positional argument: 'self'; 1764166" error

Question:

Answers: