How to use nested proto.Message with BigQuery Storage API Writer python client? Question: Based on the snippet from https://github.com/googleapis/python-bigquery-storage/issues/398 which uses proto-plus package to define protobuff message in python, is very helpful and works well as it is, but in case of the nested message it does not work. The below adapted code throws the …
Exporting a BigQuery table with Airflow: "extract_table() missing 1 required positional argument: 'self'; 1764166" error Question: I’m trying to use an airflow task to export a BigQuery table to a Google Cloud Storage, but I’m getting the following error message: {standard_task_runner.py:93} ERROR – Failed to execute job 1819 for task export_objs_table_to_bucket (extract_table() missing 1 required …
Replace a Regex look ahead for Google Big Query Question: I have some data for which i want to select the LAST VALUE BEFORE THE DELIMITER Example- A -> B -> C -> D In this case, i want to select "C" so i used a positive look ahead but BQ doesnt allow look aheads/behinds …
Dataproc: Errors when reading and writing data from BigQuery using PySpark Question: I am trying to read some BigQuery data, (ID: my-project.mydatabase.mytable [original names protected]) from a user-managed Jupyter Notebook instance, inside Dataproc Workbench. What I am trying is inspired in this, and more specifically, the code is (please read some additional comments, on the …
Handling newline character while executing Bigquery cmd via Python Question: I have sql query in a string variable that I am executing via Python. The problem I am facing is that I can’t pass ‘n’ in the string variable. When using CHR(13) || CHR(10) I get following error – Argument 2 to STRING_AGG must be …
Passing Dynamic Date Variable in Big Query| Python Question: I’m trying to run this bigQuery [in python]: start_date= "2022-03-08" end_date = "2022-05-03" query = f""" SELECT SUM(CostInBillingCurrency) as cost,EXTRACT(MONTH from date) as MONTH, SubscriptionId, SubscriptionName FROM `table_name` WHERE Date BETWEEN TIMESTAMP({start_date}) AND TIMESTAMP({end_date}) GROUP BY SubscriptionId, SubscriptionName,MONTH """ this is the error I’m getting: google.api_core.exceptions.BadRequest: …
How to update rows in a BigQuery table using airflow Question: I am coding a DAG and want to execute an UPDATE statement to selectively set the values of certain fields in certain rows. The SQL statement is easy, but I am not sure how to execute it via Airflow. The documentation on BigQueryUpdateTableOperator here …
Google BigQuery query in Python works when using result(), but Permission issue when using to_dataframe() Question: I’ve run into a problem after upgrades of my pip packages and my bigquery connector that returns query results suddenly stopped working with following error message from google.cloud import bigquery from google.oauth2 import service_account credentials = service_account.Credentials.from_service_account_file(‘path/to/file’, scopes=[‘https://www.googleapis.com/auth/cloud-platform’, ‘https://www.googleapis.com/auth/drive’, …
Use query string template from config in bigquery result Question: Trying to use the dynamic query string from config file and format in the result section using pyodbc, where the query string is printing as static value. Expecting to print the DB result populate dynamically. Code: def bq_exec_sql(sql): client = bigquery.Client(project=’development’) return client.query(sql) def generate_select(sql, …
ValueError: install DBtypes to use this function Question: I’m using BigQuery for the first time. client.list_rows(table, max_results = 5).to_dataframe(); Whenever I use to_dataframe() it raises this error: ValueError: Please install the ‘db-dtypes’ package to use this function. I found this similar problem (almost exactly the same), but I can’t understand how to implement their proposed …
