How to limit BigQuery job by slots Question: I’d like to limit a BigQuery job (query) submitted with the Python SDK to use a certain amount of slots. For example, I have a provision with 400 slots in BigQuery (flat-rate). And let’s say I’ve got a heavy query which uses 300 slots during execution (according …
Cloud Functions: How to send a daily email at 10am with an extract of a table? Question: New to GCP Cloud Functions, and I am wanting to send a daily email at 10am, which would contain an extract of a specific table in BigQuery. So far, I have written my code to send an email; …
BigQuery : Python Client not capturing errors found in audit logs when running multiple statements in a single query Question: BigQuery : Python Client not capturing errors found in audit logs , when running multiple statements in a single query (for eg a declare variable statement before a create table statement ). There was a …
Python to SQL using strip Question: gday I would like to know how to convert the following python script to sql and with its source file being a txt file Data = [ r[0: 2].strip(), # Column 1 r[2: 14].strip() # Column 2 ] I’ve loaded the txt file into a table with a single …
How to pivot without aggregation in python or sql Question: I want to pivot a table such that each row in the type column is now its own row. Each metric is then a row and the values are the intersection of the metric and type. There are a variable number of types and metrics.Example. …
Jupyter lab syntax error for the same %%bigquery command on the GCP environment Question: I am trying to fetch a BigQuery table as a Python dataframe in the GCP’s Jupyter lab environment. It worked perfectly fine and also performed analysis on the dataframe. However, after restarting the kernel, I am unable to recreate the df …
Airflow: BigQuery SQL Insert empty data to the table Question: Using Airflow, I am trying to get the data from one table to insert it into another in BigQuery. I have 5 origin tables and 5 destination tables. My SQL query and python logic work for the 4 tables where it successfully gets the data …
How to deploy BigQuery and access BigQuery to public internet? Question: I want to BigQuery as datawarehouse in my company. How BigQuery can access to all of data team member? Is install BigQuery in SSH Compute Engine ? Or don’t need install BigQuery in SSH Compute Engine and just give user access in IAM then …
How to read bytes type from bigquery in Java? Question: We have a legacy dataflow job in Scala which basically reads from Bigquery and then dumps it into Postgres. In Scala we read from bigquery, map it onto a case class and then dump it into Postgres, and it works perfectly for bigquery’s Bytes type …
Can overlapping matches with the same start position be found using regex? Question: I am looking for a regex or a regex flag in python/BigQuery that enables me to find overlapping occurrences. For example, I have the string 1.2.5.6.8.10.12 and I would like to extract: [1., 1.2., 1.2.5., 1.2.5.6., …, 1.2.5.6.8.10.12] I tried running the …
