Use query string template from config in bigquery result

Question

Trying to use the dynamic query string from config file and format in the result section using pyodbc, where the query string is printing as static value. Expecting to print the DB result populate dynamically.

Code:

def bq_exec_sql(sql):
    client = bigquery.Client(project='development')
    return client.query(sql)

def generate_select(sql, templete_select):
    job = bq_exec_sql(sql)
    result = ''
    print(templete_select)
    try:
        for row in job:
            result += templete_select
            
        print(result)
                
if __name__ == '__main__':
    for source in dashboard_activity_list:
       sql = config.get(source).source_select // from config file
       templete_select =config.get(source).select_template   // from config file
       generate_select(sql, templete_select)

Actual Output:

 select '"+row['table_id']+"' as table_id, '"+row['frequency']+"' as frequency from `trend-dev.test.table1`
 sselect '"+row['table_id']+"' as table_id, '"+row['info']+"' as info from `trend-dev.test.table2`

Expected Output:

 select table_name1 as table_id, daily from `trend-dev.test.table1`
 select table_name2 as table_id, chart_data from `trend-dev.test.table2`

Config file:

   dashboard_activity_list: [source_partner_trend, source_partner_normal]
   source_partner_trend: 
       source_select : select * from `trend-dev.test.trend_partner` 
       source_key : trend_source_partner
       distination_table : test.trend_partner_dashboard_feed
       select_template : select '"+row['table_id']+"' as table_id, '"+row['frequency']+"' as frequency from `trend-dev.test.table1`
   source_partner_normal: 
       source_select : select * from `trend-dev.test.normal_partner` 
       source_key : normal_source_partner
       distination_table : test.normal_partner_dashboard_feed
       select_template : select '"+row['table_id']+"' as table_id, '"+row['info']+"' as info from `trend-dev.test.table2`

Asked By: Charles

||

Source

Answer 1

I have replicated your code and it appears that its passing the whole query in templete_select variable as text string, I modified it so that It can separate the row from text string at the select statement and produced the expected output.

Working Snippet:

from google.cloud import bigquery

def bq_exec_sql(sql):
    client = bigquery.Client(project='development')
    return client.query(sql)

def generate_select(sql, templete_select):
    job = bq_exec_sql(sql)
    result = ''
    
    try:
        for row in job:
            result += templete_select.format(row['table_id'])
            
        print(result)
    except:
        print('error')

if __name__ == '__main__':
    sql = 'select * from `<table_name>` limit 2'
    templete_select =" select {} as table_id"
    generate_select(sql, templete_select)

Answered By: Nestor Ceniza Jr

Answer 2

I found out one of best option to sort out the question, where use eval() method to assign dynamic string value formated to the resulted loop.

result += eval(f’f{templete_select!r}’)

Ref : https://python-forum.io/thread-24481.html

Answered By: Charles

Use query string template from config in bigquery result

Question:

Answers: