Dataflow Job to start based on PubSub Notification – Python Question: I am writing a Dataflow job which reads from BigQuery and does a few transformations. data = ( pipeline | beam.io.ReadFromBigQuery(query=”’ SELECT * FROM `bigquery-public-data.chicago_crime.crime` LIMIT 100 ”’, use_standard_sql=True) | beam.Map(print) ) But my requirement is to read from BigQuery only after receiving a …
Google cloud PubSub service not working (Python) Question: I am trying to use the pub sub service on my python application. When I am running the code it get stuck on the last publisher line for some reason and the code never end. The subscriber seems fine.Does someone know what is wrong with my code? …
Bigquery dataset using pub/sub subscription Question: Trying to create a dataset using pub/sub subscription message. https://cloud.google.com/pubsub/docs/bigquery. This one provides how to write message to the table, but how can I create a dataset using the message. For example – if I have this is test1 in my subscription, then I should be able to take …
How to access Google Pub/Sub Custom Attributes Question: I have a set of Cloud Functions that send Pub/Sub messages to other functions in my pipeline. One of these functions is an email function. Ideally, I’d like to use Jinja2 templates in this email function that get filled in based on the custom attributes of my …
ImportError: cannot import name pubsub_v1 Question: I need to import Pubsub_v1 and bigquery from google.cloud module. I have installed it and pip freeze shows below : gapic-google-cloud-pubsub-v1==0.15.4 google-cloud-bigquery==0.26.0 google-cloud-pubsub==0.27.0 proto-google-cloud-pubsub-v1==0.15.4 In my python script, i am importing the modules as below: import os from google.cloud import pubsub_v1 import time import json from google.cloud import bigquery …
How to use Pubsub Emulator with PublisherClient from google-cloud-pubsub version 0.28 Question: I just update to google-cloud-pubsub version 0.28, where Google restructured the Python client into a PublisherClient and SubscriberClient. I was trying to connect to the Pubsub Emulator on localhost, but despite setting the environment variables, it still connects to the cloud pubsub service, …
