ConfluentKafka in Python: Using consumer to consumer set of records Question: I have started to learn confluent kafka(python). There is 1 producer, 1 topic, 1 partition and 1 consumer(a simple setup). My requirement is that I wish to fetch data collectively. I read that using poll(some_time) will wait for the required time and fetch records …
Kafka consumer error: failed to deserialize – the unknown protocol id Question: I am running Kafka locally, and sending a "hello, world!" message using the kafka-console-producer.sh using the following command: kafka-console-producer.sh –topic quickstart-events –bootstrap-server localhost:9092 > hello, world! I have a Kafka consumer running using the Quix Streams library. The code for the consumer is …
Saving Python dictionary as JSON file using Kafka and NiFi Question: I am attempting to send JSON data from Python, through Kafka, to a NiFi container where it will be saved. The Python script is below: server_address = "foobar:1234" producer = KafkaProducer(bootstrap_servers=server_address) data = {"type": "record", "name": "CSV", "namespace": "nifi", "fields": {"Id": 0, "Value": 10.1}} …
Python Kafka consumer message deserialisation using AVRO, without schema registry – problem Question: I have a problem with Kafka message deserializing. I use confluent kafka. There is no schema registry – schemas are hardcoded. I can connect consumer to any topic and receive messages, but I can’t deserialise these messages. Output after deserialisation looks something …
connecting python script to kafka docker container Question: I may overthinking this and I most likely am, cause I’ve gotten this to work before but not sure how. I have a docker container that boots up kafka and I’m trying to produce some records but getting a kafka.errors.NoBrokersAvailable: NoBrokersAvailable in my code so I’m not …
Spark + Kafka app, getting "CassandraCatalogException: Attempting to write to C* Table but missing primary key columns: [col1,col2,col3]" Question: Run env kafka —-ReadStream—-> local —-WriteStream—-> cassandra source code place on local and kafka, local, writeStream is different IP Table columns are: col1 | col2 | col3 | col4 | col5 | col6 | col7 df.printSchema …
Get consumer group state using confluent kafka python Question: I am doing this way currently to get the groupmetadata for list of consumers admin = AdminClient({ ‘bootstrap.servers’ : config[‘kafka’][‘brokers’] }) for group in config[‘kafka’][‘groups’]: metadata = admin.list_groups(group) print(metadata[0].state) Is there a way to achieve the below metadata = admin.list_groups(config[‘kafka’][‘groups’]) //list of groups It throws an …
GCP Dataflow Kafka and missing SSL certificates Question: I’m trying to fetch the data from Kafka to Bigquery using GCP Dataflow. My Dataflow template is based on Python SDK 2.42 + Container registry + apache_beam.io.kafka. There is my pipeline: def run( bq_dataset, bq_table_name, project, pipeline_options ): with Pipeline(options=pipeline_options) as pipeline: kafka = pipeline | ReadFromKafka( …
Can´t consume messages from topic Question: I have the following code: from confluent_kafka.admin import AdminClient, NewTopic a = AdminClient({‘bootstrap.servers’: ‘localhost:9092’}) new_topics = [NewTopic(topic, num_partitions=3, replication_factor=1) for topic in ["topic1", "topic2"]] fs = a.create_topics(new_topics) for topic, f in fs.items(): try: f.result() print("Topic {} created".format(topic)) except Exception as e: print("Failed to create topic {}: {}".format(topic, e)) Creating …
Ensure faust consumer/agent seeks to offset 0 after rebalance Question: Not much point explaining why this needs to happen, but we need to have a worker always read from the beginning of a topic on startup and after a rebalance. The following snippet does work on start to ensure that when you start a worker …
