ConfluentKafka in Python: Using consumer to consumer set of records Question: I have started to learn confluent kafka(python). There is 1 producer, 1 topic, 1 partition and 1 consumer(a simple setup). My requirement is that I wish to fetch data collectively. I read that using poll(some_time) will wait for the required time and fetch records …
Get consumer group state using confluent kafka python Question: I am doing this way currently to get the groupmetadata for list of consumers admin = AdminClient({ ‘bootstrap.servers’ : config[‘kafka’][‘brokers’] }) for group in config[‘kafka’][‘groups’]: metadata = admin.list_groups(group) print(metadata[0].state) Is there a way to achieve the below metadata = admin.list_groups(config[‘kafka’][‘groups’]) //list of groups It throws an …
Can´t consume messages from topic Question: I have the following code: from confluent_kafka.admin import AdminClient, NewTopic a = AdminClient({‘bootstrap.servers’: ‘localhost:9092’}) new_topics = [NewTopic(topic, num_partitions=3, replication_factor=1) for topic in ["topic1", "topic2"]] fs = a.create_topics(new_topics) for topic, f in fs.items(): try: f.result() print("Topic {} created".format(topic)) except Exception as e: print("Failed to create topic {}: {}".format(topic, e)) Creating …
How to delete and then create topics correctly in a script for Kafka? Question: I am working on a script to refresh my topics on an AWS managed Kafka cluster. I need to wipe out the existing data whenever I run the script and I did it by deleting and creating the same topics again. …
consuming Kafka Avro massages in Python Question: I am trying to consume messages from Kafka Avro in Python. We have it in Java, and it’s working, but when trying to consume it in the Jupyter notebook, Parsing does not work. I followed the example given by the documentation: (I’ve removed conf information for security reasons) …
Error installing confluent kafka – ld: library not found for -lrdkafka Question: I am trying to install confluent kafka python module on my mac os. My python version is 3.9 I am keep getting below error. Does anyone know how to fix this? clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 …
How to get parameter hints in VSCode for pyd files in general and confluent_kafka specifically? Question: I am new to kafka, and trying work some basic examples via VScode. The problem is I can’t seem to make parameter hints work at all for all the artefacts imported via confluent_kafka. The module itself is a wrapper, …
