connecting python script to kafka docker container

Question:

I may overthinking this and I most likely am, cause I’ve gotten this to work before but not sure how. I have a docker container that boots up kafka and I’m trying to produce some records but getting a kafka.errors.NoBrokersAvailable: NoBrokersAvailable in my code so I’m not sure I’m calling it right. So here is a portion of my docker compose that creates the kafka instance.

  kafka:
    container_name: kafka
    image: debezium/kafka:latest
    ports:
      - "0.0.0.0:9092:9092"
    links:
      - zookeeper
    environment: 
      - ZOOKEEPER_CONNECT=zookeeper:2181
    volumes: 
        - ./kafka/data:/kafka/data
        - ./kafka/logs:/kafka/logs

And here is my python code I’m trying to run. (I also have zookeeper in my docker image above, but feeling like this piece is the piece that is relevant).

from kafka import KafkaConsumer, KafkaProducer
import json
from json import loads
from csv import DictReader 


bootstrap_servers = ['kafka:9092']
topicname='test-topic'
producer=KafkaProducer(bootstrap_servers=bootstrap_servers)
producer=KafkaProducer()

with open('test_data_producer.csv', 'r') as new_obj:
   csv_dict_reader=DictReader(new_obj)
   for row in csv_dict_reader:
       ack = producer.send(topicname, json.dumps(row).encode('utf-8'))
       metadata = ack.get()
       print(metadata.topic, metadata.partition)
Asked By: Alex

||

Answers:

producer is not able to connect broker since kafka is not valid dns name, if you add following in your hosts file 127.0.0.1 kafka and 127.0.0.1 zookeeper then your producer should be able to connect kafka.

Answered By: Niranjan
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.