Kafka consumer error: failed to deserialize – the unknown protocol id

Question

I am running Kafka locally, and sending a "hello, world!" message using the kafka-console-producer.sh using the following command:

kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
> hello, world!

I have a Kafka consumer running using the Quix Streams library. The code for the consumer is

import quixstreams as qx
import pandas as pd

client = qx.KafkaStreamingClient('127.0.0.1:9092')
topic_consumer = client.get_topic_consumer("quickstart-events", consumer_group=None)

def on_stream_received_handler(stream_received: qx.StreamConsumer):
    stream_received.timeseries.on_dataframe_received = on_dataframe_received_handler

def on_dataframe_received_handler(stream: qx.StreamConsumer, df: pd.DataFrame):
    print(df.to_string())

topic_consumer.on_stream_received = on_stream_received_handler

print("Listening to streams. Press CTRL-C to exit.")
qx.App.run()

When I send the "hello, world" message to Kafka, I get the following error from the Kafka consumer:

[23-03-13 16:00:43.516 (4) ERR] Exception while processing package    
System.Runtime.Serialization.SerializationException: Failed to deserialize - the unknown protocol id '104'                        
   at QuixStreams.Transport.Fw.Helpers.TransportPackageValueCodec.Deserialize(Byte[]) + 0x135                                        at QuixStreams.Transport.Fw.DeserializingModifier.Publish(Package, CancellationToken) + 0x1b0                                     at QuixStreams.Transport.TransportConsumer.<>c__DisplayClass5_2.<.ctor>b__8(Package p) + 0x54
   at QuixStreams.Transport.Fw.ByteMergingModifier.<Publish>d__13.MoveNext() + 0x531                                              
--- End of stack trace from previous location ---                                                                                    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x32                                                          at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xe9 
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task) + 0x7b
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task) + 0x2b                                                           at System.Runtime.CompilerServices.TaskAwaiter.GetResult() + 0x1a                                                                 at QuixStreams.Transport.Kafka.KafkaConsumer.<AddMessage>d__66.MoveNext() + 0x333

Asked By: kovac

||

Source

Answer 1

From Quix Docs, it says "Quix Streams serializes and deserializes time-series data using different codecs and optimizations to minimize payloads in order to increase throughput and reduce latency.", so most probably it is not accepting plain text data, have you tried sending data from Quix producer to validate this assumption.

Answered By: Karim Tawfik

Kafka consumer error: failed to deserialize – the unknown protocol id

Question:

Answers: