In this tutorial, we are going to learn how to build simple Kafka Consumer in Java. We will understand properties that we need to set while creating Consumer and how to handle topic offset to read messages from the beginning of topic or just latest messages.

Prerequisite

For building Kafka Consumer, We need to have one or more topic present in Kafka server. That topic should have some messages published already, or some Kafka producer is going to publish messages to that topic when we are going to read those messages from Consumer. You can learn how to create a topic in Kafka here and how to write Kafka Producer here.

Kafka Consumer in Java

Let us see how we can write Kafka Consumer now. In the following code, we can see essential imports and properties that we need to set while creating Consumer.

We need to pass bootstrap server details so that Consumer can connect to Kafka server. In this example, we are reading from the topic which has Keys and Messages in String format. So we need to use String Deserializer for reading Keays and messages from that topic. We need to send group name for that consumer. If Consumer group has more than one consumer, then they can read messages in parallel from the topic. You can learn more about Kafka consumers here.

We need to tell Kafka from which point we want to read messages from that topic. For this purpose, we are passing offset reset property. Setting it to earliest means Consumer will start reading messages from the beginning of that topic.

Reading messages

We need to create consumer record for reading messages from the topic. We can use following code to keep on reading from the consumer.

Here we are reading from the topic and displaying value, key and partition of each message. We are using ‘poll’ method of Kafka Consumer which will make consumer wait for 1000 milliseconds if there are no messages in the queue to read. If there are messages, it will return immediately with the new message. We can start another consumer with same group id and they will read messages from different partitions of the topic in parallel. you can get all this code at git repository

Conclusion

We have learned how to build Kafka consumer and read messages from the topic using Java language. In last few articles, we have seen how to create the topic, Build Producer, send messages to that topic and read those messages from the Consumer. In future, we will learn more use cases of Kafka. Till then, happy learning !!!

I am passionate about data analytics, machine learning, and artificial intelligence. Recently I have started blogging about my experience while learning these exciting technologies.

Leave a Reply

Close Menu