Simplified Kafka concept and example with nodejs

Introduction

Kafka is a distributed-event store and stream-processing platform supported by Apache.

How it works

Kafka improves the traditional server-client mechanism by putting a layer between server and client which acts as a connector storing the events, produced from the servers(producers) through the specified topic, and serving the event stream to the client(consumer).

Prerequisites

• Need to have docker application on your machine
• Need to have an intermediate knowledge of how nodejs application works

Let's start coding

In this article, we are gonna use ready-made docker compose file, which setup for kafka service, To create a kafka-node application, follow the steps described below.

1. Open docker application
2. Create a new file named docker-compose.yml in your project directory and copy the following codes into the file

   version: "3"
   services:
   zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
   kafka:
    image: 'bitnami/kafka:latest'
    container_name: 'kafka'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper
   

    
3. Run the following command in the directory same as the docker file to start the kafka

   docker-compose up
   

    
4. Run the following commands to initialize the node project

   npm init
   

    
5. Run this to install the node package which will let you connect to kafka server in your node application

   npm install node-rdkafka
   

    

6. For producer and consumer, create index.js files in their respective folder

   

7. You can emit events from producer by the following code,

    import Kafka from 'node-rdkafka';
    console.log('Producer ...');
    const stream = Kafka.Producer.createWriteStream({
        "metadata.broker.list": "localhost:9092"
        },
        {},
        { topic: 'test' }
    )
    function queueMessage()
    {
        const message = 'Hello world'
        const success = stream.write(Buffer.from(message))
        if (success)
        {
            console.log('success');
        }
        else 
        {
            console.log('something went wrong!')
        }
    }
    setInterval(() => {
        queueMessage()
    }, 3000)
   

    

8. You can subscribe the specified topic and get the sequence of events streamed from the producers as follow,

    console.log('Comsumer ...')
    import Kafka from 'node-rdkafka';
   
    const consumer = Kafka.KafkaConsumer({
        "group.id": "kafka",
        "metadata.broker.list": "localhost:9092"
        },
        {}
    );
   
    consumer.connect();
   
    consumer.on('ready', () => {
        console.log('consumer ready . . . ')
        consumer.subscribe(['test']);
        consumer.consume();
    }).on('data', (data) => {
        const buf = Buffer.from(data.value);
        console.log('Message received: ', buf.toString());
    });
   

    

9. We need to create commands in the package.json file to run the producer and consumer.

    "scripts": {
        "start:producer": "node producer/index.js",
        "start:consumer": "node consumer/index.js"
    },
   

   Add the above lines inside the scripts block.  
10. Finally, we can run both producer and consumer by enter the following lines into the console.

    npm run start:producer
    

    
    

    npm run start:consumer
    

You can see the message emitted by producer inside the consumer console.

Conclusion

Producer can emit events to the kafka server through the topic. Theses events are persistant inside the server and queued like a log file. When consumer make a request with the certain topic, kafka server response all the queued event reside in it back to consumer.

Reminder

Please note that this is the simplified version of how kafka works. Hope you will have the overview of Kafka.