This project provides a Golang implementation of the Wikimedia project as discussed by Stephane Maarek in his Kafka beginners course. The original Java consumer and producer repositories can be found here:

Prerequisites

Before you begin, ensure the following software is installed on your system:

Running the Project

1. Start Kafka in Kraft Mode

Start Kafka by executing the following command in your terminal:

docker-compose -f docker-compose-kafka-kraft.yml up --build

2. Start OpenSearch

Open a new terminal window and execute:

docker-compose -f docker-compose-opensearch.yml up --build

3. Create Kafka Topic

Once Kafka is up, create the Kafka topic with the following command:

kafka-topics.sh --bootstrap-server localhost:9092 --topic wikimedia.recentchange --create --partitions 3 --replication-factor 2

4. Start Wikimedia Kafka Producer

Navigate to the producer-wikimedia directory in a new terminal window and run:

cd producer-wikimedia
go run ./producer.go

5. Start OpenSearch Kafka Consumer

Navigate to the consumer-opensearch directory in another terminal window and run:

cd consumer-opensearch
go run ./consumer.go

Viewing Consumed Data in OpenSearch Dashboard

To view the consumed data in the OpenSearch dashboard, go to OpenSearch Dev Tools.

Username: admin
Password: Toor1234_

Retrieve Consumed Data

Execute the following command to retrieve data consumed and sent to OpenSearch by the Kafka consumer:

GET /wikimedia/_search