Deploy Kafka on local kubernetes cluster

Hello everyone, So in this blog we will look into how we can deploy kafka on our local kubernetes cluster for testing and POC.

So to start with kafka, first you need to install the zookeeper into your cluster which kafka uses to store its configuration.

Below is the deployment/service file for deploying zookeeper on kubernetes.

	apiVersion: apps/v1
	kind: Deployment
	metadata:
	labels:
	app: zookeeper
	name: zookeeper
	spec:
	replicas: 1
	selector:
	matchLabels:
	app: zookeeper
	strategy: {}
	template:
	metadata:
	labels:
	app: zookeeper
	spec:
	containers:
	- image: wurstmeister/zookeeper
	name: zookeeper
	resources: {}
	restartPolicy: Always
	---
	apiVersion: v1
	kind: Service
	metadata:
	labels:
	app: zookeeper
	name: zookeeper
	spec:
	ports:
	- name: "client"
	port: 2181
	targetPort: 2181
	selector:
	app: zookeeper

view raw zookeeper.yaml hosted with

by GitHub

This service is deployed using kubectl apply -f zookeeper.yaml

After that you can install the kafka service

	apiVersion: apps/v1
	kind: Deployment
	metadata:
	labels:
	app: kafka
	name: kafka
	spec:
	replicas: 1
	selector:
	matchLabels:
	app: kafka
	strategy: {}
	template:
	metadata:
	labels:
	app: kafka
	spec:
	containers:
	- env:
	- name: KAFKA_ADVERTISED_HOST_NAME
	value: kafka-svc
	- name: KAFKA_ZOOKEEPER_CONNECT
	value: zookeeper:2181
	- name: KAFKA_BROKER_ID
	value: "500"
	image: wurstmeister/kafka
	name: kafka
	ports:
	- containerPort: 9092
	resources: {}
	restartPolicy: Always
	---
	apiVersion: v1
	kind: Service
	metadata:
	labels:
	app: kafka-svc
	name: kafka-svc
	spec:
	type: NodePort
	ports:
	- name: "9092"
	port: 9092
	targetPort: 9092
	nodePort: 31005
	selector:
	app: kafka
	status:
	loadBalancer: {}

view raw kafka.yaml hosted with

by GitHub

Once that is installed, you also might want to install the UI for checking the status about you kafka cluster or just creating/viewing the queues from UI.

for that you can use below service to deploy kafka ui.

	apiVersion: apps/v1
	kind: Deployment
	metadata:
	name: kafka-ui-deployment
	labels:
	app: kafka-ui
	# namespace: mstore
	spec:
	replicas: 1
	selector:
	matchLabels:
	app: kafka-ui
	template:
	metadata:
	labels:
	app: kafka-ui
	spec:
	containers:
	- name: kafka-ui
	image: provectuslabs/kafka-ui:latest
	env:
	- name: KAFKA_CLUSTERS_0_NAME
	value: "K8 Kafka Cluster"
	- name: KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS
	value: kafka-svc:9092
	imagePullPolicy: Always
	resources:
	requests:
	memory: "256Mi"
	cpu: "100m"
	limits:
	memory: "1024Mi"
	cpu: "1000m"
	ports:
	- containerPort: 8080
	---
	apiVersion: v1
	kind: Service
	metadata:
	name: kafka-ui-service
	spec:
	selector:
	app: kafka-ui
	type: NodePort
	ports:
	- protocol: TCP
	port: 8080
	targetPort: 8080
	nodePort: 31006

view raw kafka-ui.yaml hosted with

by GitHub

with that setup, now you can access the kafka ui service on your local machine on localhost:31006

Github Link : local-kafka

Thanks for reading…

Deploy Kafka on local kubernetes cluster

Published by Ashish Maurya

Leave a Reply Cancel reply