Elastic On K8S

Elasticsearch (Elastic) is a powerful distributed search and analytics engine, and running it on Kubernetes offers scalability, flexibility, and simplified management. This blog explores how to deploy Elasticsearch on Kubernetes, discusses its benefits, and provides best practices for optimizing the setup.

Why Run Elasticsearch on Kubernetes?

Benefits:

• Scalability: Easily scale Elasticsearch nodes based on workload.
• High Availability: Kubernetes ensures pods are rescheduled on failures.
• Resource Management: Fine-grained control over CPU, memory, and storage resources.
• Integration: Native support for other Kubernetes-hosted applications and services.

Prerequisites

1. Kubernetes Cluster:
• At least three nodes for high availability.
• kubectl CLI installed and configured.
2. Storage Solution:
• Persistent volumes provisioned for Elasticsearch data.
• Dynamic provisioning (e.g., using CSI drivers) or static PersistentVolume setup.
3. Elastic Helm Chart (optional):
• Elasticsearch can be deployed using the Elastic-provided Helm charts for streamlined configuration.

Deploying Elasticsearch on Kubernetes

Step 1: Create a Namespace

Organize Elasticsearch resources within a dedicated namespace:

kubectl create namespace elasticsearch

Step 2: Deploy Elasticsearch Using YAML

Create a StatefulSet for Elasticsearch to ensure stable network identities and persistent storage.

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: elasticsearch

  namespace: elasticsearch

spec:

  serviceName: elasticsearch

  replicas: 3

  selector:

    matchLabels:

      app: elasticsearch

  template:

    metadata:

      labels:

        app: elasticsearch

    spec:

      containers:

      - name: elasticsearch

        image: docker.elastic.co/elasticsearch/elasticsearch:8.10.0

        resources:

          requests:

            memory: 1Gi

            cpu: 500m

          limits:

            memory: 2Gi

            cpu: 1

        ports:

        - containerPort: 9200

          name: http

        - containerPort: 9300

          name: transport

        volumeMounts:

        - name: elasticsearch-data

          mountPath: /usr/share/elasticsearch/data

      volumes:

        - name: elasticsearch-data

          persistentVolumeClaim:

            claimName: elasticsearch-pvc

---

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: elasticsearch-pvc

  namespace: elasticsearch

spec:

  accessModes:

    - ReadWriteOnce

  resources:

    requests:

      storage: 10Gi

Apply the YAML:

Step 3: Verify Deployment

Check the pods and services:

kubectl get pods -n elasticsearch
kubectl get svc -n elasticsearch

Optimizing Elasticsearch on Kubernetes

1. Resource Allocation

• Set CPU and memory requests/limits to ensure Elasticsearch pods run reliably.
• Use node affinity or taints to schedule pods on dedicated high-memory nodes.

2. Persistent Storage

• Use SSD-backed storage for better I/O performance.
• Ensure data volumes are backed up regularly.

3. Monitoring and Logging

• Use tools like Kibana and Prometheus to monitor cluster health.
• Integrate Elasticsearch logs into centralized logging systems.

4. Networking

• Use a dedicated load balancer or ingress controller to expose Elasticsearch services securely.
• Implement network policies to restrict access.

Scaling Elasticsearch

Horizontal Scaling:

Increase the replicas count in the StatefulSet YAML to add more nodes:

spec:
  replicas: 5

Vertical Scaling:

Modify resource requests and limits for more powerful nodes:

resources:

  requests:

    memory: 2Gi

    cpu: 1

  limits:

    memory: 4Gi

    cpu: 2

Security Best Practices

1. Enable Transport Layer Security (TLS):
• Use Elastic’s official documentation to configure TLS for node-to-node and client communication.
2. Set Strong Authentication:
• Enable Basic Authentication or use API keys.
3. Limit Pod Privileges:
• Use Kubernetes PodSecurityPolicies to enforce least privilege.
4. Restrict External Access:
• Expose services only through a secure ingress or VPN.

Conclusion

Running Elasticsearch on Kubernetes offers unmatched flexibility and scalability for modern applications. By following the steps and best practices outlined in this guide, you can deploy, manage, and secure an Elasticsearch cluster optimized for your workload.

Have questions or insights? Share them in the comments below!