Monitoring and Logging in Kubernetes: Tools and Strategies

Introduction

In a Kubernetes environment, visibility is crucial for maintaining the health and performance of your applications. Monitoring and logging are two fundamental aspects of observability that enable you to detect problems, understand their root cause, and take preventive action.

This article explores the tools and strategies for setting up a complete monitoring and logging system in Kubernetes.

Monitoring in Kubernetes

Monitoring in Kubernetes involves collecting, analyzing, and visualizing metrics about the state of your cluster and your applications. Here are the main approaches and tools.

Basic Metrics with kubectl

Kubernetes provides basic commands to monitor the state of your cluster:

# Vérifier l'état des nœuds
kubectl get nodes

# Vérifier l'état des pods
kubectl get pods

# Vérifier l'état des services
kubectl get services

# Vérifier les événements du cluster
kubectl get events

# Vérifier l'utilisation des ressources des nœuds
kubectl top nodes

# Vérifier l'utilisation des ressources des pods
kubectl top pods

Kubernetes Metrics with Metrics Server

Metrics Server is a component that collects basic Kubernetes metrics (CPU, memory) and exposes them through the Kubernetes API.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-delegator
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: Metrics-server
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server/metrics-server:v0.5.0
        args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls
        ports:
        - name: https
          containerPort: 4443
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
          limits:
            cpu: 200m
            memory: 400Mi
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
      volumes:
      - name: tmp-dir
        emptyDir: {}

Prometheus: Advanced Metrics Collection

Prometheus is an open-source monitoring system that collects and stores metrics as time series. It is particularly well suited to cloud-native environments.

Installing Prometheus with Helm

# Ajouter le repo Helm de Prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Installer Prometheus
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

Configuring Prometheus for Kubernetes

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-monitor
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: http
    interval: 15s
    path: /metrics

Grafana: Metrics Visualization

Grafana is a visualization tool that lets you build interactive dashboards for your Prometheus metrics.

Installing Grafana with Helm

# Ajouter le repo Helm de Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Installer Grafana
helm install grafana grafana/grafana \
  --namespace monitoring \
  --create-namespace

Grafana Dashboard Example

{
  "annotations": {
    "list": []
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"default\"}[5m])) by (pod) * 100",
          "refId": "A"
        }
      ],
      "title": "CPU Usage by Pod",
      "type": "timeseries"
    }
  ],
  "refresh": "5s",
  "schemaVersion": 38,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Kubernetes Monitoring",
  "uid": "kubernetes",
  "version": 1,
  "weekStart": ""
}

Alertmanager: Alert Management

Alertmanager handles the alerts sent by Prometheus and routes them to various notification channels.

Configuring Alertmanager

apiVersion: monitoring.coreos.com/v1
kind: AlertmanagerConfig
metadata:
  name: app-alerts
  namespace: default
spec:
  route:
    receiver: 'slack'
    groupBy: ['alertname', 'namespace']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
  receivers:
  - name: 'slack'
    slackConfigs:
    - apiURL: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
      channel: '#alerts'
      sendResolved: true

Logging in Kubernetes

Logging in Kubernetes involves collecting, storing, and analyzing the logs of your applications and the system. Here are the main approaches and tools.

Basic Logging with kubectl

Kubernetes provides basic commands to access the logs of your pods:

# Voir les logs d'un pod
kubectl logs <pod-name>

# Voir les logs d'un conteneur spécifique dans un pod
kubectl logs <pod-name> -c <container-name>

# Voir les logs en temps réel
kubectl logs -f <pod-name>

# Voir les logs d'un pod précédent (en cas de redémarrage)
kubectl logs <pod-name> --previous

Centralized Logging Architecture

For a complete logging solution, you can set up a centralized architecture with the following components:

1. Log collector: Collects the logs from all pods
2. Storage: Stores the logs for later analysis
3. Visualization interface: Lets you search and analyze the logs

ELK Stack: Elasticsearch, Logstash, Kibana

The ELK Stack is a popular solution for collecting, storing, and analyzing logs.

Installing the ELK Stack with Helm

# Ajouter le repo Helm d'Elastic
helm repo add elastic https://helm.elastic.co
helm repo update

# Installer Elasticsearch
helm install elasticsearch elastic/elasticsearch \
  --namespace logging \
  --create-namespace

# Installer Kibana
helm install kibana elastic/kibana \
  --namespace logging \
  --create-namespace

# Installer Logstash
helm install logstash elastic/logstash \
  --namespace logging \
  --create-namespace

Configuring Filebeat for Log Collection

Filebeat is a lightweight agent that collects logs and sends them to Elasticsearch or Logstash.

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Filebeat
metadata:
  name: filebeat
  namespace: logging
spec:
  version: 7.14.0
  elasticsearchRef:
    name: elasticsearch
  config:
    filebeat.inputs:
    - type: container
      paths:
      - /var/log/containers/*.log
    processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
  daemonSet:
    podTemplate:
      spec:
        containers:
        - name: filebeat
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

EFK Stack: Elasticsearch, Fluentd, Kibana

An alternative to the ELK Stack is the EFK Stack, which uses Fluentd instead of Logstash.

Installing the EFK Stack with Helm

# Ajouter le repo Helm de Fluentd
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

# Installer Fluentd
helm install fluentd fluent/fluentd \
  --namespace logging \
  --create-namespace

Configuring Fluentd

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>

    <match kubernetes.**>
      @type elasticsearch
      host elasticsearch
      port 9200
      logstash_format true
      logstash_prefix k8s
      <buffer>
        @type memory
        flush_interval 5s
        retry_max_times 5
      </buffer>
    </match>

Loki: A Lightweight Logging Solution

Loki is a lightweight logging solution developed by Grafana Labs, designed to be highly efficient and easy to use.

Installing Loki with Helm

# Ajouter le repo Helm de Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Installer Loki
helm install loki grafana/loki-stack \
  --namespace logging \
  --create-namespace

Configuring Promtail

Promtail is the log collection agent for Loki.

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
  namespace: logging
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    positions:
      filename: /tmp/positions.yaml

    clients:
      - url: http://loki:3100/loki/api/v1/push

    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_name]
            regex: (.+)
            target_label: pod
          - source_labels: [__meta_kubernetes_namespace_name]
            regex: (.+)
            target_label: namespace
        pipeline_stages:
          - docker: {}
          - cri: {}

Best Practices for Monitoring and Logging

1. Application-Level Metrics Collection

Instrument your applications to expose metrics in Prometheus format:

// Example with Micrometer for Java
@RestController
public class MetricsController {
    private final Counter requestCounter;
    
    public MetricsController(MeterRegistry registry) {
        this.requestCounter = Counter.builder("app_requests_total")
            .description("Total number of requests")
            .register(registry);
    }
    
    @GetMapping("/api/resource")
    public String handleRequest() {
        requestCounter.increment();
        return "Response";
    }
}

2. Log Structuring

Use a structured format (JSON) for your logs to make analysis easier:

// Example with SLF4J and Logback
<configuration>
  <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <includeMDC>true</includeMDC>
      <customFields>{"app_name":"myapp"}</customFields>
    </encoder>
  </appender>
  <root level="INFO">
    <appender-ref ref="JSON" />
  </root>
</configuration>

3. Labeling Metrics and Logs

Add relevant labels to your metrics and logs to make filtering and analysis easier:

# Example Prometheus configuration
scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_name]
        regex: (.+)
        target_label: pod
      - source_labels: [__meta_kubernetes_namespace_name]
        regex: (.+)
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: (.+)
        target_label: app

4. Defining Meaningful Alerts

Define alerts that detect real problems without generating false positives:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: app-alerts
  namespace: monitoring
spec:
  groups:
  - name: app
    rules:
    - alert: HighErrorRate
      expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: High error rate detected
        description: Error rate is above 5% for the last 5 minutes

5. Retention and Archiving

Configure a retention and archiving strategy for your metrics and logs:

# Example Elasticsearch configuration for retention
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 7.14.0
  nodeSets:
  - name: default
    count: 3
    volumeClaimTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
  indices:
    lifecycle:
      policies:
      - name: logs-policy
        phases:
          hot:
            min_age: 0s
            actions:
              rollover:
                max_age: 1d
                max_size: 50gb
          warm:
            min_age: 7d
            actions:
              forcemerge:
                max_num_segments: 1
          cold:
            min_age: 30d
            actions:
              freeze: {}
          delete:
            min_age: 90d
            actions:
              delete: {}

Complete Example: Monitoring and Logging Stack

Here is a complete example of a configuration for a monitoring and logging stack in Kubernetes:

# Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      release: prometheus
  resources:
    requests:
      memory: 2Gi
      cpu: 1
    limits:
      memory: 4Gi
      cpu: 2
  storage:
    volumeClaimTemplate:
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi
---
# Grafana
apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: grafana
  namespace: monitoring
spec:
  config:
    auth:
      adminUser: admin
      adminPassword: admin
    dashboard:
      defaultDashboard: kubernetes
  dashboardLabelSelector:
    - matchLabels:
        grafana_dashboard: "1"
  resources:
    requests:
      memory: 1Gi
      cpu: 500m
    limits:
      memory: 2Gi
      cpu: 1
---
# Loki
apiVersion: loki.grafana.com/v1
kind: Loki
metadata:
  name: loki
  namespace: logging
spec:
  size: 1x.small
  storage:
    type: filesystem
    size: 10Gi
  resources:
    requests:
      memory: 512Mi
      cpu: 250m
    limits:
      memory: 1Gi
      cpu: 500m
---
# Promtail
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
  namespace: logging
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    positions:
      filename: /tmp/positions.yaml

    clients:
      - url: http://loki:3100/loki/api/v1/push

    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_name]
            regex: (.+)
            target_label: pod
          - source_labels: [__meta_kubernetes_namespace_name]
            regex: (.+)
            target_label: namespace
        pipeline_stages:
          - docker: {}
          - cri: {}
---
# ServiceMonitor for the application
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-monitor
  namespace: monitoring
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: http
    interval: 15s
    path: /metrics
---
# Service for the application with metrics
apiVersion: v1
kind: Service
metadata:
  name: app-service
  labels:
    app: myapp
spec:
  selector:
    app: myapp
  ports:
  - name: http
    port: 80
    targetPort: 8080
  - name: metrics
    port: 8080
    targetPort: 8080

Conclusion

A robust monitoring and logging system is essential for maintaining the health and performance of your applications in Kubernetes. By combining tools like Prometheus, Grafana, and Loki, you can get complete visibility into your cluster and your applications.

Best practices such as collecting application-level metrics, structuring your logs, and defining meaningful alerts will help you detect and resolve problems before they affect your users.

In the next article, we will explore security in Kubernetes, a crucial aspect of protecting your applications and your data.