Persistent Storage

Everything you’ve learned so far rests on a convenient assumption: Pods are interchangeable and disposable. That assumption holds for a stateless API, and it collapses as soon as there is data to keep. A container’s filesystem dies with it: destroy the Pod of a naively deployed PostgreSQL database, and the tables are gone. Not corrupted — gone.

It isn’t enough to “mount a volume.” The difficulty is that in a cluster, disk and compute are separate: a Pod can be rescheduled onto another node, and a local directory on the original node then becomes inaccessible. Storage must therefore be a resource of the cluster, not of the node, and that resource must follow the Pod.

Kubernetes answers with a decoupling into three objects that is confusing at first but has a clear logic. The PersistentVolume represents a piece of real storage. The PersistentVolumeClaim is a request made by the application: “I need 10 GiB of read-write storage.” The StorageClass describes how these volumes are created on demand. The developer writes only the claim; they don’t know — and don’t need to know — whether they’ll get an EBS disk, a Ceph volume, or a local directory.

Then comes a question you can’t dodge: even with a volume that survives, a Deployment remains unsuited to a database. Its Pods are anonymous, created in parallel, and would share the same volume. The StatefulSet exists for this precise case, and this lesson explains why it isn’t just an improved Deployment.

Ephemeral Volumes and Their Proper Use

Before persistent storage, a word about emptyDir, which is often misunderstood. It creates an empty directory when the Pod starts, shared among its containers, and destroyed when the Pod goes away.

apiVersion: v1
kind: Pod
metadata:
  name: pod-avec-cache
spec:
  containers:
    - name: app
      image: mon-api:1.0.0
      volumeMounts:
        - name: cache
          mountPath: /var/cache/app
    - name: collecteur-logs
      image: busybox:1.36
      command: ["sh", "-c", "tail -F /var/cache/app/app.log"]
      volumeMounts:
        - name: cache
          mountPath: /var/cache/app
  volumes:
    - name: cache
      emptyDir:
        sizeLimit: 500Mi

This isn’t a stopgap: it’s the right tool for a cache, a temporary directory, or a scratch space shared between two containers in the same Pod. emptyDir becomes a mistake only when you entrust it with data that must survive. Add medium: Memory for a RAM-backed tmpfs — fast, but counted against the Pod’s memory limit.

PersistentVolume and PersistentVolumeClaim

The Volume, Infrastructure Side

A PersistentVolume is a cluster object, independent of any Pod. It has its own lifecycle.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-donnees-01
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manuel
  hostPath:
    path: /mnt/donnees/pv-01     # local demonstration only

hostPath deserves a frank warning: it points to a directory on the node, so the volume can only be used by a Pod placed on that node. It’s fine for a single-node local cluster to understand the mechanism, and it should be avoided everywhere else. In production, this type field is replaced by a CSI driver that talks to the provider’s real storage.

The Claim, Application Side

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: donnees-app
  namespace: atelier
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: manuel     # must match the targeted PV's class

Kubernetes looks for a compatible PersistentVolume — sufficient capacity, compatible access mode, same storageClassName — and binds them. A PV bound to a PVC is reserved exclusively: no other PVC will get it.

kubectl get pv
kubectl get pvc -n atelier

# STATUS = Pending? The cause is in the events.
kubectl describe pvc donnees-app -n atelier

A PVC stuck in Pending almost always has one of these three causes: no PV with enough capacity, a storageClassName that matches nothing, or an access mode the storage doesn’t support. The events in describe state it explicitly — it’s one of the few places where Kubernetes’s error message is directly actionable.

Using the PVC in a Pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-avec-donnees
spec:
  replicas: 1              # see below: ReadWriteOnce requires 1 replica
  selector:
    matchLabels:
      app: app-donnees
  template:
    metadata:
      labels:
        app: app-donnees
    spec:
      containers:
        - name: app
          image: mon-api:1.0.0
          volumeMounts:
            - name: donnees
              mountPath: /var/lib/app/donnees
      volumes:
        - name: donnees
          persistentVolumeClaim:
            claimName: donnees-app

The container sees an ordinary directory. All the binding machinery is invisible to it, and that is precisely the goal of the decoupling.

Access Modes

This is the field most often misunderstood, because it depends not on Kubernetes but on the underlying storage technology.

What your storage actually supports can be read from existing volumes, never from generic documentation:

kubectl get pv -o custom-columns=NOM:.metadata.name,MODES:.spec.accessModes,CLASSE:.spec.storageClassName

Two decisive clarifications. First, ReadWriteOnce applies to the node, not the Pod: several Pods on the same node can share an RWO volume. It’s ReadWriteOncePod that guarantees Pod-level exclusivity, which is what a database wants.

Second, and this is what trips up beginners: the vast majority of block storage provides only RWO. AWS EBS disks, Google Persistent Disks, and Azure Managed Disks attach to only one machine at a time. ReadWriteMany requires a network filesystem — NFS, AWS EFS, Azure Files, CephFS — often slower and always provisioned separately.

Immediate practical consequence: a Deployment with replicas: 3 and a ReadWriteOnce PVC won’t work reliably. If the Pods land on different nodes, those not on the node that owns the volume stay in Pending or ContainerCreating. This isn’t a Kubernetes quirk; it’s the nature of block storage.

StorageClasses and Dynamic Provisioning

Creating PersistentVolumes by hand doesn’t scale. A StorageClass describes a type of storage, and a provisioner creates volumes on demand, at the moment a PVC arrives.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rapide-ssd
provisioner: kubernetes.io/no-provisioner   # replace with the actual driver
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true

Three fields to understand rather than copy:

• provisioner designates the driver that creates the storage. It is provider-specific: ebs.csi.aws.com on AWS, pd.csi.storage.gke.io on Google Cloud, disk.csi.azure.com on Azure, rancher.io/local-path with kind’s local provisioner.
• volumeBindingMode: WaitForFirstConsumer delays volume creation until a Pod is scheduled. This is almost always the right setting: the volume is then created in the same availability zone as the Pod. With Immediate, a volume can be created in a zone where the Pod can never be placed, and the Pod stays stuck.
• allowVolumeExpansion: true allows the PVC to be enlarged later. Without this field, the only route is backup, new volume, restore.

With a StorageClass, the PVC gets shorter, and no PV is written by hand:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: donnees-postgres
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: rapide-ssd

# Available classes; (default) marks the one used if the field is omitted
kubectl get storageclass

# The PV was created automatically at binding time
kubectl get pv

Let’s be clear: what you get depends entirely on the provider. The same StorageClass doesn’t exist from one cluster to another, a manifest with storageClassName: gp3 fails everywhere except on AWS, and there is no universal name. This is where Kubernetes’s portability ends, and it’s better to know that before promising a multi-cloud deployment.

Omitting storageClassName applies the class marked as default, if the cluster has one. That’s convenient in development, and fragile as soon as a manifest travels.

The Reclaim Policy

It decides the fate of the real storage when the PVC is deleted:

• Delete: the volume and the data are deleted. This is the default for dynamic provisioning.
• Retain: the volume remains with its data, in Released status. You must intervene manually to reuse it.

For any data that matters, set Retain. It’s your only protection against an unfortunate kubectl delete pvc — and your backups, which remain indispensable: Kubernetes backs up nothing.

# Switch an existing PV to Retain
kubectl patch pv pv-donnees-01 \
  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Why a Deployment Is Not Suitable for a Database

Let’s gather the reasons, because this is the most frequent question and it deserves a precise answer.

1. Pods in a Deployment are anonymous. Their name contains a random suffix that changes with every recreation. PostgreSQL replication needs to address a specific instance, stably.
2. They share the same PVC. The Pod template is single, so claimName is the same for all. Three replicas would write to the same volume, which corrupts a database — when the storage allows it, which RWO generally doesn’t.
3. They are created and destroyed in parallel, in an undefined order. A database with a primary and secondaries requires the primary to be ready before the secondaries.
4. Deletion destroys everything. No notion of stable identity survives.

The symptom is immediate on a three-replica Deployment sharing a ReadWriteOnce PVC:

kubectl get pods -l app=postgres
# postgres-6b9f7c4d8-2xk4m   1/1   Running             (on the volume's node)
# postgres-6b9f7c4d8-7fd2p   0/1   ContainerCreating   (volume already attached elsewhere)
# postgres-6b9f7c4d8-q8v1t   0/1   ContainerCreating

The StatefulSet

The StatefulSet answers this list point by point.

apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  clusterIP: None          # "Headless" Service: required by the StatefulSet
  selector:
    app: postgres
  ports:
    - port: 5432
      name: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres    # must refer to the headless Service above
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
              name: postgres
          env:
            - name: POSTGRES_DB
              value: "appdb"
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: DB_USER
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: DB_PASSWORD
            # PostgreSQL requires a subdirectory when the mount point
            # contains a lost+found: without PGDATA, initialization fails.
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: donnees
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "app_user", "-d", "appdb"]
            initialDelaySeconds: 10
            periodSeconds: 5
  volumeClaimTemplates:
    # A separate PVC is created for EACH replica, automatically.
    - metadata:
        name: donnees
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi

What this manifest produces, and what no Deployment produces:

kubectl get pods -l app=postgres
# postgres-0, postgres-1, postgres-2 — stable, ordinal, predictable names

kubectl get pvc
# donnees-postgres-0, donnees-postgres-1, donnees-postgres-2

Four concrete guarantees:

• Stable identity. A recreated postgres-1 is still called postgres-1 and gets donnees-postgres-1 back, with the same data.
• A dedicated volume per replica, thanks to volumeClaimTemplates. No more sharing, no more corruption.
• Guaranteed order. Creation of 0, then 1, then 2, each Pod waiting for the previous one to be ready. Deletion in reverse order.
• Individual DNS. The headless Service gives each Pod a reachable name: postgres-0.postgres.atelier.svc.cluster.local. This is what makes it possible to configure replication.

Two points of caution. First, PVCs created by volumeClaimTemplates are not deleted along with the StatefulSet: this is a deliberate safeguard, and it means a complete cleanup requires explicit deletion.

kubectl delete statefulset postgres
kubectl get pvc                       # the PVCs are still there, data intact
kubectl delete pvc donnees-postgres-0 # explicit deletion if desired

Second, an honest note on operations: a StatefulSet provides a stable identity and volume, nothing more. It doesn’t configure replication, doesn’t handle primary failovers, doesn’t back anything up. For a production database, you use a dedicated operator (CloudNativePG, Zalando Postgres Operator) that provides this logic, or a managed service outside the cluster. The StatefulSet is the foundation, not the complete solution.

Expanding a Volume

If the StorageClass has allowVolumeExpansion: true, you just need to increase the request:

kubectl patch pvc donnees-postgres-0 \
  -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

kubectl get pvc donnees-postgres-0

There are three limitations to be aware of: shrinking a volume is impossible, some drivers require a Pod restart before the file system is expanded, and the operation doesn’t work at all on a manually provisioned PV. Size your volumes with a safety margin from the start.

Key takeaways

• A container’s file system dies with it. emptyDir is meant for caches and for sharing data between containers in a Pod, never for data you need to keep.
• The PV / PVC / StorageClass trio decouples infrastructure from the application: the developer only writes a claim, and the provisioner creates the volume. A PVC stuck in Pending almost always comes down to a nonexistent class or an unsupported access mode.
• ReadWriteOnce applies at the node level, and it is the only mode most block storage offers. ReadWriteMany requires a network file system that you provision separately — a multi-replica Deployment on an RWO PVC won’t work.
• The StorageClass depends on the provider: the provisioner, class names, and reclaim policies vary from one cloud to another. Set Retain on anything that matters, and back it up: Kubernetes doesn’t back up anything.
• A database needs a StatefulSet: stable ordinal names, one PVC per replica via volumeClaimTemplates, ordered startup, and individual DNS through a headless Service. It provides the foundation, not replication or failover — that is the job of an operator or a managed service.

Tip: practice deleting a StatefulSet Pod (kubectl delete pod postgres-1) and verify that it comes back with the same name and the same data. This demonstrates what a Deployment can’t offer.