Persistent Volumes and StatefulSet in Kubernetes — Storage That Survives Pod Crashes

Your PostgreSQL pod in Kubernetes just disappeared with all your data? Or worse, you lost records because the volume was tied to the container lifecycle? It happens more often than you think. At Meteora Web, we see this in projects that come to us after a disaster: ephemeral volumes, misconfigured claims, StatefulSets used as Deployments. The difference between an infrastructure that holds and one that collapses at the first crash is here: Persistent Volumes and StatefulSet. If you manage stateful applications in production, this guide will help you avoid the expensive mistakes we fix during recovery.

Why is storage in Kubernetes different from a single server?

On a physical server or VM, the disk is there. You mount, write, reboot: data stays. In Kubernetes, pods are ephemeral. They come and go, replicate, and reschedule on different nodes. If a pod writes to a local filesystem and then gets recreated on another node, the data disappears. That's why you need PersistentVolume (PV) and PersistentVolumeClaim (PVC): they separate the declaration of storage need from the physical resource.

With a PVC you declare: "I need 10 GiB of storage with ReadWriteOnce access." Kubernetes finds a matching PV and binds it to your pod. If the pod dies and is reborn on another node, the PVC re-attaches to the same PV — data survives. Sounds simple, but the details (storage classes, reclaim policies, access modes) can mean the difference between a working setup and one that loses data on the first scale-down.

How do PersistentVolume and PersistentVolumeClaim work in practice?

Let's break down the concepts. A PV is a cluster resource: a piece of storage that is pre-provisioned or dynamic. It has a storageClassName, a capacity, an access mode (ReadWriteOnce, ReadOnlyMany, ReadWriteMany), and a reclaim policy: Retain, Delete, Recycle. Retain keeps the volume even after the PVC is deleted — very useful for disaster recovery. Delete removes the volume with the PVC — careful with sensitive data.

A PVC is the request. Example YAML for a MySQL database PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard

Kubernetes looks for a matching PV. If the StorageClass is dynamic (e.g., provisioner: kubernetes.io/aws-ebs), the PV is created automatically. This is the most common method in production: you declare the need, and the cluster provisions it.

Common mistakes to avoid with PV and PVC

• Wrong AccessMode: a clustered database with ReadWriteOnce won't work across multiple replicas. Use ReadWriteMany on NFS volumes, but watch out for performance.
• ReclaimPolicy Delete on critical volumes: if you accidentally delete the PVC, the volume is gone. Choose Retain for production data.
• No StorageClass specified: if you omit storageClassName, Kubernetes uses the default — which might be slow or non-persistent. Always verify.

Why is StatefulSet mandatory for stateful applications?

A Deployment creates identical, interchangeable pods. If a pod dies, a new one gets a different name and IP. For stateful applications (databases, message queues, systems that need stable network identity), that's a problem. StatefulSet guarantees:

• Stable identity: each pod gets an ordinal name (mysql-0, mysql-1) and a fixed hostname.
• Persistent storage: each pod has its own PVC created from a template, permanently tied to that pod.
• Ordered startup/shutdown: pods start in sequence — ideal for clusters that need a master before replicas.

We've seen teams use a Deployment for a database because "it's not production." Result: after a rolling update, the new pod couldn't find its data because the volume was attached to the old pod. With StatefulSet and volumeClaimTemplates, each pod gets a unique volume and always reattaches to the same volume – even after a crash or update.

How to configure a PVC in a StatefulSet — complete example

Here's a StatefulSet for MySQL with persistent storage. Note volumeClaimTemplates, not a simple volume mount:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "password"
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: standard
      resources:
        requests:
          storage: 10Gi

Kubernetes creates a PVC per replica: data-mysql-0, data-mysql-1… and pod names remain fixed. If pod mysql-0 is deleted and recreated, it gets the same PVC and therefore the same data. Perfect for databases.

Handling ReadWriteOnce in multi‑pod clusters

If you have a MySQL cluster with replicas that need to read from the same data, ReadWriteOnce isn't enough — each replica gets its own volume. Possible solutions:

• Use a natively distributed database (Galera, Vitess) that handles replication at the application layer.
• Use an NFS volume with ReadWriteMany, but be careful: NFS is not performant for synchronous writes.
• Deploy an operator like MySQL Operator that automatically creates separate volumes for each node.

The choice depends on your workload and budget. At Meteora Web we always recommend testing with real loads before going to production.

Which storage back-end to choose for production Kubernetes?

Not all CSI drivers are equal. Here are the most common ones with pros and cons:

• Cloud providers (EBS, GCE Persistent Disk, AzureDisk): performant, resilient, but tied to a single availability zone. Good for transactional workloads.
• Longhorn (Rancher): open source, uses local node disks and replicates across nodes. Great for on‑premise clusters.
• Rook/Ceph: powerful but complex. Ideal for large clusters needing object and block storage together.
• Local Persistent Volume: uses the node's local disk. Very fast, but if the node dies the volume becomes unreachable. Suitable for etcd on dedicated nodes.

We've configured Longhorn on a bare‑metal cluster for a client who wanted to avoid cloud lock‑in. It works, but you must monitor replication status and have a failover plan ready.

How to handle backup, snapshot, resize, and disaster recovery of volumes?

A persistent PV is not a backup. If an admin accidentally deletes the PVC with a Delete reclaim policy, data is gone. Here are practices we apply in our projects:

• CSI snapshots: every CSI driver supports VolumeSnapshot. Create periodic snapshots (e.g., every hour) and keep them for 7 days. Example for Longhorn: kubectl apply -f volume-snapshot.yaml.
• External backup with Velero: an open‑source tool that exports PVs to object storage (S3, MinIO). It allows full cluster restore. We use it for disaster recovery.
• Online resize: if you need more space, edit the PVC and make sure the storage class supports expansion. Example: kubectl edit pvc mysql-data-0 -n database and increase storage. Works if the CSI driver supports it.
• Disaster recovery: keep your manifests in Git (GitOps) and use Velero to back up PVs to a second cluster or cloud region.

A client e‑commerce shop was losing snapshots because the default CSI driver had no retention. We added a cron job that creates snapshots with a retain: 7d label and deletes them after one week.

What to do now

Now you have the foundations to avoid data loss on Kubernetes. Here are three immediate actions:

1. Check the reclaim policy of your current PVs. If it's Delete, consider switching to Retain for critical volumes. Test in a staging environment first.
2. Replace Deployments of stateful applications with StatefulSet. Use volumeClaimTemplates and verify that PVCs are named correctly.
3. Implement a backup tool like Velero or CSI snapshots. You need it to sleep soundly. Don't wait for the first crash.

If you want to dive deeper into full container orchestration for production, read our pillar guide on Kubernetes (Italian). For the rest, we are here: we work with code and territory every day. Theory helps, but practice saves data.