Back Up Redis to S3 and Restore It (Even With AOF On)

Persistence protects restarts, backups protect data

Redis has two persistence modes. RDB writes a compact snapshot of the whole dataset at intervals. AOF (append-only file) logs every write and replays the log on start, so at most a second of writes is lost with the default appendfsync everysec. Both protect you when the process or pod restarts.

Neither protects you when the disk goes: a deleted PersistentVolume, a lost EBS volume, a FLUSHALL that reaches the AOF too, an account mistake. For that you need a copy elsewhere, taken on a schedule, and a restore you have actually run.

Take a snapshot with redis-cli --rdb

redis-cli --rdb connects as a replica would, asks for a full sync and writes the RDB it receives to a local file. It works against any server you can reach and authenticate to, including a managed one, and does not need a shell in the Redis container.

export REDISCLI_AUTH='<password>'   # keeps the password off the command line
redis-cli -h redis.cache.svc.cluster.local -p 6379 --rdb /tmp/dump.rdb
gzip /tmp/dump.rdb                   # RDB compresses well; smaller uploads and storage

The server forks to produce the snapshot, so it briefly needs extra memory for pages written during the copy. That is one reason to set maxmemory below the container’s memory limit.

A nightly CronJob that uploads to S3

Use two containers sharing an emptyDir: the Redis image takes and compresses the snapshot, the AWS CLI image uploads it. The pod’s service account should have an IAM role (IRSA or EKS Pod Identity) that can write only to the backup prefix.

apiVersion: batch/v1
kind: CronJob
metadata: {name: redis-backup, namespace: cache}
spec:
  schedule: "0 3 * * *"          # 03:00 UTC
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          serviceAccountName: redis-backup   # IAM role: s3:PutObject on the prefix
          volumes: [{name: work, emptyDir: {}}]
          initContainers:
          - name: dump
            image: valkey/valkey:8.1
            env:
            - name: REDISCLI_AUTH
              valueFrom: {secretKeyRef: {name: redis-auth, key: password}}
            command: [sh, -c, "redis-cli -h redis -p 6379 --rdb /work/dump.rdb && gzip /work/dump.rdb"]
            volumeMounts: [{name: work, mountPath: /work}]
          containers:
          - name: upload
            image: public.ecr.aws/aws-cli/aws-cli:2.27.0
            command: [sh, -c]
            args:
            - aws s3 cp /work/dump.rdb.gz "s3://my-redis-backups/cache/$(date -u +%Y%m%dT%H%M%SZ).rdb.gz"
                --sse aws:kms --only-show-errors
            volumeMounts: [{name: work, mountPath: /work}]

Retention and encryption

• Add an S3 lifecycle rule that expires objects under the prefix after N days, or list and delete old keys at the end of the job.
• --sse aws:kms uses the bucket’s KMS key; add --sse-kms-key-id for a specific one. The role then also needs kms:GenerateDataKey on that key.
• Write a small status object (last success time, size) next to the backups and alert when it goes stale. A backup job that has been failing silently for a month is the usual discovery during an incident.

Why copying dump.rdb restores nothing

Most restore guides say: stop Redis, copy the backup to dump.rdb in the data directory, start Redis. That works only with AOF off. The Redis persistence documentation is explicit that when AOF is enabled, Redis uses the AOF to rebuild the dataset on start, because it is the more complete of the two.

Since Redis 7, the AOF is a directory (appendonlydir) with a base file, incremental files and a manifest. If you move that directory aside to start fresh, a server with appendonly yes creates a new, empty AOF and starts with zero keys. Your dump.rdb sits next to it unread. We hit exactly this while testing restores on Redis 8.2 and Valkey 8.1: the pod came up healthy and empty.

The fix is to install the snapshot as the base of a new multi-part AOF. A base file may be in RDB format, so the backup goes in unchanged, plus a one-line manifest that points at it.

Restore script for Kubernetes (tested)

This assumes a StatefulSet named $DEP with one replica, its data on the PVC data-$DEP-0 mounted at /data, and the server running as UID 999 (the Redis and Valkey images’ default). Everything written after the backup is lost.

NS=cache; DEP=redis; SRC=s3://my-redis-backups/cache/<file>.rdb.gz
SA=redis-backup   # a service account that can read the backup

kubectl -n $NS scale statefulset $DEP --replicas=0
kubectl -n $NS wait --for=delete pod/$DEP-0 --timeout=180s

kubectl -n $NS apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata: {name: $DEP-restore}
spec:
  restartPolicy: Never
  serviceAccountName: $SA
  volumes: [{name: d, persistentVolumeClaim: {claimName: data-$DEP-0}}]
  initContainers:
  - name: download
    image: public.ecr.aws/aws-cli/aws-cli:2.27.0
    args: [s3, cp, "$SRC", /data/restore.rdb.gz]
    volumeMounts: [{name: d, mountPath: /data}]
  containers:
  - name: restore
    image: busybox:1.37
    volumeMounts: [{name: d, mountPath: /data}]
    command:
    - sh
    - -c
    - |
      set -e
      gunzip -t /data/restore.rdb.gz
      mv /data/appendonlydir /data/appendonlydir.old-$(date +%s)
      mkdir /data/appendonlydir
      gunzip -c /data/restore.rdb.gz > /data/appendonlydir/appendonly.aof.1.base.rdb
      printf 'file appendonly.aof.1.base.rdb seq 1 type b\n' > /data/appendonlydir/appendonly.aof.manifest
      chown -R 999:999 /data/appendonlydir
      chmod 700 /data/appendonlydir
      rm /data/restore.rdb.gz
      echo restore-ok
EOF
kubectl -n $NS wait --for=jsonpath='{.status.phase}'=Succeeded pod/$DEP-restore --timeout=300s
kubectl -n $NS logs $DEP-restore          # restore-ok
kubectl -n $NS delete pod $DEP-restore

kubectl -n $NS scale statefulset $DEP --replicas=1
kubectl -n $NS wait --for=condition=ready pod/$DEP-0 --timeout=300s
kubectl -n $NS logs $DEP-0 | grep "keys loaded"   # e.g. Done loading RDB, keys loaded: 3

Three details that break other versions of this

• The default service account usually has no S3 access. Use the one your backup job runs as.
• The AWS CLI image has no gunzip, so the download and the unpacking run in different containers.
• The new directory must belong to the Redis user (UID 999). The restore pod runs as root, and the server must be able to write new AOF files into that directory.

Test the restore before you need it

Restore last night’s backup into a scratch instance once a month and compare DBSIZE and a few known keys with production. It takes ten minutes and is the only proof that your backups contain what you think they do. A restore you have never run is a hope, not a plan.

Backups built in: the SeaGit Redis Stack

SeaGit’s Redis Stack deploys Valkey or Redis into a Kubernetes cluster in your own AWS account and sets up this backup chain for you: a snapshot right after deploy, one every night at 03:00 UTC kept for the number of days you choose, and a final one before the stack is deleted. Backups go to a new S3 bucket or a folder of one you already own, in the same account, encrypted with S3-managed keys or KMS. Inside the cluster, only the stack’s own service account has an IAM role for that bucket or folder.

The S3 location is shown on the Stack tab, and the restore procedure above is the one documented and tested for it. See Deploy Redis in Minutes to set one up, or Redis use cases for which data needs backups at all.

Frequently asked questions