User Tools

Site Tools


runbooks:backup:ghost

Ghost Backup CronJob

Meaning

These alerts monitor the Kubernetes Job created by the Ghost backup CronJob.

Alerts:

  • `GhostBackupSucceeded` → Backup Job succeeded; MySQL database and Ghost content folders were archived and uploaded to MinIO.
  • `GhostBackupFailed` → Backup Job failed; database dump or content archive was not completed or failed to upload.

Impact

* Success → Backup completed successfully. Data is safely stored in MinIO. * Failure → Ghost website or database backup may be missing. Could affect disaster recovery if restoration is needed.

Diagnosis

1. Check Kubernetes Job status:

kubectl get job ghost-backup-job -n <NAMESPACE>
kubectl describe job ghost-backup-job -n <NAMESPACE>

2. Check logs of the Job pod:

kubectl logs job/ghost-backup-job -n <NAMESPACE>

3. Verify backup in MinIO:

mc ls <MINIO_ALIAS>/ghost-backups/
mc stat <MINIO_ALIAS>/ghost-backups/<backup_file>

4. Check PVC mounts if used:

kubectl get pvc -n <NAMESPACE>
kubectl describe pvc <PVC_NAME> -n <NAMESPACE>

Possible Causes of Failure

* Pod in CrashLoopBackOff, OOMKilled, or Failed * PVC mount unavailable or insufficient space * MinIO credentials missing or misconfigured * Network issues preventing upload to MinIO * Disk space or permissions issues on the node * CronJob manifest misconfiguration

Mitigation

1. Inspect Job pod logs to identify errors. 2. Verify MinIO credentials and connectivity. 3. Check PVC status and node disk availability. 4. Retry backup manually if needed:

kubectl create job --from=cronjob/ghost-backup-job ghost-backup-job-manual -n <NAMESPACE>

5. Correct any misconfigurations in CronJob YAML or MinIO bucket policy. 6. Escalate to SRE or admin team if repeated failures occur.

Escalation

* Escalate if backups fail for more than one consecutive run. * Notify on-call engineer if production Ghost data may not be recoverable.

* GhostBackupSucceeded * GhostBackupFailed * HostOutOfDiskSpace (node running backup Job) * KubernetesPodCrashLooping

* Kubernetes → Jobs & CronJobs (namespace: <NAMESPACE>) * Grafana → Backup Job status metrics * MinIO → Backup object listings

runbooks/backup/ghost.txt · Last modified: by admin