Vforeseetech

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Serpbear Backup CronJob ======

===== Meaning =====
These alerts monitor the Kubernetes Job created by the Serpbear backup CronJob.

Alerts:
  * `SerpbearBackupSucceeded` → Backup Job succeeded; Serpbear PVC data was archived and uploaded to MinIO.
  * `SerpbearBackupFailed` → Backup Job failed; PVC data backup did not complete or failed to upload.

===== Impact =====
* Success → Backup completed successfully. Serpbear data is safely stored in MinIO.
* Failure → Serpbear PVC data may not be backed up. Could affect disaster recovery if restoration is needed.

===== Diagnosis =====
1. Check Kubernetes Job status:

<code bash>
kubectl get job serpbear-backup-job -n <NAMESPACE>
kubectl describe job serpbear-backup-job -n <NAMESPACE>
</code>

2. Check logs of the Job pod:

<code bash>
kubectl logs job/serpbear-backup-job -n <NAMESPACE>
</code>

3. Verify backup in MinIO:

<code bash>
mc ls <MINIO_ALIAS>/serpbear-backups/
mc stat <MINIO_ALIAS>/serpbear-backups/<backup_file>
</code>

4. Check PVC mounts:

<code bash>
kubectl get pvc -n <NAMESPACE>
kubectl describe pvc <PVC_NAME> -n <NAMESPACE>
</code>

===== Possible Causes of Failure =====
* Pod in CrashLoopBackOff, OOMKilled, or Failed
* PVC mount unavailable or insufficient space
* MinIO credentials missing or misconfigured
* Network issues preventing upload to MinIO
* Disk space or permissions issues on the node
* CronJob manifest misconfiguration

===== Mitigation =====
1. Inspect Job pod logs to identify errors.
2. Verify MinIO credentials and connectivity.
3. Check PVC status and node disk availability.
4. Retry backup manually if needed:

<code bash>
kubectl create job --from=cronjob/serpbear-backup-job serpbear-backup-job-manual -n <NAMESPACE>
</code>

5. Correct any misconfigurations in CronJob YAML, PVC, or MinIO bucket policy.
6. Escalate to SRE or admin team if repeated failures occur.

===== Escalation =====
* Escalate if backups fail for more than one consecutive run.
* Notify on-call engineer if Serpbear data may not be recoverable.

===== Related Alerts =====
* SerpbearBackupSucceeded
* SerpbearBackupFailed
* HostOutOfDiskSpace (node running backup Job)
* KubernetesPodCrashLooping

===== Related Dashboards =====
* Kubernetes → Jobs & CronJobs (namespace: <NAMESPACE>)
* Grafana → Backup Job status metrics
* MinIO → Backup object listings