Vforeseetech

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== n8n Backup CronJob ======

===== Meaning =====
These alerts monitor the Kubernetes Job created by the n8n backup CronJob.

Alerts:
  * `N8nBackupSucceeded` → Backup Job succeeded; n8n PVC data was archived and uploaded to MinIO.
  * `N8nBackupFailed` → Backup Job failed; PVC data backup did not complete or failed to upload.

===== Impact =====
* Success → Backup completed successfully. n8n workflow data is safely stored in MinIO.
* Failure → n8n workflow data may not be backed up. Could affect disaster recovery if restoration is needed.

===== Diagnosis =====
1. Check Kubernetes Job status:

<code bash>
kubectl get job n8n-backup-job -n <NAMESPACE>
kubectl describe job n8n-backup-job -n <NAMESPACE>
</code>

2. Check logs of the Job pod:

<code bash>
kubectl logs job/n8n-backup-job -n <NAMESPACE>
</code>

3. Verify backup in MinIO:

<code bash>
mc ls <MINIO_ALIAS>/n8n-backups/
mc stat <MINIO_ALIAS>/n8n-backups/<backup_file>
</code>

4. Check PVC mounts:

<code bash>
kubectl get pvc -n <NAMESPACE>
kubectl describe pvc <PVC_NAME> -n <NAMESPACE>
</code>

===== Possible Causes of Failure =====
* Pod in CrashLoopBackOff, OOMKilled, or Failed
* PVC mount unavailable or insufficient space
* MinIO credentials missing or misconfigured
* Network issues preventing upload to MinIO
* Disk space or permissions issues on the node
* CronJob manifest misconfiguration

===== Mitigation =====
1. Inspect Job pod logs to identify errors.
2. Verify MinIO credentials and connectivity.
3. Check PVC status and node disk availability.
4. Retry backup manually if needed:

<code bash>
kubectl create job --from=cronjob/n8n-backup-job n8n-backup-job-manual -n <NAMESPACE>
</code>

5. Correct any misconfigurations in CronJob YAML, PVC, or MinIO bucket policy.
6. Escalate to SRE or admin team if repeated failures occur.

===== Escalation =====
* Escalate if backups fail for more than one consecutive run.
* Notify on-call engineer if n8n workflow data may not be recoverable.

===== Related Alerts =====
* N8nBackupSucceeded
* N8nBackupFailed
* HostOutOfDiskSpace (node running backup Job)
* KubernetesPodCrashLooping

===== Related Dashboards =====
* Kubernetes → Jobs & CronJobs (namespace: <NAMESPACE>)
* Grafana → Backup Job status metrics
* MinIO → Backup object listings