====== Matomo Backup CronJob ======

===== Meaning =====
These alerts monitor the Kubernetes Job created by the Matomo backup CronJob.

Alerts:
  * `MatomoBackupSucceeded` → Backup Job succeeded; MariaDB database and Matomo application data were archived and uploaded to MinIO.
  * `MatomoBackupFailed` → Backup Job failed; database dump or application data archive did not complete or failed to upload.

===== Impact =====
* Success → Backup completed successfully. Data is safely stored in MinIO.
* Failure → Matomo database or application data may not be backed up. Could affect disaster recovery if restoration is needed.

===== Diagnosis =====
1. Check Kubernetes Job status:

<code bash>
kubectl get job matomo-backup-job -n <NAMESPACE>
kubectl describe job matomo-backup-job -n <NAMESPACE>
</code>

2. Check logs of the Job pod:

<code bash>
kubectl logs job/matomo-backup-job -n <NAMESPACE>
</code>

3. Verify backup in MinIO:

<code bash>
mc ls <MINIO_ALIAS>/matomo-backups/
mc stat <MINIO_ALIAS>/matomo-backups/<backup_file>
</code>

4. Check PVC mounts if used:

<code bash>
kubectl get pvc -n <NAMESPACE>
kubectl describe pvc <PVC_NAME> -n <NAMESPACE>
</code>

===== Possible Causes of Failure =====
* Pod in CrashLoopBackOff, OOMKilled, or Failed
* PVC mount unavailable or insufficient space
* MinIO credentials missing or misconfigured
* Network issues preventing upload to MinIO
* Disk space or permissions issues on the node
* CronJob manifest misconfiguration
* Database credentials invalid or inaccessible

===== Mitigation =====
1. Inspect Job pod logs to identify errors.
2. Verify MinIO credentials and connectivity.
3. Check PVC status and node disk availability.
4. Verify database credentials and connectivity.
5. Retry backup manually if needed:

<code bash>
kubectl create job --from=cronjob/matomo-backup-job matomo-backup-job-manual -n <NAMESPACE>
</code>

6. Correct any misconfigurations in CronJob YAML, database, or MinIO bucket policy.
7. Escalate to SRE or admin team if repeated failures occur.

===== Escalation =====
* Escalate if backups fail for more than one consecutive run.
* Notify on-call engineer if production Matomo data may not be recoverable.

===== Related Alerts =====
* MatomoBackupSucceeded
* MatomoBackupFailed
* HostOutOfDiskSpace (node running backup Job)
* KubernetesPodCrashLooping

===== Related Dashboards =====
* Kubernetes → Jobs & CronJobs (namespace: <NAMESPACE>)
* Grafana → Backup Job status metrics
* MinIO → Backup object listings