Vforeseetech

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Jenkins Trigger Job / Backup Pipeline ======

===== Meaning =====
This alert is triggered whenever a Jenkins job executes to:
  1. Create a Kubernetes Job from a CronJob manifest
  2. Trigger the backup pipeline for a specific tool

Alerts:
  * `JenkinsBuildSuccess` → Jenkins successfully created the Job and triggered the pipeline
  * `JenkinsBuildFailed` → Jenkins failed to create the Job or trigger the pipeline

This runbook applies to all tools using Jenkins to trigger Kubernetes Jobs or backup pipelines.

===== Impact =====
* Success → Backup Job or pipeline has been successfully triggered. The actual backup may run later via the Job.
* Failure → Backup Job was not created; backup may not occur, potentially affecting RPO.

===== Diagnosis =====
1. Check Jenkins build status and logs:

<code bash>
# From Jenkins UI or CLI
BUILD_NUMBER=<build_number>
Jenkins → Job → Build #${BUILD_NUMBER} → Console Output
</code>

2. Check Kubernetes Job creation:

<code bash>
kubectl get jobs -n <NAMESPACE>
kubectl describe job <JOB_NAME> -n <NAMESPACE>
</code>

3. Check if the Job has started successfully:

<code bash>
kubectl logs job/<JOB_NAME> -n <NAMESPACE>
</code>

4. Check Alertmanager alert (optional):

<code bash>
curl -s http://<alertmanager-endpoint>/api/v2/alerts | jq '.[] | select(.labels.build=="<BUILD_NUMBER>")'
</code>

===== Possible Causes of Failure =====
* Jenkins agent cannot start or run kubectl commands
* Kubernetes secret (kubeconfig) missing or invalid
* Incorrect namespace or Job manifest YAML error
* RBAC/permissions issues preventing Job creation
* Network issues accessing Kubernetes API
* Pipeline script errors

===== Mitigation =====
1. Verify Jenkins agent and kubeconfig secret exist and are correct
2. Validate the Kubernetes CronJob YAML:

<code bash>
kubectl apply -f <cronjob.yaml> --dry-run=client
</code>

3. Retry Jenkins build manually if needed
4. Ensure the namespace exists and has correct permissions
5. Ensure Jenkins agent can communicate with the Kubernetes API

===== Escalation =====
* Escalate if Job cannot be created for more than one consecutive run
* Notify on-call engineer if production backup pipeline may be missed

===== Related Alerts =====
* JenkinsBuildSuccess
* JenkinsBuildFailed

===== Related Dashboards =====
* Jenkins → All Jobs Overview
* Kubernetes → Jobs & CronJobs for backups
* Alertmanager → Jenkins Alerts