Backing up Prometheus using TSDB Snapshots : Kubernetes/OpenShift

Karan Singh

·

Follow

2 min read

·

May 12, 2021

--

Share

These are my quick-and-dirty brain-dump notes to myself on how to backup prometheus database running on k8s or OpenShift

• Get Token for API Authentication and Prometheus API Route URL

oc whoami -t
oc get route -n openshift-monitoring | grep -i prometheus

• Run sample curl request

curl -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/query?query=ALERTS' | python -m json.tool

• Create TSDB Snapshot

curl -X ‘POST’ -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot' | python -m json.tool

• You might get an error, so you first need to enable Admin APIs

{
 “error”: “Admin APIs are disabled”,
 “message”: “Admin APIs are disabled”,
 “code”: 14
}

• Enable AdminAPI

oc -n openshift-monitoring patch prometheus k8s \
 — type merge — patch ‘{“spec”:{“enableAdminAPI”:true}}’

• Verify Admin API is enabled

oc describe po prometheus-k8s-1 | grep -i admin
 —…