It’s a bit tricky to figure out how to use a recording rule together with a Prometheus installation using the CoreOS Prometheus Operator. Here’s how I got it working.
If you take a look at Documentation/user-guides/Alerting.md, you’ll see that Kubernetes ConfigMaps are used to load the Prometheus configuration for each Prometheus (deployed via a CRD) in your cluster. These ConfigMaps are selected via two labels,
role: alert-rules and
So if you are using kube-prometheus to deploy the Prometheus Operator to your cluster (it gets a bit complicated with all the layers of abstraction), then if you want to deploy a recording rule directly into the
k8s Prometheus instance, you’ll want to do this:
Create the file with your recording rules and save it as something like
kind: ConfigMap apiVersion: v1 metadata: name: prometheus-example-rules labels: role: alert-rules prometheus: k8s data: example.rules.yaml: |+ groups: - name: ./example.rules rules: - alert: ExampleAlert expr: vector(1)
kubectl apply -f customrules.yaml -n monitoring
You can then go see if it worked by connecting to the Prometheus instance and looking at /rules to see if it was applied.