Using Recording Rules with the Prometheus Operator

It’s a bit tricky to figure out how to use a recording rule together with a Prometheus installation using the CoreOS Prometheus Operator. Here’s how I got it working.

If you take a look at Documentation/user-guides/Alerting.md, you’ll see that Kubernetes ConfigMaps are used to load the Prometheus configuration for each Prometheus (deployed via a CRD) in your cluster. These ConfigMaps are selected via two labels, role: alert-rules and prometheus: <example>.

So if you are using kube-prometheus to deploy the Prometheus Operator to your cluster (it gets a bit complicated with all the layers of abstraction), then if you want to deploy a recording rule directly into the k8s Prometheus instance, you’ll want to do this:

Create the file with your recording rules and save it as something like custom-rules.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-example-rules
  labels:
    role: alert-rules
    prometheus: k8s
data:
  example.rules.yaml: |+
    groups:
    - name: ./example.rules
      rules:
      - alert: ExampleAlert
        expr: vector(1)

run kubectl apply -f customrules.yaml -n monitoring

You can then go see if it worked by connecting to the Prometheus instance and looking at /rules to see if it was applied.