Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot print rules since loading a recording rule #120

Open
adeverteuil opened this issue Nov 20, 2020 · 0 comments
Open

Cannot print rules since loading a recording rule #120

adeverteuil opened this issue Nov 20, 2020 · 0 comments

Comments

@adeverteuil
Copy link

I loaded this:

namespace: test
groups:
- name: default
  rules:
    - alert: AlwaysFiring
      record: ""
      for: 0s
      expr: 1 == bool 1
    - record: "agent:custom_server_info:up"
      alert: ""
      expr: |2
          custom_server_info * 0
        unless on (agent_hostname)
          up{job="integrations/agent"}
        or on (agent_hostname)
          custom_server_info

Since then, I can't print the rules anymore.

FATA[0000] unable to read rules from cortex, yaml: line 10: did not find expected key

I can query the API and get the expected YAML.

curl -u $CORTEX_USER:$CORTEX_KEY "$CORTEX_URL/api/v1/rules"
bug:
    - name: default
      rules:
        - record: test:scalar:bug
          expr: vector(1)
test:
    - name: default
      rules:
        - alert: AlwaysFiring
          expr: 1 == bool 1
        - record: agent:custom_server_info:up
          expr: |4
              custom_server_info * 0
            unless on (agent_hostname)
              up{job="integrations/agent"}
            or on (agent_hostname)
              custom_server_info

To try to isolate the bug, I deleted all the rules and I tried loading this one:

namespace: bug
groups:
- name: test
  rules:
    - record: "test:scalar:bug"
      expr: |2
          vector(1)
        or
          vector(2)
cortextool rules load rules-bug.yml \
--address=$CORTEX_URL \
--id=$CORTEX_USER \
--key=$CORTEX_KEY \
--log.level=debug

INFO[0000] log level set to debug
DEBU[0000] path built to request rule group              url=/api/prom/rules/bug/test
DEBU[0000] sending request to cortex api                 method=GET url="https://prometheus-us-central1.grafana.net/api/prom/rules/bug/test"
DEBU[0000] checking response                             status="404 Not Found"
DEBU[0000] resource not found                            fields.msg="request failed with response body group does not exist\n" status="404 Not Found"
DEBU[0000] sending request to cortex api                 method=POST url="https://prometheus-us-central1.grafana.net/api/prom/rules/bug"
DEBU[0000] checking response                             status="400 Bad Request"
ERRO[0000] requests failed                               fields.msg="request failed with response body unable to decoded rule group\n" status="400 Bad Request"
ERRO[0000] unable to load rule group                     error="failed request to the cortex api" group=test namespace=bug

I was able to load this rule group using curl.

name: bug
rules:
  - record: "test:scalar:bug"
    expr: |2
        vector(1)
      or
        vector(2)
curl -u $CORTEX_USER:$CORTEX_KEY "$CORTEX_URL/api/prom/rules/bug" -H "Content-Type: application/yaml" --data-binary @rules-bug-api.yml -i
HTTP/2 202                                                                                                                                                                                                           
content-length: 58                                                                                                                                                                                                   
content-type: application/json                                                                            
date: Fri, 20 Nov 2020 16:41:23 GMT                                                                                                                                                                                  
via: 1.1 google                                                                                                                                                                                                      
alt-svc: clear                                                                                            
                                                                                                          
{"status":"success","data":null,"errorType":"","error":""}

I'm still unable to print the rules:

INFO[0000] log level set to debug
DEBU[0000] sending request to cortex api                 method=GET url="https://prometheus-us-central1.grafana.net/api/prom/rules"
DEBU[0000] checking response                             status="200 OK"
FATA[0000] unable to read rules from cortex, yaml: line 3: did not find expected key

But I can GET them from the API:

curl -u $CORTEX_USER:$CORTEX_KEY "$CORTEX_URL/api/prom/rules"
bug:
    - name: bug
      rules:
        - record: test:scalar:bug
          expr: |4
              vector(1)
            or
              vector(2)

Also, this rule does not run! I don't see the test:scalar:bug metric in my database.

If I create the same rule on a single line, then it works, so I think both Cortex and Cortextool has an issue with the YAML block quotes with an indentation indicator syntax as described in Prometheus docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant