Skip to content
This repository has been archived by the owner on Sep 2, 2020. It is now read-only.

Graphite-api sends requests for multiple metrics but cyanite responds with one #283

Open
dancb10 opened this issue Dec 6, 2017 · 2 comments

Comments

@dancb10
Copy link

dancb10 commented Dec 6, 2017

We have created a setup of 3 Cassandra nodes in a cluster and two cyanite+graphite-api nodes. We have created all infrastructure with elbs and such and we have added the data source the graphite-api elb in grafana. Everything is connected and all flows are opened. But there is one problem when trying to build dashboards.
We are trying to get multiple metrics visible on the same dashboard in grafana. While we can see data from each metric individually, when asking for more metrics, we receive only one:

DEBUG [2017-12-06 17:40:18,696] epollEventLoopGroup-3-1 - io.cyanite.api got request:  {:uri "/paths", :get-params {:query "servers.X.disk.xvda1.disk_octets.read"}, :params {:query "servers.X.disk.xvda1.disk_octets.read"}, :request-method :get, :version "HTTP/1.1", :headers {:host "127.0.0.1:8080", :connection "keep-alive", :accept-encoding "gzip, deflate", :accept "*/*", :user-agent "python-requests/2.18.4"}, :body #object[clojure.core.async.impl.channels.ManyToManyChannel 0x52ef24df "clojure.core.async.impl.channels.ManyToManyChannel@52ef24df"]}
DEBUG [2017-12-06 17:40:18,696] epollEventLoopGroup-3-1 - io.cyanite.api path fetch request for: "servers.X.disk.xvda1.disk_octets.read"
DEBUG [2017-12-06 17:40:18,702] epollEventLoopGroup-3-1 - io.cyanite.api got request:  {:uri "/paths", :get-params {:query "servers.X.load.load.shortterm"}, :params {:query "servers.X.load.load.shortterm"}, :request-method :get, :version "HTTP/1.1", :headers {:host "127.0.0.1:8080", :connection "keep-alive", :accept-encoding "gzip, deflate", :accept "*/*", :user-agent "python-requests/2.18.4"}, :body #object[clojure.core.async.impl.channels.ManyToManyChannel 0x63e298b4 "clojure.core.async.impl.channels.ManyToManyChannel@63e298b4"]}
DEBUG [2017-12-06 17:40:18,702] epollEventLoopGroup-3-1 - io.cyanite.api path fetch request for: "servers.X.load.load.shortterm"
DEBUG [2017-12-06 17:40:18,706] epollEventLoopGroup-3-1 - io.cyanite.api got request:  {:uri "/metrics", :get-params {:path "servers.X.disk.xvda1.disk_octets.read", :from "1512578418", :to "1512582018"}, :params {:path "servers.X.xvda1.disk_octets.read", :from "1512578418", :to "1512582018"}, :request-method :get, :version "HTTP/1.1", :headers {:host "127.0.0.1:8080", :connection "keep-alive", :accept-encoding "gzip, deflate", :accept "*/*", :user-agent "python-requests/2.18.4"}, :body #object[clojure.core.async.impl.channels.ManyToManyChannel 0x490637d8 "clojure.core.async.impl.channels.ManyToManyChannel@490637d8"]}
DEBUG [2017-12-06 17:40:18,707] epollEventLoopGroup-3-1 - io.cyanite.api metric fetch request for: "servers.X.disk.xvda1.disk_octets.read"

So basically we query for disk.xvda1.disk_octets.read and load.load.shortterm and we receive only disk_octets.read. This issue is blocking us from making complex dashboards with multiple metrics.

We have tried also with curl and we get the same behavior:

curl -X POST https://X/render --data ‘target=servers.X.disk.xvda1.disk_octets.read&target=servers.X.load.load.shortterm&from=-1h&until=now&format=json&maxDataPoints=1311’

How can we fix this issue? (we are using Graphite-Api 1.1.2, latest cyanite version and grafana 4.6.x)
Note that I have replaced our node name with X

@dancb10
Copy link
Author

dancb10 commented Dec 7, 2017

@pyr seems like this issue is related to another opened issue #279

@dancb10
Copy link
Author

dancb10 commented Dec 7, 2017

I have compiled the project after this commit #265 and it seems that now we can see data but the log file of cyanite is spamming with the following message:
INFO [2017-12-07 13:11:24,242] cyanite-snapshot - io.cyanite.engine.writer starting snapshot.
INFO [2017-12-07 13:11:25,243] cyanite-snapshot - io.cyanite.engine.writer starting snapshot.
INFO [2017-12-07 13:11:26,242] cyanite-snapshot - io.cyanite.engine.writer starting snapshot.
INFO [2017-12-07 13:11:27,243] cyanite-snapshot - io.cyanite.engine.writer starting snapshot.
INFO [2017-12-07 13:11:28,242] cyanite-snapshot - io.cyanite.engine.writer starting snapshot.
INFO [2017-12-07 13:11:29,242] cyanite-snapshot - io.cyanite.engine.writer starting snapshot.
INFO [2017-12-07 13:11:30,243] cyanite-snapshot - io.cyanite.engine.writer starting snapshot.
INFO [2017-12-07 13:11:31,243] cyanite-snapshot - io.cyanite.engine.writer starting snapshot.
INFO [2017-12-07 13:11:32,243] cyanite-snapshot - io.cyanite.engine.writer starting snapshot.
INFO [2017-12-07 13:11:33,243] cyanite-snapshot - io.cyanite.engine.writer starting snapshot.
INFO [2017-12-07 13:11:34,243] cyanite-snapshot - io.cyanite.engine.writer starting snapshot.

It also seems like we are loosing data somehow. We are still debugging

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant