feat(logs): Loki integration #108
Conversation
Signed-off-by: Zander Franks <[email protected]>
Signed-off-by: Zander Franks <[email protected]>
|
I'm undrafting this because it is not feature-complete in terms of auth and Grafana, but it is functional, and if we want it merged in before running any number of Canarynet nodes, we can merge it in early. |
| thread::spawn(|| { | ||
| let rt = tokio::runtime::Runtime::new().unwrap(); | ||
| let handle = rt.spawn(task); | ||
| rt.block_on(handle).unwrap(); | ||
| }); |
There was a problem hiding this comment.
This might cause a panic. Could happen when we run the node and it delegates a tokio task to this thread, but it already has a runtime.
There was a problem hiding this comment.
Thinking about how to deal with this... Might switch to futures::executor::block_on
| @@ -47,8 +45,7 @@ pub fn init(state: Arc<GlobalState>) { | |||
| .await | |||
| { | |||
| Ok(response) => response, | |||
| Err(e) => { | |||
| warn!("failed to scrape latest metrics: {e}"); | |||
There was a problem hiding this comment.
We no longer want this warning?
There was a problem hiding this comment.
Can re-add once we have something that checks if the node is actually running. At the moment it is always spam
| @@ -93,7 +92,7 @@ pub async fn log_request(uri: Uri, method: Method, req_stamp: ReqStamp, res: Res | |||
| }; | |||
|
|
|||
| // TODO: send to logging services | |||
| debug!("REQUEST LOG LINE:\n{}", json!(log_line)); | |||
| // debug!("REQUEST LOG LINE:\n{}", json!(log_line)); | |||
There was a problem hiding this comment.
We should keep this. it's useful for debugging, we can drop it down to a trace level if preferred.
We should also send these to loki so all requests to the cp are logged for tracing purposes.
There was a problem hiding this comment.
It's very noisy with Prometheus running as it checks the service discovery API every few seconds. Could forward to Loki though
There was a problem hiding this comment.
I could add a filter to ignore the Loki spam too
|
|
||
| AOT_BIN="$(pwd)/target/$PROFILE/snarkos-aot" \ | ||
| AGENT_BIN="$(pwd)/target/$PROFILE/snops-agent" \ | ||
| cargo watch -x 'run -p snops -- --prometheus http://127.0.0.1:9090 --loki http://127.0.0.1:3100' \ |
There was a problem hiding this comment.
I wonder if we should make some of these things an ENV var in the future so you can just have .env file.
|
merging this for two reasons:
|
Loki integration to track agent node logs.