Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cloudflare worker to collect API key usage inside Influx DB #1

Merged
merged 40 commits into from
Dec 17, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
07f4429
Cache response inside cloudflare worker
pyropy Dec 10, 2024
6c5318d
Do not cache POST requests
pyropy Dec 10, 2024
e2ca738
Do not cache POST request responses
pyropy Dec 10, 2024
51b147b
Simplify request formating code
pyropy Dec 10, 2024
4c547d7
Remove unused function arguments
pyropy Dec 10, 2024
467e081
Use default project layout
pyropy Dec 10, 2024
0bba446
Update example env
pyropy Dec 10, 2024
15abebe
Move request logic to worker
pyropy Dec 10, 2024
5891c88
Add unit tests for worker.fetch
pyropy Dec 10, 2024
df26c99
Add basic docs
pyropy Dec 10, 2024
1ab824c
Update lib/request.js
pyropy Dec 11, 2024
3525f3e
Update lib/metrics.js
pyropy Dec 11, 2024
f6c88ac
Update lib/request.js
pyropy Dec 11, 2024
8732dc3
Update lib/request.js
pyropy Dec 11, 2024
882792d
Update .env.example
pyropy Dec 11, 2024
d27848d
Move all influx db related code to lib/influx.js
pyropy Dec 11, 2024
116b146
Add basic github actions
pyropy Dec 11, 2024
bb1ae1c
Fix failing test
pyropy Dec 11, 2024
a8d3bb9
Fix missing wranger.toml in test job
pyropy Dec 11, 2024
6b76ef4
Update readme with deployment steps
pyropy Dec 11, 2024
6143e4c
Fix README URL for getting CF API token
pyropy Dec 12, 2024
8c65bab
Improve api key parsing to be more readable
pyropy Dec 12, 2024
c3fd71f
Simplify worker code; Don't compress metrics
pyropy Dec 12, 2024
4bd5fb2
Remove unused env variable
pyropy Dec 12, 2024
700b15a
Report api key as value
pyropy Dec 12, 2024
201b375
Edit default env variables and secrets
pyropy Dec 12, 2024
7f844c6
Reformat files with newline at EOF
pyropy Dec 12, 2024
16cae7e
Refactor influx.js module
pyropy Dec 12, 2024
ef2ecd0
Reformat JSDoc
pyropy Dec 12, 2024
842eaf5
Update test name
pyropy Dec 12, 2024
81e8b56
Add table tests for influx lib
pyropy Dec 12, 2024
8fc77ed
Refactor how influx lib is tested
pyropy Dec 12, 2024
c43b0d5
Add test for reportRequestMetric
pyropy Dec 12, 2024
081d4ea
Improve influx test
pyropy Dec 13, 2024
bb531a9
Use envsubst to replace env vars in gh actions
pyropy Dec 13, 2024
357da93
Update lib/influx.js
pyropy Dec 16, 2024
82feb43
Update bin/worker.js
pyropy Dec 16, 2024
bdc81ee
Fix imports and mocks for renamed functions in tests
pyropy Dec 16, 2024
99c0884
Rename project to spark-stats-request-metrics
pyropy Dec 17, 2024
8f5ef87
Rename import for reporting metrics to influx
pyropy Dec 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
INFLUX_HOST=http://localhost:8086
INFLUX_TOKEN=example
INFLUX_DATABASE=example
REQUEST_URL="https://example.com"
INFLUX_METRIC="example"
pyropy marked this conversation as resolved.
Show resolved Hide resolved
INFLUX_URL="http://localhost:8086"
INFLUX_TOKEN="example"
INFLUX_DATABASE="example"
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
/dist
**/*.rs.bk
Cargo.lock
bin/
pkg/
wasm-pack.log
worker/
Expand Down
22 changes: 6 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,12 @@

Send your page views from [Cloudflare worker](https://developers.cloudflare.com/workers/) to InfluxDB.

![Dashboard views](static/dashboard.png)
## Get started

## Requirements

1. Your site need to be setup behind Cloudflare CDN.
2. You need to setup InfluxDB with external access (make sure you have set [authentication](https://docs.influxdata.com/influxdb/v1.7/administration/authentication_and_authorization/#set-up-authentication))
1. Make sure InfluxDB is hosted under [supported port](https://blog.cloudflare.com/cloudflare-now-supporting-more-ports/) for Workers. Best option is 80 or 443.

## How to use

1. Install wrangler package
1. Install dependencies

```
npm i @cloudflare/wrangler -g
npm install
```

2. Copy example files
Expand All @@ -25,10 +17,8 @@ cp .env.example .env
cp wrangler.toml.example wrangler.toml
```

3. Deploy your worker to a site with wrangler
3. Deploy your worker

```
wrangler publish --env production
```

4. (Optional) If you're using Grafana with InfluxDB, then you can import [example Dashboard](static/dashboard.json) from first screen.
npm run deploy
juliangruber marked this conversation as resolved.
Show resolved Hide resolved
```
16 changes: 16 additions & 0 deletions bin/worker.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
import { reportMetric } from "../lib/metrics";

export const handleRequest = async (request, env) => {
const url = new URL(request.url);
const newRequest = new Request(`${env.REQUEST_URL}${url.pathname}${url.search}`, request);

return await fetch(newRequest);
}
pyropy marked this conversation as resolved.
Show resolved Hide resolved

export default {
async fetch(request, env, ctx) {
const response = await handleRequest(request, env);
ctx.waitUntil(reportMetric(request, response, env));
return response;
}
}
66 changes: 0 additions & 66 deletions index.js

This file was deleted.

25 changes: 25 additions & 0 deletions lib/metrics.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import zlib from 'node:zlib';
import { promisify } from 'node:util';
import { formatRequestData } from './request.js';

export const reportMetric = async (request, response, env) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
export const reportMetric = async (request, response, env) => {
export const reportRequest = async (request, response, env) => {

Together with the file name, this makes more sense to me

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @pyropy

const compress = promisify(zlib.gzip);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be defined outside the reportMetric handler

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping

const requestData = formatRequestData(request, response, env);

// Define API endpoint and headers
const url = `${env.INFLUX_URL}/api/v2/write?&bucket=${env.INFLUX_DATABASE}&precision=ms`;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can also be defined outside this function

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping


// Compress the string using gzip
const compressedData = await compress(requestData);

// Make the POST request
return fetch(url, {
method: 'POST',
headers: {
'Authorization': `Token ${env.INFLUX_TOKEN}`,
'Content-Encoding': 'gzip',
juliangruber marked this conversation as resolved.
Show resolved Hide resolved
'Content-Type': 'text/plain; charset=utf-8',
},
body: compressedData,
pyropy marked this conversation as resolved.
Show resolved Hide resolved
})
}
11 changes: 11 additions & 0 deletions lib/request.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
export const formatRequestData = (request, response, env) => {
juliangruber marked this conversation as resolved.
Show resolved Hide resolved
pyropy marked this conversation as resolved.
Show resolved Hide resolved
const url = new URL(request.url);
const timestamp = Date.now();
const originResponse = response || {};
pyropy marked this conversation as resolved.
Show resolved Hide resolved
const apiKey = request.headers.get('api-key') || url.searchParams.get('api-key') || (request.headers.get('Authorization')?.startsWith('Bearer ') ? request.headers.get('Authorization').substring(7) : 'unknown');
const cfCache = (originResponse) ? (response.headers.get('CF-Cache-Status') || 'miss') : 'miss'
pyropy marked this conversation as resolved.
Show resolved Hide resolved
const formattedUrl = url.toString().replaceAll('=', '\\=');

// We're setting field value to 1 to count the number of requests
return `${env.INFLUX_METRIC},status_code=${originResponse.status},url=${formattedUrl},hostname=${url.hostname},pathname="${url.pathname}",method=${request.method},cf_cache=${cfCache},api_key=${apiKey} value=1 ${timestamp}`
pyropy marked this conversation as resolved.
Show resolved Hide resolved
}
Loading