Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Clickhouse Storage Driver #1342

Merged
merged 46 commits into from
Jul 3, 2024
Merged
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
f9a22ee
feat: add clickhouse implementation
luk3skyw4lker Apr 9, 2024
5ecaadb
chore: update branch with main
luk3skyw4lker Apr 9, 2024
efea419
fix: invalid go version error
luk3skyw4lker Apr 10, 2024
b8c51d1
fix: add benchmark and fix mismatched type on set function
luk3skyw4lker Apr 10, 2024
de69aa6
Merge branch 'main' into feat/add-clickhouse-storage
luk3skyw4lker Apr 10, 2024
2baf7d7
revert: go.mod golang version change
luk3skyw4lker Apr 16, 2024
de3083f
chore: add endline to root go.mod
luk3skyw4lker Apr 16, 2024
4c5f2ef
chore: remove endline from go.mod file
luk3skyw4lker Apr 16, 2024
8f7781c
Merge branch 'main' into feat/add-clickhouse-storage
luk3skyw4lker Apr 16, 2024
8a64cac
Update go.mod
gaby Apr 16, 2024
81b668a
Update test-clickhouse.yml
gaby Apr 16, 2024
99c0994
Update test-cloudflarekv.yml
gaby Apr 16, 2024
21ac41f
ci: add clickhosue to benchmarks.yml
luk3skyw4lker Apr 18, 2024
3f5fda6
Merge branch 'main' into feat/add-clickhouse-storage
luk3skyw4lker Apr 18, 2024
3b83def
Update test-cloudflarekv.yml
gaby Apr 19, 2024
f212e66
chore: change clickhouse to port 9001
luk3skyw4lker Apr 22, 2024
2faf158
Merge branch 'main' into feat/add-clickhouse-storage
luk3skyw4lker Apr 22, 2024
8120322
Merge branch 'main' into feat/add-clickhouse-storage
gaby Apr 30, 2024
bfd7ccc
refactor: change some namings and behaviors on Get
luk3skyw4lker May 6, 2024
0c53f45
chore: return nil for error on storage creation
luk3skyw4lker May 6, 2024
ca256c0
chore: return nil for error on storage creation
luk3skyw4lker May 6, 2024
11d7b5f
refactor: improve comparision beteween times
luk3skyw4lker May 6, 2024
bdf87e3
refactor: check for connection errors before returning storage
luk3skyw4lker May 6, 2024
df5c2b3
Merge branch 'main' into feat/add-clickhouse-storage
luk3skyw4lker May 16, 2024
3ef0002
feat: add zstd compression to columns
luk3skyw4lker Jun 20, 2024
c62c0af
fix: broken tests and benchmarks
luk3skyw4lker Jun 20, 2024
ad436b5
Merge branch 'main' into feat/add-clickhouse-storage
luk3skyw4lker Jun 20, 2024
c70e449
refactor: minor code rabbit changes
luk3skyw4lker Jun 20, 2024
0887a66
chore: update clickhouse driver
luk3skyw4lker Jun 25, 2024
1f3a53a
chore: update supported versions of Go in README
luk3skyw4lker Jun 25, 2024
bf0d8fc
chore: add release-drafter-clickhouse.yml and config
luk3skyw4lker Jun 25, 2024
d27ee36
chore: add clickhouse to dependabot.yml
luk3skyw4lker Jun 25, 2024
292cee6
docs: add clickhouse README reference docs
luk3skyw4lker Jun 25, 2024
275be22
Merge branch 'main' into feat/add-clickhouse-storage
luk3skyw4lker Jun 25, 2024
4311191
docs: update clickhouse/README.md
luk3skyw4lker Jun 26, 2024
f4ee2db
docs: update markdown formatting
luk3skyw4lker Jun 26, 2024
6b75862
Merge branch 'main' into feat/add-clickhouse-storage
ReneWerner87 Jun 26, 2024
d872a85
chore: tidying up modules
luk3skyw4lker Jun 26, 2024
f27f7b1
fix: time.Time.IsZero returning false negative
luk3skyw4lker Jul 1, 2024
0228c63
Merge branch 'main' into feat/add-clickhouse-storage
luk3skyw4lker Jul 1, 2024
e744bc0
refactor: change some error returns to nil instead of &Storage
luk3skyw4lker Jul 1, 2024
f9d76aa
Update clickhouse README.md
gaby Jul 3, 2024
337546b
Update comments in clickhouse.go
gaby Jul 3, 2024
ea065c4
Enable -race for clickhouse tests
gaby Jul 3, 2024
39412ce
Update go.mod for Clickhouse
gaby Jul 3, 2024
cfd5f69
Merge branch 'main' into feat/add-clickhouse-storage
ReneWerner87 Jul 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,9 @@ updates:
- "🤖 Dependencies"
schedule:
interval: "daily"
- package-ecosystem: "gomod"
directory: "/clickhouse/" # Location of package manifests
labels:
- "🤖 Dependencies"
schedule:
interval: "daily"
50 changes: 50 additions & 0 deletions .github/release-drafter-clickhouse.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name-template: 'ClickHouse - v$RESOLVED_VERSION'
tag-template: 'clickhouse/v$RESOLVED_VERSION'
tag-prefix: clickhouse/v
include-paths:
- clickhouse
categories:
- title: '❗ Breaking Changes'
labels:
- '❗ BreakingChange'
- title: '🚀 New'
labels:
- '✏️ Feature'
- title: '🧹 Updates'
labels:
- '🧹 Updates'
- '🤖 Dependencies'
- title: '🐛 Fixes'
labels:
- '☢️ Bug'
- title: '📚 Documentation'
labels:
- '📒 Documentation'
change-template: '- $TITLE (#$NUMBER)'
change-title-escapes: '\<*_&' # You can add # and @ to disable mentions, and add ` to disable code blocks.
exclude-contributors:
- dependabot
- dependabot[bot]
version-resolver:
major:
labels:
- 'major'
- '❗ BreakingChange'
minor:
labels:
- 'minor'
- '✏️ Feature'
patch:
labels:
- 'patch'
- '📒 Documentation'
- '☢️ Bug'
- '🤖 Dependencies'
- '🧹 Updates'
default: patch
template: |
$CHANGES

**Full Changelog**: https://github.com/$OWNER/$REPOSITORY/compare/$PREVIOUS_TAG...clickhouse/v$RESOLVED_VERSION

Thank you $CONTRIBUTORS for making this update possible.
5 changes: 5 additions & 0 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,11 @@ jobs:
docker run --name scylladb -p 9042:9042 -p 19042:19042 -p 9160:9160 -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9180:9180 -d scylladb/scylla:latest --broadcast-address 127.0.0.1 --listen-address 0.0.0.0 --broadcast-rpc-address 127.0.0.1
sleep 15 # Wait for ScyllaDb to initialize

- name: Startup Clickhouse
run: |
docker run -d -p 9001:9000 --name clickhouse --ulimit nofile=262144:262144 clickhouse/clickhouse-server
sleep 10 # Wait for Clickhouse to initialize

- name: Setup Redis
uses: shogo82148/actions-setup-redis@v1
with:
Expand Down
19 changes: 19 additions & 0 deletions .github/workflows/release-drafter-clickhouse.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: Release Drafter Clickhouse
on:
push:
# branches to consider in the event; optional, defaults to all
branches:
- master
- main
paths:
- 'clickhouse/**'
jobs:
draft_release_clickhouse:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: release-drafter/release-drafter@v6
with:
config-name: release-drafter-clickhouse.yml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
32 changes: 32 additions & 0 deletions .github/workflows/test-clickhouse.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
on:
push:
branches:
- master
- main
paths:
- 'clickhouse/**'
pull_request:
paths:
- 'clickhouse/**'
name: 'Tests Clickhouse'
jobs:
Tests:
runs-on: ubuntu-latest
strategy:
matrix:
go-version:
- 1.21.x
- 1.22.x
steps:
- name: Fetch Repository
uses: actions/checkout@v4
- name: Startup Clickhouse
run: |
docker run -d -p 9001:9000 --name clickhouse --ulimit nofile=262144:262144 clickhouse/clickhouse-server
sleep 30
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: '${{ matrix.go-version }}'
- name: Run Test
run: cd ./clickhouse && go clean -testcache && go test ./... -v
8 changes: 5 additions & 3 deletions .github/workflows/test-cloudflarekv.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,13 @@ name: Tests CloudflareKV
on:
push:
branches:
- master
- main
paths:
- 'cloudflarekv/**'
pull_request:
branches:
- main
paths:
- 'cloudflarekv/**'

jobs:
Tests:
Expand All @@ -16,7 +19,6 @@ jobs:
go-version:
- 1.21.x
- 1.22.x

steps:
- name: Checkout Repository
uses: actions/checkout@v4
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,3 +74,4 @@ type Storage interface {
- [S3](./s3/README.md) <a href="https://github.com/gofiber/storage/actions?query=workflow%3A%22Tests+S3%22"> <img src="https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-s3.yml?branch=main&label=%F0%9F%A7%AA%20&style=flat&color=75C46B" /> </a>
- [ScyllaDB](./scylladb/README.md) <a href="https://github.com/gofiber/storage/actions?query=workflow%3A%22Tests+scylladb%22"> <img src="https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-scylladb.yml?branch=main&label=%F0%9F%A7%AA%20&style=flat&color=75C46B" /> </a>
- [SQLite3](./sqlite3/README.md) <a href="https://github.com/gofiber/storage/actions?query=workflow%3A%22Tests+Sqlite3%22"> <img src="https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-sqlite3.yml?branch=main&label=%F0%9F%A7%AA%20&style=flat&color=75C46B" /> </a>
- [ClickHouse](./clickhouse/README.md) <a href="https://github.com/gofiber/storage/actions?query=workflow%3A%22Tests+Clickhouse%22"> <img src="https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-clickhouse.yml?branch=main&label=%F0%9F%A7%AA%20&style=flat&color=75C46B" /> </a>
119 changes: 119 additions & 0 deletions clickhouse/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# Clickhouse
luk3skyw4lker marked this conversation as resolved.
Show resolved Hide resolved

A Clickhouse storage driver using [https://github.com/ClickHouse/clickhouse-go](https://github.com/ClickHouse/clickhouse-go).

### Table of Contents
luk3skyw4lker marked this conversation as resolved.
Show resolved Hide resolved

- [Signatures](#signatures)
- [Installation](#installation)
- [Examples](#examples)
- [Config](#config)
- [Default Config](#default-config)

### Signatures
luk3skyw4lker marked this conversation as resolved.
Show resolved Hide resolved

```go
func New(config ...Config) (*Storage, error)
func (s *Storage) Get(key string) ([]byte, error)
func (s *Storage) Set(key string, val []byte, exp time.Duration) error
func (s *Storage) Delete(key string) error
func (s *Storage) Reset() error
func (s *Storage) Close() error
func (s *Storage) Conn() *Session
```

### Installation
luk3skyw4lker marked this conversation as resolved.
Show resolved Hide resolved

Clickhouse is supported on the latest two versions of Go:

Install the clickhouse implementation:
```bash
go get github.com/gofiber/storage/clickhouse
```

Before running or testing this implementation, you must ensure a Clickhouse cluster is available.
For local development, we recommend using the Clickhouse Docker image; it contains everything
necessary for the client to operate correctly.

To start Clickhouse using Docker, issue the following:

```bash
docker run -d -p 9000:9000 --name some-clickhouse-server --ulimit nofile=262144:262144 clickhouse/clickhouse-server
```

After running this command you're ready to start using the storage and connecting to the database.

### Examples
Import the storage package.
```go
import "github.com/gofiber/storage/coherence"
```

You can use the following possibilities to create a storage:
```go
// Initialize default config, to connect to localhost:9000 using the memory engine and with a clean table.
store, err := clickhouse.New(clickhouse.Config{
Host: "localhost",
Port: 9000,
Clean: true,
})

// Initialize custom config to connect to a different host/port and use custom engine and with clean table.
store, err := clickhouse.New(clickhouse.Config{
Host: "some-ip-address",
Port: 9000,
Engine: clickhouse.MergeTree,
Clean: true,
})

// Initialize to connect with TLS enabled with your own tls.Config and with clean table.
tlsConfig := config := &tls.Config{...}

store, err := clickhouse.New(coherence.Config{
Host: "some-ip-address",
Port: 9000,
Clean: true,
TLSConfig: tlsConfig,
})
```

### Config

```go
// Config defines configuration options for Clickhouse connection.
type Config struct {
// The host of the database. Ex: 127.0.0.1
Host string
// The port where the database is supposed to listen to. Ex: 9000
Port int
// The database that the connection should authenticate from
Database string
// The username to be used in the authentication
Username string
// The password to be used in the authentication
Password string
// The name of the table that will store the data
Table string
// The engine that should be used in the table
Engine string
// Should start a clean table, default false
Clean bool
// TLS configuration, default nil
TLSConfig *tls.Config
// Should the connection be in debug mode, default false
Debug bool
// The function to use with the debug config, default print function. It only works when debug is true
Debugf func(format string, v ...any)
}
```

### Default Config

```go
var DefaultConfig = Config{
Host: "localhost",
Port: 9000,
Engine: "Memory",
Clean: false,
}
```
Loading
Loading