Skip to content

Commit

Permalink
Declare module as github.com/gojek/work (#16)
Browse files Browse the repository at this point in the history
Earlier, the module was meant to be used with a replace directive for
github.com/gocraft/work. Declaring it as a separate module makes it
easier to build wrapper libraries as replace directives always need to
be specified in the main module. If the library uses a feature not
present in gocraft/work, it would result in hard-to-follow compilation
errors:

	# source.company.io/mygroup/worker-adapter
	../worker-adapter/adapter.go:289:89: queue.MaxConcurrency undefined (type *work.Queue has no field or method MaxConcurrency)
	../worker-adapter/adapter.go:290:84: queue.LockCount undefined (type *work.Queue has no field or method LockCount)

A somewhat better alternative is to declare as a separate module which
would result in easier to fix compilation errors.
  • Loading branch information
sudo-suhas authored Mar 16, 2021
1 parent fc020d2 commit 56b5ef4
Show file tree
Hide file tree
Showing 12 changed files with 33 additions and 32 deletions.
37 changes: 19 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# gocraft/work [![PkgGoDev][pkg-go-dev-xgo-badge]][pkg-go-dev-xgo] [![build][github-workflow-badge]][github-workflow]
# gojek/work [![PkgGoDev][pkg-go-dev-xgo-badge]][pkg-go-dev-xgo] [![build][github-workflow-badge]][github-workflow]

gocraft/work lets you enqueue and processes background jobs in Go. Jobs are durable and backed by Redis. Very similar to Sidekiq for Go.
gojek/work lets you enqueue and processes background jobs in Go. Jobs are durable and backed by Redis. Very similar to Sidekiq for Go.

* Fast and efficient. Faster than [this](https://www.github.com/jrallison/go-workers), [this](https://www.github.com/benmanns/goworker), and [this](https://www.github.com/albrow/jobs). See below for benchmarks.
* Reliable - don't lose jobs even if your process crashes.
Expand All @@ -18,10 +18,11 @@ gocraft/work lets you enqueue and processes background jobs in Go. Jobs are dura

#### Usage

The module is backward compatible with github.com/gocraft/work. To switch to this module, simply replace
`github.com/gocraft/work` with `github.com/gojek/work` in Go source files and run `go get`:

```shell
go get github.com/gocraft/work && \
go mod edit -replace github.com/gocraft/work=github.com/gojek/work@latest && \
go mod tidy
go get github.com/gojek/work
```

#### Refresh Node.js dependencies for WebUI (`99f237a`).
Expand All @@ -44,15 +45,15 @@ in-progress queue was lost.
#### Expose lock count & max concurrency for each job (#2)

Added to the queue info accessible from
[`work.Client.Queues()`](/client.go#L205-L212). Useful for alerting when lock count is consistently equal to the max
[`work.Client.Queues()`](https://pkg.go.dev/github.com/gojek/work#Client.Queues). Useful for alerting when lock count is consistently equal to the max
concurrency possibly indicating that stale lock count is resulting in jobs not being picked up.

For the cleanup to be thorough, [`work.(*WorkerPool).Stop`](https://pkg.go.dev/github.com/gojek/work#WorkerPool.Stop)
would need to be called on each worker pool instance.

#### Worker pool started check
#### Worker pool started check (#15)

Expose [`work.(*WorkerPool).Started`](/worker_pool.go#L195-L198) which can be used to check if the worker pool has been
Expose [`work.(*WorkerPool).Started`](https://pkg.go.dev/github.com/gojek/work#WorkerPool.Started) which can be used to check if the worker pool has been
started and is running.

---
Expand All @@ -66,7 +67,7 @@ package main

import (
"github.com/gomodule/redigo/redis"
"github.com/gocraft/work"
"github.com/gojek/work"
)

// Make a redis pool
Expand Down Expand Up @@ -102,7 +103,7 @@ package main

import (
"github.com/gomodule/redigo/redis"
"github.com/gocraft/work"
"github.com/gojek/work"
"os"
"os/signal"
)
Expand Down Expand Up @@ -188,7 +189,7 @@ func (c *Context) Export(job *work.Job) error {
```

## Redis Cluster
If you're attempting to use gocraft/work on a `Redis Cluster` deployment, then you may encounter a `CROSSSLOT Keys in request don't hash to the same slot` error during the execution of the various lua scripts used to manage job data (see [Issue 93](https://github.com/gocraft/work/issues/93#issuecomment-401134340)). The current workaround is to force the keys for an entire `namespace` for a given worker pool on a single node in the cluster using [Redis Hash Tags](https://redis.io/topics/cluster-spec#keys-hash-tags). Using the example above:
If you're attempting to use gojek/work on a `Redis Cluster` deployment, then you may encounter a `CROSSSLOT Keys in request don't hash to the same slot` error during the execution of the various lua scripts used to manage job data (see [Issue 93](https://github.com/gocraft/work/issues/93#issuecomment-401134340)). The current workaround is to force the keys for an entire `namespace` for a given worker pool on a single node in the cluster using [Redis Hash Tags](https://redis.io/topics/cluster-spec#keys-hash-tags). Using the example above:

```go
func main() {
Expand All @@ -206,7 +207,7 @@ func main() {
### Contexts
Just like in [gocraft/web](https://www.github.com/gocraft/web), gocraft/work lets you use your own contexts. Your context can be empty or it can have various fields in it. The fields can be whatever you want - it's your type! When a new job is processed by a worker, we'll allocate an instance of this struct and pass it to your middleware and handlers. This allows you to pass information from one middleware function to the next, and onto your handlers.
Just like in [gocraft/web](https://www.github.com/gocraft/web), gojek/work lets you use your own contexts. Your context can be empty or it can have various fields in it. The fields can be whatever you want - it's your type! When a new job is processed by a worker, we'll allocate an instance of this struct and pass it to your middleware and handlers. This allows you to pass information from one middleware function to the next, and onto your handlers.
Custom contexts aren't really needed for trivial example applications, but are very important for production apps. For instance, one field in your context can be your tagged logger. Your tagged logger augments your log statements with a job-id. This lets you filter your logs by that job-id.
Expand All @@ -222,7 +223,7 @@ func (c *Context) Export(job *work.Job) error {
for i, row := range rowsToExport {
exportRow(row)
if i % 1000 == 0 {
job.Checkin("i=" + fmt.Sprint(i)) // Here's the magic! This tells gocraft/work our status
job.Checkin("i=" + fmt.Sprint(i)) // Here's the magic! This tells gojek/work our status
}
}
}
Expand Down Expand Up @@ -266,7 +267,7 @@ For information on how this map will be serialized to form a unique key, see (ht
### Periodic Enqueueing (Cron)
You can periodically enqueue jobs on your gocraft/work cluster using your worker pool. The [scheduling specification](https://godoc.org/github.com/robfig/cron#hdr-CRON_Expression_Format) uses a Cron syntax where the fields represent seconds, minutes, hours, day of the month, month, and week of the day, respectively. Even if you have multiple worker pools on different machines, they'll all coordinate and only enqueue your job once.
You can periodically enqueue jobs on your gojek/work cluster using your worker pool. The [scheduling specification](https://godoc.org/github.com/robfig/cron#hdr-CRON_Expression_Format) uses a Cron syntax where the fields represent seconds, minutes, hours, day of the month, month, and week of the day, respectively. Even if you have multiple worker pools on different machines, they'll all coordinate and only enqueue your job once.
```go
pool := work.NewWorkerPool(Context{}, 10, "my_app_namespace", redisPool)
Expand All @@ -286,12 +287,12 @@ You can control job concurrency using `JobOptions{MaxConcurrency: <num>}`. Unlik
## Run the Web UI
The web UI provides a view to view the state of your gocraft/work cluster, inspect queued jobs, and retry or delete dead jobs.
The web UI provides a view to view the state of your gojek/work cluster, inspect queued jobs, and retry or delete dead jobs.
Building an installing the binary:
```bash
go get github.com/gocraft/work/cmd/workwebui
go install github.com/gocraft/work/cmd/workwebui
go get github.com/gojek/work/cmd/workwebui
go install github.com/gojek/work/cmd/workwebui
```
Then, you can run it:
Expand Down Expand Up @@ -332,7 +333,7 @@ You'll see a view that looks like this:
### Workers and WorkerPools
* WorkerPools provide the public API of gocraft/work.
* WorkerPools provide the public API of gojek/work.
* You can attach jobs and middleware to them.
* You can start and stop them.
* Based on their concurrency setting, they'll spin up N worker goroutines.
Expand Down
2 changes: 1 addition & 1 deletion benches/bench_work/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import (
"time"

"github.com/gocraft/health"
"github.com/gocraft/work"
"github.com/gojek/work"
"github.com/gomodule/redigo/redis"
)

Expand Down
2 changes: 1 addition & 1 deletion cmd/workenqueue/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import (
"os"
"time"

"github.com/gocraft/work"
"github.com/gojek/work"
"github.com/gomodule/redigo/redis"
)

Expand Down
2 changes: 1 addition & 1 deletion cmd/workfakedata/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import (
"math/rand"
"time"

"github.com/gocraft/work"
"github.com/gojek/work"
"github.com/gomodule/redigo/redis"
)

Expand Down
2 changes: 1 addition & 1 deletion cmd/workwebui/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (
"strconv"
"time"

"github.com/gocraft/work/webui"
"github.com/gojek/work/webui"
"github.com/gomodule/redigo/redis"
)

Expand Down
4 changes: 2 additions & 2 deletions dead_pool_reaper_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -334,15 +334,15 @@ func TestDeadPoolReaperCleanStaleLocks(t *testing.T) {

reaper := newDeadPoolReaper(ns, pool, jobNames)
// clean lock info for workerPoolID1
reaper.cleanStaleLockInfo(workerPoolID1, jobNames)
err = reaper.cleanStaleLockInfo(workerPoolID1, jobNames)
assert.NoError(t, err)
assert.EqualValues(t, 2, getInt64(pool, lock1)) // job1 lock should be decr by 1
assert.EqualValues(t, 1, getInt64(pool, lock2)) // job2 lock is unchanged
v, _ := conn.Do("HGET", lockInfo1, workerPoolID1) // workerPoolID1 removed from job1's lock info
assert.Nil(t, v)

// now clean lock info for workerPoolID2
reaper.cleanStaleLockInfo(workerPoolID2, jobNames)
err = reaper.cleanStaleLockInfo(workerPoolID2, jobNames)
assert.NoError(t, err)
// both locks should be at 0
assert.EqualValues(t, 0, getInt64(pool, lock1))
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
module github.com/gocraft/work
module github.com/gojek/work

go 1.14

Expand Down
2 changes: 1 addition & 1 deletion webui/internal/assets/build/work.js

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion webui/internal/assets/src/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ class App extends React.Component {
render() {
return (
<div className={styles.container} style={{marginTop: 30, marginBottom: 60}}>
<header><h1>gocraft/work</h1></header>
<header><h1>gojek/work</h1></header>
<hr />
<div className={styles.row}>
<main className={styles.colMd10}>
Expand Down
6 changes: 3 additions & 3 deletions webui/webui.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ import (

"github.com/braintree/manners"
"github.com/gocraft/web"
"github.com/gocraft/work"
"github.com/gocraft/work/webui/internal/assets"
"github.com/gojek/work"
"github.com/gojek/work/webui/internal/assets"
"github.com/gomodule/redigo/redis"
)

// Server implements an HTTP server which exposes a JSON API to view and manage gocraft/work items.
// Server implements an HTTP server which exposes a JSON API to view and manage gojek/work items.
type Server struct {
namespace string
pool *redis.Pool
Expand Down
2 changes: 1 addition & 1 deletion webui/webui_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
"time"

"github.com/alicebob/miniredis/v2"
"github.com/gocraft/work"
"github.com/gojek/work"
"github.com/gomodule/redigo/redis"
"github.com/stretchr/testify/assert"
)
Expand Down
2 changes: 1 addition & 1 deletion worker_pool.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
"github.com/robfig/cron/v3"
)

// WorkerPool represents a pool of workers. It forms the primary API of gocraft/work. WorkerPools provide the public API of gocraft/work. You can attach jobs and middlware to them. You can start and stop them. Based on their concurrency setting, they'll spin up N worker goroutines.
// WorkerPool represents a pool of workers. It forms the primary API of gojek/work. WorkerPools provide the public API of gojek/work. You can attach jobs and middlware to them. You can start and stop them. Based on their concurrency setting, they'll spin up N worker goroutines.
type WorkerPool struct {
workerPoolID string
concurrency uint
Expand Down

0 comments on commit 56b5ef4

Please sign in to comment.