Amboy is a collection of interfaces and tools for running and managing asynchronous background work queues in the context of Go programs, and provides a number of interchangeable and robust methods for running tasks.
Queue implementations impose ordering and dispatching behavior, and describe the storage of tasks before and after work is complete. Current queue implementations include:
- an ordered queue that dispatches tasks ordered by dependency information to ensure that dependent tasks that are completed before the tasks that depend on them.
- an unordered queue that ignores dependency information in tasks. For most basic cases these queues are ideal. (LocalUnordered as implementation detail this queue dispatches tasks in a FIFO order.)
- a limited size queue that keep a fixed number of completed jobs in memory, which is ideal for long-running background processes.
- priority queues that dispatch tasks according to priority order.
- remote queues that store all tasks in an external storage system (e.g. a database) to support architectures where multiple processes can service the same underlying queue.
The QueueGroup interface provides a mechanism to manage collections of queues. There are remote and local versions of the queue group possible, but these groups make it possible to create new queues at runtime, and improve the isolation of queues from each other.
Runners are the execution component of the worker pool, and are embedded within the queues, and can be injected at run time before starting the queue pool. The LocalWorkers implementation executes tasks in a fixed-size worker pool, which is the default of most queue implementations.
Additional implementation provide rate limiting, and it would be possible to implement runners which used the REST interface to distribute workers to a larger pool of processes, where existing runners simply use go routines.
The DependencyManager interface makes it possible for tasks to express relationships to each other and to their environment so that Job operations can noop or block if their requirements are not satisfied. The data about relationships between jobs can inform task ordering as in the LocalOrdered queue.
The handling of dependency information is the responsibility of the queue implementation.
The management package centers around a management interface that provides methods for reporting and safely interacting with the state of jobs.
The REST interface provides tools to submit jobs to an Amboy queue provided as a service. The rest package in Amboy provides the tools to build clients and services, although any client that can construct JSON formated Job object can use the REST API.
Additionally the REST package provides remote implementations of the management interface which makes it possible to manage and report on the jobs in an existing queue, and the abortable pool interface, that makes it possible to abort running jobs. These management tools can help administrators of larger amboy systems gain insights into the current behavior of the system, and promote safe and gentle operational interventions.
See the documentation of the REST package
The Logger package provides amboy.Queue backed implementation of the grip logging system's sender interface for asynchronous log message delivery. These jobs do not support remote-backed queues.
The following patterns have emerged during our use of Amboy.
Embed the job.Base
type in your amboy.Job implementations. This provides a number of
helpers for basic job defintion in addition to implementations of all
general methods in the interface. With the Base, you only need to
implement a Run() method and whatever application logic is required
for the task.
The only case where embedding the Base type may be contraindicated is in conjunction with the REST interface, as the Base type may require more complicated initialization processes.
If your core application operations are implemented in terms of
amboy.Jobs, then you can: execute them independently of queues by
calling the Run() method, use a locally backed queue for
synchronous operation for short running queues, and use a limited size
queue or remote-backed queue as part of a long running service.
Please submit pull requests or issues with additional examples of amboy use.
See the godoc API documentation <http://godoc.org/github.com/tychoish/amboy> for more information about amboy interfaces and internals.
Amboy is available for use under the terms of the Apache License (v2).
If you encounter a problem with amboy, or would like to see a feature added, please open an issue on the GitHub project!
All project automation is managed by a makefile, with all output captured in the build directory. Consider the following operations:
make build # runs a test compile make test # tests all packages make lint # lints all packages make test-<package> # runs the tests only for a specific packages make lint-<package> # lints a specific package make html-coverage-<package> # generates the coverage report for a specific package make coverage-html # generates the coverage report for all packages
The buildsystem also has a number of flags, which may be useful for more iterative development workflows:
RUN_TEST=<TestName> # specify a test name or regex to run a subset of tests RUN_COUNT=<num> # run a test more than once to isolate an intermittent failure RACE_DETECTOR=true # run specified tests with the race detector enabled.
These features are speculative and there's not estimated time for completion, but are provided here
- API Change: Remove or change the
Jobs()andJobStats()methods on the queue so that they don't return channels. Use either iterators or provide other mechanisms for supporting the higher level functionality that these methods support. - API Change: Replace the
Runner()method on the queue interface with aClose()method. - Feature: Add a queue implementation that job data jobs in a local, on-disk store, potentially using badger for the backing store so jobs can persist between process starts without depending on MongoDB.
- Refactor: Simplify the MongoDB-based queues, to avoid the (now internal) driver interface.