Skip to content

Commit

Permalink
API Update API to reflect changes to CLI interaction (#445)
Browse files Browse the repository at this point in the history
  • Loading branch information
GuySartorelli committed Sep 26, 2024
1 parent 0ee9911 commit ba77548
Show file tree
Hide file tree
Showing 17 changed files with 287 additions and 275 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ composer require symbiote/silverstripe-queuedjobs
Setup a cron job:

```sh
*/1 * * * * /path/to/silverstripe/vendor/bin/sake dev/tasks/ProcessJobQueueTask
*/1 * * * * /path/to/silverstripe/vendor/bin/sake tasks:ProcessJobQueueTask
```

* To schedule a job to be executed at some point in the future, pass a date through with the call to queueJob
Expand Down Expand Up @@ -252,7 +252,7 @@ In addition to the config setting there is a task that can be used with a cron t
detected:

```
*/5 * * * * /path/to/silverstripe/vendor/bin/sake dev/tasks/CheckJobHealthTask
*/5 * * * * /path/to/silverstripe/vendor/bin/sake tasks:CheckJobHealthTask
```
## Special job variables

Expand Down
6 changes: 6 additions & 0 deletions _config/cli.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
Name: queuedjobs-cli
---
SilverStripe\Cli\Sake:
commands:
- 'Symbiote\QueuedJobs\Cli\ProcessJobQueueChildCommand'
7 changes: 3 additions & 4 deletions _config/taskrunner.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@ After:
- DevelopmentAdmin
---
SilverStripe\Dev\DevelopmentAdmin:
registered_controllers:
controllers:
tasks:
controller: Symbiote\QueuedJobs\Controllers\QueuedTaskRunner
links:
tasks: 'See a list of build tasks to run (QueuedJobs version)'
class: Symbiote\QueuedJobs\Controllers\QueuedTaskRunner
description: 'See a list of build tasks to run (QueuedJobs version)'
2 changes: 1 addition & 1 deletion docs/en/defining-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
The best way to learn about defining your own jobs is by checking the examples

* `PublishItemsJob` - A job used to publish all the children of a particular node. To create this job, run the PublishItemsTask passing in the parent as a request var (eg ?parent=1)
* `GenerateGoogleSitemapJob` - A job used to create a google sitemap. If the googlesitemaps module is installed it will include priority settings as defined there, otherwise just produces a generic structure. To create an initial instance of this job, call dev/tasks/CreateDummyJob?name=GenerateGoogleSitemapJob. This will create the initial job and queue it; once the job has been run once, it will automatically schedule itself to be run again 24 hours later.
* `GenerateGoogleSitemapJob` - A job used to create a google sitemap. If the googlesitemaps module is installed it will include priority settings as defined there, otherwise just produces a generic structure. To create an initial instance of this job, run `sake tasks:CreateDummyJob --name=Symbiote\QueuedJobs\Jobs\GenerateGoogleSitemapJob`. This will create the initial job and queue it; once the job has been run once, it will automatically schedule itself to be run again 24 hours later.
* `CreateDummyJob` - A very simple skeleton job.

## API Overview
Expand Down
10 changes: 5 additions & 5 deletions docs/en/immediate-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,12 @@ inotifywait and then call the task when a new job is ready to run.
SILVERSTRIPE_ROOT=/path/to/silverstripe
SILVERSTRIPE_CACHE=/path/to/silverstripe-cache
inotifywait --monitor --event attrib --format "php $SILVERSTRIPE_ROOT/vendor/bin/sake dev/tasks/ProcessJobQueueTask job=%f" $SILVERSTRIPE_CACHE/queuedjobs | sh
inotifywait --monitor --event attrib --format "php $SILVERSTRIPE_ROOT/vendor/bin/sake tasks:ProcessJobQueueTask --job=%f" $SILVERSTRIPE_CACHE/queuedjobs | sh
```

You can also turn this into an `init.d` service:

```
```sh
#!/bin/bash
#
# /etc/init.d/queue_processor
Expand Down Expand Up @@ -73,7 +73,7 @@ start() {
for PATH in ${SILVERSTRIPE_ROOT[@]};
do
INOTIFY_OPTS="--monitor --event attrib -q"
INOTIFY_ARGS="--format 'php ${PATH}/vendor/bin/sake dev/tasks/ProcessJobQueueTask job=%f' ${PATH}/silverstripe-cache/queuedjobs | /bin/sh"
INOTIFY_ARGS="--format 'php ${PATH}/vendor/bin/sake tasks:ProcessJobQueueTask job=%f' ${PATH}/silverstripe-cache/queuedjobs | /bin/sh"
daemon --user apache /usr/bin/inotifywait ${INOTIFY_OPTS} ${INOTIFY_ARGS} &
/bin/touch /var/lock/subsys/queue_processor
done
Expand Down Expand Up @@ -113,7 +113,7 @@ Similar concept to `inotifywait`, but with the `lsyncd` system utility.

The following is an example config `/etc/lsyncd.conf`

```
```sh
-- Queue Processor configuration, typically placed in /etc/lsyncd.conf
settings = {
Expand All @@ -124,7 +124,7 @@ settings = {
-- Define the command and path for the each system being monitored here, where webuser is the user your webserver
-- runs as
runcmd = "/sbin/runuser webuser -c \"/usr/bin/php /var/www/sitepath/framework/cli-script.php dev/tasks/ProcessJobQueueTask job=\"$1\" /var/www/sitepath/framework/silverstripe-cache/queuedjobs\""
runcmd = "/sbin/runuser webuser -c \"/usr/bin/php /var/www/sitepath/vendor/bin/sake tasks:ProcessJobQueueTask job=\"$1\" /var/www/sitepath/framework/silverstripe-cache/queuedjobs\""
site_processor = {
onCreate = function(event)
Expand Down
16 changes: 8 additions & 8 deletions docs/en/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,14 @@
Install the cronjob needed to manage all the jobs within the system. It is best to have this execute as the
same user as your webserver - this prevents any problems with file permissions.

```
*/1 * * * * /path/to/silverstripe/vendor/bin/sake dev/tasks/ProcessJobQueueTask
```sh
*/1 * * * * /path/to/silverstripe/vendor/bin/sake tasks:ProcessJobQueueTask
```

To test things are working, run the following command to create a dummy task

```
vendor/bin/sake dev/tasks/CreateQueuedJobTask
```sh
vendor/bin/sake tasks:CreateQueuedJobTask
```

Every job is tracked as a database record, through `QueuedJobDescriptor` objects.
Expand All @@ -31,8 +31,8 @@ Open up `/admin/queuedjobs` in a browser.
You should see the new job with `Status=New`.
Now either wait for your cron to execute, or trigger execution manually.

```
vendor/bin/sake dev/tasks/ProcessJobQueueTask
```sh
vendor/bin/sake tasks:ProcessJobQueueTask
```

The job should now be marked with `Status=Completed`.
Expand Down Expand Up @@ -116,8 +116,8 @@ Symbiote\QueuedJobs\Jobs\CleanupJob:
If your code is to make use of the 'long' jobs, ie that could take days to process, also install another task
that processes this queue. Its time of execution can be left a little longer.

```
*/15 * * * * /path/to/silverstripe/vendor/bin/sake dev/tasks/ProcessJobQueueTask queue=large
```sh
*/15 * * * * /path/to/silverstripe/vendor/bin/sake tasks:ProcessJobQueueTask queue=large
```

From your code, add a new job for execution.
Expand Down
38 changes: 38 additions & 0 deletions src/Cli/ProcessJobQueueChildCommand.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
<?php

namespace Symbiote\QueuedJobs\Cli;

use Symbiote\QueuedJobs\Services\QueuedJobService;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

#[AsCommand('queuedjobs:process-queue-child', hidden: true)]
class ProcessJobQueueChildCommand extends Command
{
protected function execute(InputInterface $input, OutputInterface $output): int
{
$task = @unserialize(@base64_decode($input->getArgument('base64-task')));
if ($task) {
$this->getService()->runJob($task->getDescriptor()->ID);
}
return Command::SUCCESS;
}

/**
* Returns an instance of the QueuedJobService.
*
* @return QueuedJobService
*/
protected function getService()
{
return QueuedJobService::singleton();
}

protected function configure()
{
$this->addArgument('base64-task', InputArgument::REQUIRED);
}
}
42 changes: 12 additions & 30 deletions src/Controllers/QueuedTaskRunner.php
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@
use Symbiote\QueuedJobs\Services\QueuedJobService;
use Symbiote\QueuedJobs\Tasks\CreateQueuedJobTask;
use Symbiote\QueuedJobs\Tasks\DeleteAllJobsTask;
use Symbiote\QueuedJobs\Tasks\ProcessJobQueueChildTask;
use Symbiote\QueuedJobs\Tasks\ProcessJobQueueTask;

/**
Expand All @@ -29,55 +28,34 @@
*/
class QueuedTaskRunner extends TaskRunner
{
/**
* @var array
*/
private static $url_handlers = [
private static array $url_handlers = [
'queue/$TaskName' => 'queueTask',
];

/**
* @var array
*/
private static $allowed_actions = [
private static array $allowed_actions = [
'queueTask',
];

/**
* @var array
*/
private static $css = [
private static array $css = [
'symbiote/silverstripe-queuedjobs:client/styles/task-runner.css',
];

/**
* Tasks on this list will be available to be run only via browser
*
* @config
* @var array
* Tasks on this list will not be available to run via the jobs queue
*/
private static $task_blacklist = [
private static array $task_blacklist = [
ProcessJobQueueTask::class,
ProcessJobQueueChildTask::class,
CreateQueuedJobTask::class,
DeleteAllJobsTask::class,
];

/**
* Tasks on this list will be available to be run only via jobs queue
*
* @config
* @var array
*/
private static $queued_only_tasks = [];
private static array $queued_only_tasks = [];

public function index()
{
if (Director::is_cli()) {
// CLI mode - revert to default behaviour
return parent::index();
}

$baseUrl = Director::absoluteBaseURL();
$tasks = $this->getTasks();

Expand Down Expand Up @@ -108,6 +86,8 @@ public function index()
'Title' => $task['title'],
'Description' => $task['description'],
'Type' => 'universal',
'Parameters' => $task['parameters'],
'Help' => $task['help'],
]));
}

Expand All @@ -118,18 +98,20 @@ public function index()
'Title' => $task['title'],
'Description' => $task['description'],
'Type' => 'immediate',
'Parameters' => $task['parameters'],
'Help' => $task['help'],
]));
}

// Queue only tasks
$queueOnlyTaskList = ArrayList::create();

foreach ($queuedOnlyTasks as $task) {
$taskList->push(ArrayData::create([
'QueueLink' => Controller::join_links($baseUrl, 'dev/tasks/queue', $task['segment']),
'Title' => $task['title'],
'Description' => $task['description'],
'Type' => 'queue-only',
'Parameters' => $task['parameters'],
'Help' => $task['help'],
]));
}

Expand Down
10 changes: 7 additions & 3 deletions src/Jobs/RunBuildTaskJob.php
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,16 @@
use SilverStripe\Control\HTTPRequest;
use SilverStripe\Core\Injector\Injector;
use SilverStripe\Dev\BuildTask;
use SilverStripe\PolyExecution\PolyOutput;
use SilverStripe\ORM\DataObject;
use Symbiote\QueuedJobs\Services\AbstractQueuedJob;
use Symbiote\QueuedJobs\Services\QueuedJob;
use Symfony\Component\Console\Input\ArrayInput;

/**
* A convenience wrapper for running BuildTask implementations.
* These are usually executed via synchronous web request
* or synchronous CLI execution (under dev/tasks/*).
* or synchronous CLI execution.
*
* Caution: This job can't increment steps. This is a signal
* for job health checks that a job should be considered stale
Expand Down Expand Up @@ -83,8 +85,10 @@ public function process()

$getVars = [];
parse_str($this->QueryString ?? '', $getVars);
$request = new HTTPRequest('GET', '/', $getVars);
$task->run($request);
$output = PolyOutput::create(PolyOutput::FORMAT_ANSI);
$input = new ArrayInput($getVars);
$input->setInteractive(false);
$task->run($input, $output);

$this->currentStep = 1;
$this->isComplete = true;
Expand Down
61 changes: 38 additions & 23 deletions src/Tasks/CheckJobHealthTask.php
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,21 @@

namespace Symbiote\QueuedJobs\Tasks;

use Exception;
use SilverStripe\Control\HTTPRequest;
use Composer\Console\Input\InputOption;
use Psr\Log\LoggerInterface;
use SilverStripe\Core\Injector\Injector;
use SilverStripe\Dev\BuildTask;
use Symbiote\QueuedJobs\Services\QueuedJob;
use SilverStripe\PolyExecution\PolyOutput;
use Symbiote\QueuedJobs\Services\AbstractQueuedJob;
use Symbiote\QueuedJobs\Services\QueuedJobService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;

class CheckJobHealthTask extends BuildTask
{
/**
* {@inheritDoc}
* @var string
*/
private static $segment = 'CheckJobHealthTask';
protected static string $commandName = 'CheckJobHealthTask';

/**
* {@inheritDoc}
* @return string
*/
public function getDescription()
public static function getDescription(): string
{
return _t(
__CLASS__ . '.Description',
Expand All @@ -32,30 +28,49 @@ public function getDescription()
/**
* Implement this method in the task subclass to
* execute via the TaskRunner
*
* @param HTTPRequest $request
* @return
*
* @throws Exception
*/
public function run($request)
protected function execute(InputInterface $input, PolyOutput $output): int
{
$queue = $request->requestVar('queue') ?: QueuedJob::QUEUED;
$queue = AbstractQueuedJob::getQueue($input->getOption('queue'));
if ($queue === null) {
$output->writeln('<error>queue must be one of "immediate", "queued", or "large"</>');
return Command::INVALID;
}
$jobHealth = $this->getService()->checkJobHealth($queue);

$unhealthyJobCount = 0;

foreach ($jobHealth as $type => $IDs) {
$count = count($IDs ?? []);
echo 'Detected and attempted restart on ' . $count . ' ' . $type . ' jobs';
$output->writeln('Detected and attempted restart on ' . $count . ' ' . $type . ' jobs');
$unhealthyJobCount = $unhealthyJobCount + $count;
}

if ($unhealthyJobCount > 0) {
throw new Exception("$unhealthyJobCount jobs are unhealthy");
$msg = "$unhealthyJobCount jobs are unhealthy";
/** @var LoggerInterface $logger */
$Logger = Injector::inst()->get(LoggerInterface::class . '.errorhandler');
$Logger->error($msg);
$output->writeln($msg);
return Command::FAILURE;
}

echo 'All jobs are healthy';
$output->writeln('All jobs are healthy');
return Command::SUCCESS;
}

public function getOptions(): array
{
return [
new InputOption(
'queue',
null,
InputOption::VALUE_REQUIRED,
'The queue to check',
'queued',
['immediate', 'queued', 'large']
),
];
}

protected function getService()
Expand Down
Loading

0 comments on commit ba77548

Please sign in to comment.