Skip to content

Commit

Permalink
perf(scheduler): create tasks in parallel (NangoHQ#2373)
Browse files Browse the repository at this point in the history
## Describe your changes

Tasks are currently created by scheduler one by one. We can parallelize
the tasks creation to speed up the scheduling transaction

## Checklist before requesting a review (skip if just adding/editing
APIs & templates)
- [ ] I added tests, otherwise the reason is: 
- [ ] I added observability, otherwise the reason is:
- [ ] I added analytics, otherwise the reason is:
  • Loading branch information
TBonnin authored Jun 19, 2024
1 parent 0aabb55 commit 3516678
Showing 1 changed file with 11 additions and 6 deletions.
17 changes: 11 additions & 6 deletions packages/scheduler/lib/workers/scheduling/scheduling.worker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,8 @@ export class SchedulingChild {
if (schedules.isErr()) {
return Err(`Failed to get due schedules: ${stringifyError(schedules.error)}`);
} else {
for (const schedule of schedules.value) {
const task = await tasks.create(trx, {
const createTasks = schedules.value.map((schedule) =>
tasks.create(trx, {
scheduleId: schedule.id,
startsAfter: new Date(),
name: `${schedule.name}:${new Date().toISOString()}`,
Expand All @@ -116,11 +116,16 @@ export class SchedulingChild {
createdToStartedTimeoutSecs: schedule.createdToStartedTimeoutSecs,
startedToCompletedTimeoutSecs: schedule.startedToCompletedTimeoutSecs,
heartbeatTimeoutSecs: schedule.heartbeatTimeoutSecs
});
if (task.isErr()) {
logger.error(`Failed to create task for schedule: ${schedule.id}`);
})
);
const res = await Promise.allSettled(createTasks);
for (const taskRes of res) {
if (taskRes.status === 'rejected') {
logger.error(`Failed to schedule task: ${taskRes.reason}`);
} else if (taskRes.value.isErr()) {
logger.error(`Failed to schedule task: ${stringifyError(taskRes.value.error)}`);
} else {
taskIds.push(task.value.id);
taskIds.push(taskRes.value.value.id);
}
}
}
Expand Down

0 comments on commit 3516678

Please sign in to comment.