Job Queuing 101: Start using Bull in your Node.js Project (Part II)

Let’s continue adding functionalities to our queue now that we have all set up and working. If you haven’t checked out the first part, you can find it here:

Job Queuing 101: Start using Bull in your Node.js Project (Part I)

Remembering part I, what we did is to create a Koa server that will receive orders simulating an online store, and those orders will be enqueued and later on processed as we desire.

So what’s next

Add priorities to your jobs

Let’s start by implementing priority to our jobs. This is useful if jobs are of different kind, let’s say that one job is critical for our system and the other one is a not-so-important job, then by indicating priorities, the job processor will follow those priorities and process first the ones with higher priority and leave the other ones for later.

NOTE: Highest priority is 1, and lower the larger integer you use.

In this case for simplicity reasons, we will consider of higher priority, those orders that exceed 100€ in the amount to be paid, and set a lower priority for the ones with amounts equal or below to 100. So let’s do this in our processor like this:

src/queues/orders-queue.js

As you can see in the getJobPriority method the orders with lowest priority will be the ones with no amount, then the ones with amounts equal or lower than 100€ and the rest will have the highest priority.

Add retries in case of failure

This functionality is a key one, as we can have scenarios where our order fails to be processed due to some reason but we want to give them another try, as we don’t want to loose those orders.

In this tutorial we will not show the scenario in which a job fails, but let’s add the attempts option to indicate that a job must have 2 attempts for being processed.

src/queues/orders-queue.js

To illustrate a failing scenario, lets imagine that what our process function does is to call an external API to get some extra data about the order that we are processing. If for some reason, this external API is down at a given time, we want the job processor to try again x attempts in case the API is up again.

Other options to be added

Among other options that we could implement there is an option to delay jobs, that is, jobs are delayed a certain amount of time before they will be processed.

Also, we could add jobs that repeat themselves indefinitely or until a given maximum date or the number of repetitions has been reached, according to a cron specification or a time interval.

Add a queue dashboard for some visuals

Before starting, make sure you installed bull-board by executing:

$ npm i @bull-board/koa // Koa adapter for bull-board

Now, we can start the magic. In our index.js file, we will import the Koa adapter from bull-board like this:

src/index.js

What we do here is to setup the base path for the route that will show the dashboard. Afterwards, we will create magically the actual dashboard, indicating in an array, the queues to monitor in our dashboard. In our case, our ordersQueue:

src/index.js

Finally, we will let our app know that it has to use the route created by bull-board in order to so the dashboard:

src/index.js

Putting everything together to clear our minds will be something like this:

src/index.js

And it is time to reveal our beautiful UI…. 🥁🥁🥁🥁🥁

Queue dashboard

All we need to know is to go to the browser and visit http://localhost:3000/admin.

Bull-board will redirect us to the route that it created which will add /queue/orders in our case. And we can see that we can check in individual tabs, the jobs that failed, that were completed, etc.

And there you go! Now you have implemented your first job queue with a cool UI to monitor it. Hope you enjoyed the full tutorial! 😃

You can check the code here: https://github.com/rluque8/job-queuing-bull