Scheduling tasks to be executed later is a pretty common scenario in backend development. Say you want to send a welcome followup email to a new user 24 hours after they sign up for your service. There’s a couple ways to approach this.

Bad approach - using a timeout

Pretty simple: 

User.onSignup((user) => {  
  setTimeout(() => {
    sendFollowupEmailToUser(user);
  }, FOLLOWUP_EMAIL_DELAY_MS);
})

Also: wrong.

Don’t do this. If the server restarts, the delayed task is lost; and since we’re in the age of clouds, you should expect your server to restart anytime and surprisingly often (see e.g. Heroku dyno cycling). It’s wrong if you wait just for a couple seconds, but it’s even wronger (read: pretty much useless) if you wait twenty four hours or longer.

Better approach - periodic polling

Periodically fetch for new users and send them the email. Here's the pseudocode: 

setInterval(() => {  
  const dayAgo = moment().subtract(24, ‘h’).toDate(); 
  const users = Users.find({
    createdAt: { $gt: dayAgo},
    welcomeEmailSent: { $exists: false }
  }).fetch();
  users.each((user) => {
    try {
      sendFollowupEmailToUser(user);
      Users.update({ _id: user._id}, { $set: { welcomeEmailSent: true } });
    } catch (e) {
      logger.error(‘Failed to send welcome followup email’, e);
    }
  })
}, POLLING_INTERVAL_MS);

This approach is better in that it is robust with regards to server restarting. But it’s quite wordy. And you have to manually keep track of which users have been sent the welcome email which is slightly annoying and it pollutes the user document (or becomes more annoying if you decide to keep the flag outside of the user document).

Best approach - delayed task in a task queue

Here we go: 

const WELCOME_FOLLOWUP_EMAIL_TASK_KEY = ‘sendWelcomeFollowupEmail’;

User.onSignup((user) => {  
  const delay = moment.duration(24, ‘h’).asMilliseconds();
  const payload = { userId: user._id };
  jobQueue.addJob(WELCOME_FOLLOWUP_EMAIL_TASK_KEY, {
    delay,
    maxRetries: 3,
    backoff: ‘exponential’
  }, payload);
});

jobQueue.processTask(WELCOME_FOLLOWUP_EMAIL_TASK_KEY, (data) => {  
  const { userId } = data;
  const user = Users.findOne({ _id: userId });
  sendFollowupEmailToUser(user);
});

I left out the necessary ceremonies to set up the task queue. Which can be very straightforward (or not - depends on the queue you’re using).

This approach is robust w/r to server restarting and we’ve “outsourced” the flag-keeping to the job queue itself, i.e., existing pending task means we haven’t sent the email yet. Also - depending on the task queue - we get (limited) retries, possibly with exponential back-off, for free.

You can choose from a number of task queues, here's just a few: Kue, Agenda, IronMQ, JobCollection - if you're into Meteor - or RabbitMQ)

What to look out for with delayed jobs

Mainly two things:

  1. Persisted delayed jobs introduce backwards compatibility issues into your system (your server code might have changed since the job was scheduled).

  2. If the delayed action is conditional, you need to make sure to check abort-condition when processing the task (e.g. sending invitation reminder only makes sense if the invited user hasn’t joined yet)

With regards to problem #1, to minimise the compatibility issues, you should always pass as little information in the payload as possible and fetch whatever you need when the job executes; e.g., pass document identifiers instead of whole documents. And if you must make breaking changes, start versioning the jobs, either by adding version field to the payload or by modifying the job key itself. Then just wait out till all old-style jobs are gone and clean up your code.

Problem #2 is fairly self-explanatory. If you use a delayed job, you schedule the job for everyone and then you need to check if the job still makes sense when it actually executes. So, e.g., in sendReminder job you need to make sure it still makes sense to actually remind the user at the time the job runs.

Happy scheduling! I’m @tomas_brambora