Routing incoming emails to Meteor (or node.js) server via Mailgun

Some time ago I was adding email messaging to sharewaste.com - a waste-reducing app that connects people with kitchen scraps to their composting neighbours. It turned out to be a tad bit trickier than originally expected.

You see, I started with the simplest thing: a "click here to reply" link in the emails. Naturally, it turned a lot of our users were trying to reply directly to the emails rather than following the link. Because people want convenience. And it makes much sense, right? After all, who has to time to follow reply links in email.

So I decided to implement email routing for ShareWaste so that users can reply to emails directly. And for posterity, here's how I went about it.

Quick Overview

We'll be using Mailgun as the email provider in this example. That said, the approach would be very much the same for any provider that offers webhooks for incoming emails (e.g. Sendgrid).

Here's what we'll do:

First, we'll set up a Mailgun "route" (an incoming traffic webhook with configurable rules).
Then we'll implement a routine in our app that sends out emails with a specific replyTo header value (so that we can match incoming replies with their respective "parent" messages)
Finally, we'll create an API endpoint that handles the incoming emails (POST requests from Mailgun).

We'll be using mg.sharewaste.com for the domain (but obviously you'd use your own app's domain instead).

Let's look at these steps in a little more detail.

Mailgun Config

After you've signed up with Mailgun and created the domain (mg.sharewaste.com in our case), go to the routes tab and create a new route. Use the following settings:

Expression type: Match recipient
Recipient: .*_reply@mg.sharewaste.com
Forward actions: add https://sharewaste.com/api/email/reply
Fill in the route description (e.g. "email reply handler")
Hit Create route.

Now any email that goes to .*_reply@mg.sharewaste.com will trigger this route which means Mailgun will send an HTTP POST request to https://sharewaste.com/api/email/reply (which we specified in Forward actions) with the parsed email data in payload. Sweet as!

Next, we need two things: a) consume Mailgun's payload; and b) figure out which message is being replied to. Let's start with the latter.

Email sending logic

In order to match the incoming messages with their thread, we'll use a Mongo collection to store the mapping between the incoming email and the thread it belongs to. Let's call the collection mailRepliesMapping.

Here's what the documents will look like:

interface IMailData {  
  _id: string;

  // Id of the thread the email reply belongs to.
  threadId: string;

  // Id of the recipient of the email (i.e., of
  // the user who will send the reply).
  replySenderId: string;

  // Created at date, 'nuff said.
  createdAt: Date; 
}

And here's a code snippet of what we'll do when a new message is added to a thread. (With Meteor, sending emails is pretty easy - just use the Email.send function. But with vanilla Node it's really not much harder.)

function sendEmailForPost(post) {  
  // Get thread, get recipients, prepare email data.
  // ...


  const emailMappingDocId = MailDataCollection.insert({
    threadId: thread._id,
    replySenderId: emailRecipient._id,
    createdAt: new Date()
  });

  Email.send({
    // ...
  })
}

The idea is that every time Mailgun tells us we have a new reply, we'll find out the relevant mailRepliesMapping and we'll fetch all the metadata we need in order to add the message where it belongs.

To make things easily configurable, we'll add a replyEmailSuffix field to our Meteor settings with the value _reply@mg.sharewaste.com. We'll use this constant to easily parse out our internal id from an incoming email and lookup the relevant mailRepliesMapping document.

When we send our email out, we'll use the following replyTo header value:

const replyEmailSuffix = Meteor.settings['replyEmailSuffix'];  
const replyTo = `${emailMappingDocId}${replyEmailSuffix}`

Now when the recipient replies, the email will be sent to an email address that looks like this: <my_document_id>_reply@mg.sharewaste.com. That way we're able to easily parse out the <my_document_id> section (as we describe in the next section).

Server Endpoint For Incoming Emails

The last bit is the "incoming email" handler. No curve ball there. Create a module file (e.g. imports/server/incoming-mail-handler.js) that looks like this:

import express from 'express';  
import bodyParser from 'body-parser';  
import { WebApp } from 'meteor/webapp';  
import { parseOneAddress } from 'email-addresses';  
import ThreadsApi from '../../api/threads';

// We need `multer` middleware because sending an email
// with attachments will produce a multipart message.
import multer from 'multer';

import { MailDataCollection } from '../../api/mailData';

// Create and set up Express app.
// Note: You would just use Express `Router` in
// production and create the Express App object in another module.
export const app = express();  
export const router = express.Router();  
const upload = multer();

// The incoming email handler, deals with Mailgun webhook payload.
router.post('/api/email/reply', upload.any(), Meteor.bindEnvironment((req, res) => {  
  console.log('MailHandler: processing email');

  // Parse out the email mapping document id from the email address.
  const replyEmailSuffix = Meteor.settings['replyEmailSuffix'];
  const parsedTo = parseOneAddress(req.body.To || req.body.to);
  const dataKey = parsedTo.address.replace(replyEmailSuffix, '');
  const data = MailDataCollection.findOne({ _id: dataKey });

  if (!data) {
    console.error(`Mailhandler: invalid mail reply key ${dataKey}`);
    return res
      .status(400)
      .send(`Invalid mail reply key ${dataKey}`);
  }

  const emailText = `${req.body['stripped-text']}\n\n${req.body['stripped-signature']}`;
  const threadId = data.threadId;
  if (!threadId) {
    console.error(`Mailhandler: invalid thread id ${threadId}`);
    return res
      .status(400)
      .send(`Invalid thread id ${threadId}`);
  }

  const sender = Meteor.users.findOne({ _id: data.replySenderId });
  if (!sender) {
    console.error(`Mailhandler: invalid sender ${data.replySenderId}`);
    return res
      .status(400)
      .send(`Invalid sender ${data.replySenderId}`);
  }

  const post = ThreadsApi.insertPost(emailText, threadId, sender);

  // Don't forget to send an email notification to the actual recipient.
  ThreadsApi.sendEmailForPost(post._id, sender);

  // In case we want to track which messages have been replied to via email.
  MailDataCollection.update({ _id: dataKey }, { $set: { emailReplyRouted: true } });

  // All done!
  res.send('ok');
}));

// Again, in production app, you'd use Express Router
// and create the app in another module.
app.use(bodyParser.urlencoded({  
  extended: true
}));
app.use(router);

// Wire up Meteor.
WebApp.connectHandlers.use(app);

And that's it! Now every time any of our users replies to a message directly from their email client, we'll route the reply through Mailgun to our server which will append the message to the right thread and will also send an email notification to the recipient.

A cool feature of Mailgun is that if your server is down, Mailgun will try to re-deliver the payload a number of times (with semi-exponential backoff) in the next eight hours before giving up.

Attachments

Please note that in our example we don't handle attachments - they will simply be ignored. Inline attachments will just show as string ids in the email body. Parsing attachments wouldn't be very hard though (although you'd need to upload them somewhere); check out Mailgun's Documentation to see how to get them from the payload.

Summary

We've learnt to route incoming emails to our Meteor app via Mailgun. Here's what we covered:

When we send a message from our app, we use a special replyTo header value with a unique identifier for each email.
When a user replies to an email directly from the email client, the message will be routed to our server via Mailgun
When we detect an incoming email reply, we parse the email address to get the email's unique identifier. Then we look up the relevant metadata in our database and we figure out who sent the message and which thread the message should be appended to.
If our server is down momentarily, Mailgun will re-send the notification a couple times (so we don't lose messages).

And that's all there is to it. Not that hard, right?

Happy routing! I'm @tomas_brambora.

PS: If you're into composting or if you simply think that composting scraps - and producing soil - is better than just chucking them into garbage, be sure check out ShareWaste. And spread the word, the more, the merrier! :-)

Ideas Into Software

Routing Emails Through Meteor Server