💾 Archived View for wilw.capsule.town › log › 2021-12-05-handling-incoming-mail.gmi captured on 2023-09-08 at 15:58:36. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-04-19)
-=-=-=-=-=-=-
Most applications include some sort of outbound transactional email as part of their normal function. These email messages could be to support account-level features (such as password-resets) or to notify the user about activity relevant to them.
In the latter case, such emails might be read, and then archived or deleted by the user, without further direct action. They aren't typically designed to be something one actions or replies to - they're mostly there to bring you back into engaging with the platform.
However, what if your users could interact with your platform via email to some extent? For example, if someone were to send you a message on the platform - and you receive an email notifying you about the message, along with its contents - then the natural thing would be for you to be able to click "reply" to the email and write a message back.
It is probably kinder to the user to provide this option, rather than always sending from a `noreply@` type address (since everyone has email clients wherever they go), and it also means that your service sees more interaction (even if indirectly).
Luckily, many current transactional email providers make it quite straight-forward to support this sort of functionality.
I will talk about Sendgrid [1] as a transactional mail provider for the benefit of this post, and will use a Python Flask app demo for handling the inbound webhooks from Sendgrid.
Essentially, the flow will be the following:
1. User "A" sends user "B" a message in our platform
1. The platform sends an email to user "B" containing the message, with a special `Reply-To` header
1. User "B" writes a message back to user "A" by directly replying to the email
1. Sendgrid processes the incoming email and sends a `POST` request with the details back to our platform
1. Our platform receives the webhook from Sendgrid, reads the message and headers and takes action.
The first thing to do is to set-up a mail provider we can use for outgoing and incoming email messages. As mentioned, in this post we'll discuss Sendgrid, but others would also do the job (such as Mailgun [2]).
Set up an account on Sendgrid and validate the domain name you'll use to carry the email messages via your DNS manager. In this post we'll assume the domain is `mail.example.com`. Copy and paste the needed connection details (such as API tokens) and configure your app to send mail in the documented way (I won't cover that in this post, as it is out of scope).
You'll also need to configure your domain's DNS with an MX record to enable Sendgrid to receive email sent to addresses at your domain. The details are as follows:
Next, we need to tell Sendgrid to handle the inbound messages. For this, navigate to Settings -> Inbound Parse on the Sendgrid dashboard (or use this direct link [3]). Click "Add Host & URL" and enter the details as needed. In the example below, I have configured Sendgrid to handle all email sent to `anything@mail.example.com` and to `POST` the parsed email to our app (https://app.example.com) at the `/webhooks/email` route.
Once that's done, we can close the Sendgrid manager.
Next, we need to make a change to the way we send our notification emails to users. For now, we'll just focus on the "new message" event, but the same process could also be applied to other types of events in our platform.
The `Reply-To` field of an email message contains an email address, and tells standards-compliant email clients to use this address (instead of the `From` address) when users click the "Reply" button on the email.
We will need to add this field to outbound emails notifying about new messages such that Sendgrid can pick them up and we can process them later. There are various ways to achieve this, depending on your existing approach to sending transactional mail, but the below is an example in Python (given we know the `message_id` of the message we are notifying about):
msg = MIMEMultipart('alternative')
msg.attach(MIMEText(text, 'plain')) # add other fields to the email
msg['Subject'] = subject # add a subject line
msg['To'] = ', '.join(to) # add "to" addresses (there can be more than one!)
msg['Reply-To'] = f'message.{message_id}@mail.example.com'
# Continued mail logic...
In this example, we have specified a `Reply-To` address that will look a little like this: `message.xxxyyyzzz@mail.example.com`. This means that when a reply to this email comes in, we can use the `To` field of the reply email to work out what the message is about - in this case a reply to the `message` with an `id` of `xxxyyyzzz`. We'll cover this later.
Since we've already configured Sendgrid to receive all emails sent to `@mail.example.com`, then we know that it will try and `POST` the data back to our app once they've been received.
The final step is to ensure our app can handle the inbound webhooks from Sendgrid.
As we saw earlier, we told Sendgrid to send the details of incoming emails to our app's `/webhooks/email` route, and so we will need to set that up. We also know that it will be a `POST` request and the payload will be standard multipart form data.
Below I have included an example of how we might handle this in a Python Flask app, but the same should be similar for other frameworks and languages:
@app.route('/webhooks/email', methods=['POST'])
def parse_email():
to_address = request.form.get('to')
from_address = request.form.get('from')
text = request.form.get('text')
subject = request.form.get('subject')
print(to_address, from_address, subject)
return jsonify({'processed': True})
We could then deploy our app, send a couple of emails to addresses the app can receive (e.g. `test.1234@mail.example.com`) and view the app's logs to make sure everything is coming through OK.
Once we're happy with the basic setup, we can make changes to the `parse_email` function to actually do something useful. For example:
# function definition, etc.
email_user = to_address.split('@')[0]
email_topic = email_user.split('.')[0] # `message`
if email_topic == 'message': # Branch code depending on the type of message
email_topic_id = email_user.split('.')[1] # `xxxyyyzzz`
message = db.getMessageWithId(email_topic_id) # Replace with your own database logic
from_user = db.getUserWithEmail(from_address) # Get the originator user
if not message or not from_user:
# Handle not found cases
if from_user.id not in message.participants:
# Handle case where the user isn't allowed to reply to the message
db.createNewMessage(from_user.id, text, in_reply_to = message.id) # Create the message
return jsonify({'processed': True})
In this example we have included some checks to ensure the user exists and has the permission needed to reply to the message. We parse the relevant message details from the `From` field as discussed in the previous section.
We've now run through the complete end-to-end set-up; users can now click "reply" to emails in order to send content back through to our platform.
Of course, this approach isn't perfect. For example, changes would be needed to handle the following:
However, hopefully this post has set the scene for how one might go about implementing such a system in their own software!