Tag: email

Why you’ve stopped receiving emails from your website

Email is terrible. Just don’t use it.

OK, that’s not realistic. Unfortunately. So I’ll address the question, and offer somewhat of an explanation.

First off, you need to understand how email works. I don’t think a lot of people do. Email doesn’t just leave your device, shoot across the Internet, and land directly in the recipient’s inbox. There are servers involved on both ends. (A server is a specialized computer sitting in a large data center that runs software for purposes like this.)

When you hit “Send,” your message goes from your device to your sending mail server. Then your sending mail server looks at the domain name on the recipient’s email address — the part after @ — and figures out where that domain’s receiving mail server is. It sends the message to that server.

Then, when the recipient wants to check their email, their device connects to their receiving mail server, which sends over the new messages. (The way this works is a lot different now than it was in the early days of the Internet, with the switch from POP3 to IMAP, but if you don’t know what those acronyms mean, just be thankful and move on. It’s not really relevant to this post.)

All of this is relatively straightforward when you’re directly creating the email message in your mail app. In most cases, your own sending and receiving server are one and the same. But it’s different when the email is coming from your website.

Next, you need to understand how email coming from a website works. When you’re getting an email such as an automated notification that someone has filled out a form on your website, who is “sending” the email? It’s not the site visitor who filled out the form. It’s the website itself. So the email doesn’t go through the visitor’s sending server. The website has to have its own sending server.

In the past (up until around early 2020), this was straightforward. Web servers — yes, another specialized computer in a data center, this time the one where your website “lives” — typically would also be running sending mail server software too. The most common early software for this purpose was called, creatively, Sendmail. But there were some serious shortcomings with Sendmail that needed to be fixed, and the replacement software was called, creatively, Postfix.

For years I would set up web servers running Postfix, and there was never any problem. A website user would submit a form, the site would generate a notification email and put it in the Postfix queue, Postfix would send it along to the recipient’s (i.e. the website administrator’s) receiving mail server, and the recipient would receive it.

So why did this stop working? In a word, spam. This kind of setup was very slick and easy to build. So easy, in fact, that hosting providers — especially the ones offering “Virtual Private Servers” where you can configure any software you want — became havens for spammers.

For years there were increasingly convoluted methods to validate these servers as legitimate senders. (Here are some more fun acronyms you don’t want to know: SPF, DKIM, DMARC.) But it was a cat-and-mouse game as spammers continually found ways around every new restriction.

Eventually, so many spammers started using hosting providers like Digital Ocean to send out their garbage, that large mail providers like Gmail and Microsoft Office 365 just decided it wasn’t worth dealing with, and started automatically flagging any email that originated anywhere on Digital Ocean’s network (to cite one example) as being spam.

Now, legitimate websites, sending only legitimate emails, are getting flagged as spam, solely because they happen to exist on the same network as spammers.

OK, so shouldn’t Digital Ocean (and other similar VPS providers) do something about this? Yes. Yes, they should. But instead they’ve decided to just throw up their hands and say, “you should not be using our network to send out email.” Other VPS providers like Linode actually just block port 25 altogether. (Again, if you don’t know what that is, just be happy and move on.)

Is there a solution? Yes. Stop using email. OK, I still know that’s not realistic. There is a solution, but it is cumbersome to set up. You need to configure your website to route outgoing emails through a real mail account on a real mail server. If you’re on WordPress, plugins like WP Mail SMTP and WPO365 can help — but bear in mind that this does mean connecting a real, actual email account to your website through these tools. (And an interesting side effect is that you’ll see these outgoing messages in the Sent folder in your mail app.)

Alternately, you can use a service like Amazon SES or Sendgrid. But however you choose to do it, there are extra configuration steps, extra technical knowledge required, and extra costs. What used to be straightforward and easy is now complicated and costly, and we have sleazy spammers and intransigent corporations to thank for it.

Google’s bad UX can even cause seasoned professionals to make novice mistakes

Today I did something that, when I realized what I had done, I metaphorically kicked myself over. It was so stupid. It reminded me of something I did at one of my earliest professional jobs… over 20 years ago.

I’ve been using email for nearly 30 years, and I’ve been a professional web developer for 25 of them. I know the difference between CC and BCC.

But today, when I was sending a mass email to a number of my clients, I made a critical mistake. I always handle these emails in the same way: I set the To field to a generic, non-existent email address on my own domain, and I put all of my clients’ email addresses — the real recipients — in the BCC field. That way, they all receive the email and, critically, they can’t see who else I sent it to.

Unfortunately, that’s not what happened today. Instead I unwittingly put all of their email addresses in the CC field. Sure, they still all received the email. But now they can also see who else received it, and, much worse, they can potentially hit Reply All and send their response to the entire list of recipients.

That’s not only embarrassing, but given the nature of the message, it could cause them to potentially blast some of my personal financial information out to a huge swath of my other clients.

Ugh.

I felt like a fool, and I nearly sent a second message (being sure to use BCC this time!) explaining my error… but then I realized that would just make me look like an even bigger fool, and the best thing to do was nothing, and just hope it goes away quietly. (So, of course, I’m writing a blog post about it.)

But the more I thought about it, the more I realized that the real culprit is Google’s bad UX design. For years I’ve been railing against Google’s “Material Design” or whatever they’re calling what they do these days. It’s too vague and unintuitive. Too much is hidden.

Good design should be obvious. Users should be able to see at a glance what they’re able to do with a piece of software. Of course in the early 2000s, Microsoft took that concept to an absurd extreme with the “Ribbon” in Office: a giant mosaic of every imaginable feature of the program, thrown together in a jumble of icons and text that would overwhelm anyone. Thankfully that approach has fallen by the wayside, but in its place is something arguably even worse: the illusion of simplicity, created by hiding so many features away that users probably don’t even know they exist, and then compounding the problem by stripping down the visual elements of the interface to such an extreme that it’s difficult to even know what’s clickable.

Such is the case with Gmail, something that should have a pretty simple interface, even with all features exposed.

I learned email back in college in the ’90s, using Eudora. Oh how I loved that program. Every mail client that has come since has been a downgrade, in my opinion. These days, practically speaking, my options are to use Apple’s Mail app, or Gmail’s web interface. For better or worse, I use Gmail. But in light of today’s debacle, I decided to do a comparison.

Since the days of Eudora, mail programs always had separate and distinct To, CC, and BCC fields, each on their own line. It made it very difficult to accidentally use the wrong field, and easy to tell if you were making a mistake. Apple Mail still does something very similar… with the modification that the BCC field is off by default, and you have to go to a menu to show it. Then it appears on its own line. All of which reinforces the deliberate choice of using BCC when you want it. All in all, it’s remarkably similar to what I remember from Eudora, but a bit cleaner.

In comparison, Gmail hides both of those fields by default, and the way to get one of them is to click the light gray text for the one you want, on the right side of the same line as the To field, right next to each other.

Oops! My cursor was a few pixels too far to the left — and since there’s no visible button, it’s not clear where exactly the clickable areas end — so I accidentally clicked CC without noticing.

And then, once you do click one of them, it appears on a new line, but again, if you’re moving quickly, as I regrettably was today, it was far too easy not to notice the mistake I had made. Since either CC or BCC doesn’t appear unless you’ve clicked it, you have to specifically look at the label on the left side of the line to know which one is on. That’s not possible in Mac Mail (or Eudora), where CC is always there, so BCC, if you’re using it, is always two lines below To.

The only way I realized what I had done today was when one of the clients replied to the email — thankfully he did not “reply all” — and I saw the “CC” dump of email addresses in my original email quoted at the bottom of his reply. Eek!

This is the current state of supposed “best practices” in UX design… flaws in things so basic, things that were already solved a generation ago, that someone who does this for a living makes novice mistakes.

Addendum, January 17, 2022: It gets worse. Over the past few days I have been working with some agency partners on a proposal for an RFP. There have been two glaring problems that have occurred as a direct result of Gmail’s interface quirks. First, I was waiting over the weekend for my partners to email me a link to their draft document. Late Friday, one of them emailed me a one-sentence message saying they’d send over the proposal on Saturday. I didn’t bother opening the email to read it, because I could read the entire thing in the preview. (This was in the Gmail app on my iPhone.) I waited Saturday and Sunday for another email from them, but I got none! Except, I did. But because Gmail only shows the first unread message in a thread, with no indication that there are more unread messages in the thread, I had no idea that they actually had sent another email until this morning, when I took the time to open the email on my desktop.

Then, to make matters worse, I was just preparing a new email to send them, with my latest draft, and as I entered their email addresses in the “To” field, Gmail suggested the RFP client as another recipient. No no no no no. It would have been far too easy, if I were in just slightly more of a hurry, for me to have accidentally clicked the client’s name, and sent them my draft of the proposal and the associated internal comments. Yikes. This wouldn’t be the first time its suggestions have led me astray… I’ve accidentally sent emails intended for the drummer in my band to a client, because they have the same first name and as soon as I started typing it, Gmail decided for me which person I was emailing and autocompleted the address.

SPF for dummies (i.e. me)

For a while I’ve known that (legitimate) outgoing email messages originating from my web server were occasionally not reaching their intended recipients. I also knew that there was a DNS change you could make to help prevent this problem, but I didn’t know any more about it and it was a marginal enough problem that I could just put it off.

Finally today I decided to deal with it. And I was (re)introduced to the SPF acronym. No, that’s not Sun Protection Factor, or Spray Polyurethane Foam, or even Single Point of Failure (although in my case perhaps that last one is accurate). No, it stands for Sender Policy Framework, and it’s an add-on to the core capabilities of DNS that provides a way to positively identify the originating servers of outgoing email messages.

My situation is simple: I have a domain name that needs to be able to send mail from either my mail server or my web server. Most of the tutorials I found for SPF were far too convoluted to address this simple arrangement. Then I found this post by Cyril Mazur which provided the very simple answer:

v=spf1 a mx ~all

Simply add the above as a new TXT record in your DNS zone file, and you should be set.