Tag Archives: Spam

Why you’ve stopped receiving emails from your website

Email is terrible. Just don’t use it.

OK, that’s not realistic. Unfortunately. So I’ll address the question, and offer somewhat of an explanation.

First off, you need to understand how email works. I don’t think a lot of people do. Email doesn’t just leave your device, shoot across the Internet, and land directly in the recipient’s inbox. There are servers involved on both ends. (A server is a specialized computer sitting in a large data center that runs software for purposes like this.)

When you hit “Send,” your message goes from your device to your sending mail server. Then your sending mail server looks at the domain name on the recipient’s email address — the part after @ — and figures out where that domain’s receiving mail server is. It sends the message to that server.

Then, when the recipient wants to check their email, their device connects to their receiving mail server, which sends over the new messages. (The way this works is a lot different now than it was in the early days of the Internet, with the switch from POP3 to IMAP, but if you don’t know what those acronyms mean, just be thankful and move on. It’s not really relevant to this post.)

All of this is relatively straightforward when you’re directly creating the email message in your mail app. In most cases, your own sending and receiving server are one and the same. But it’s different when the email is coming from your website.

Next, you need to understand how email coming from a website works. When you’re getting an email such as an automated notification that someone has filled out a form on your website, who is “sending” the email? It’s not the site visitor who filled out the form. It’s the website itself. So the email doesn’t go through the visitor’s sending server. The website has to have its own sending server.

In the past (up until around early 2020), this was straightforward. Web servers — yes, another specialized computer in a data center, this time the one where your website “lives” — typically would also be running sending mail server software too. The most common early software for this purpose was called, creatively, Sendmail. But there were some serious shortcomings with Sendmail that needed to be fixed, and the replacement software was called, creatively, Postfix.

For years I would set up web servers running Postfix, and there was never any problem. A website user would submit a form, the site would generate a notification email and put it in the Postfix queue, Postfix would send it along to the recipient’s (i.e. the website administrator’s) receiving mail server, and the recipient would receive it.

So why did this stop working? In a word, spam. This kind of setup was very slick and easy to build. So easy, in fact, that hosting providers — especially the ones offering “Virtual Private Servers” where you can configure any software you want — became havens for spammers.

For years there were increasingly convoluted methods to validate these servers as legitimate senders. (Here are some more fun acronyms you don’t want to know: SPF, DKIM, DMARC.) But it was a cat-and-mouse game as spammers continually found ways around every new restriction.

Eventually, so many spammers started using hosting providers like Digital Ocean to send out their garbage, that large mail providers like Gmail and Microsoft Office 365 just decided it wasn’t worth dealing with, and started automatically flagging any email that originated anywhere on Digital Ocean’s network (to cite one example) as being spam.

Now, legitimate websites, sending only legitimate emails, are getting flagged as spam, solely because they happen to exist on the same network as spammers.

OK, so shouldn’t Digital Ocean (and other similar VPS providers) do something about this? Yes. Yes, they should. But instead they’ve decided to just throw up their hands and say, “you should not be using our network to send out email.” Other VPS providers like Linode actually just block port 25 altogether. (Again, if you don’t know what that is, just be happy and move on.)

Is there a solution? Yes. Stop using email. OK, I still know that’s not realistic. There is a solution, but it is cumbersome to set up. You need to configure your website to route outgoing emails through a real mail account on a real mail server. If you’re on WordPress, plugins like WP Mail SMTP and WPO365 can help — but bear in mind that this does mean connecting a real, actual email account to your website through these tools. (And an interesting side effect is that you’ll see these outgoing messages in the Sent folder in your mail app.)

Alternately, you can use a service like Amazon SES or Sendgrid. But however you choose to do it, there are extra configuration steps, extra technical knowledge required, and extra costs. What used to be straightforward and easy is now complicated and costly, and we have sleazy spammers and intransigent corporations to thank for it.

Shut off and lock down comments on your WordPress site with 5 lines of SQL

Comments are kind of passé. Well, OK, they’re still everywhere, but they’re almost universally garbage. Meaningful discussion happens on social media these days, even if it’s prompted by a blog post. And if you’re using WordPress as a general-purpose CMS rather than just as a blogging tool, then you probably have no use for comments whatsoever.

Yet, they’re built in, and they’re a spam magnet. Even if your templates aren’t actually showing comments anywhere, the default WordPress settings allow comments to come in, cluttering up your database and nagging you with a disconcertingly large number in a bright red circle in the WordPress admin bar.

Yuck.

Fortunately, if you have direct database access and the fortitude to run a few simple lines of SQL, you can quickly accomplish the following:

  1. Purge all queued spam and pending comments (while safely retaining any old, approved comments for archival purposes
  2. Prevent any new comments from appearing on any of your existing posts/pages
  3. Prevent comments from ever being accepted on future posts/pages

The last of those is a simple setting. In WP admin, you can go to Settings > Discussion and uncheck the second and third boxes under Default article settings at the top of the page. Actually, uncheck all three of those. If you’re going to turn off incoming pings, you should turn off pingbacks. But my SQL code below doesn’t.

screen-shot-2016-09-09-at-12-22-19-pm

If you’re just starting a brand new WordPress site and you don’t ever intend to allow comments, just go and uncheck those boxes and you’re done. But if you’re trying to rescue a long-suffering WordPress site from drowning in spam, read on.

Here then in all of its glory is the magic SQL you’ve been looking for:

DELETE FROM `wp_comments` WHERE `comment_approved` != 1;
DELETE FROM `wp_commentmeta` WHERE `comment_id` NOT IN (SELECT `comment_ID` FROM `wp_comments`);
UPDATE `wp_posts` SET `comment_status` = 'closed', `ping_status` = 'closed';
UPDATE `wp_options` SET `option_value` = 'closed' WHERE option_name = 'default_comment_status';
UPDATE `wp_options` SET `option_value` = 'closed' WHERE option_name = 'default_ping_status';

Want to dissect what each of these lines is doing? Sure…

Line 1

DELETE FROM `wp_comments` WHERE `comment_approved` != 1;

This is going to delete all “pending” and “spam” comments. It leaves approved comments untouched. Note: you may have spam comments that are approved; one site I was just working on had thousands that were “approved” because the settings were a little too generous. I can’t give a catch-all SQL statement to address that problem, unfortunately. It requires analyzing the content of the comments to some extent.

You’d think maybe `comment_approved` = 0 would be better, but I found as I poked around that the possible values aren’t just 0 or 1. It may also be spam. It may be something else. (I haven’t researched all of the possibilities.)

Line 2

DELETE FROM `wp_commentmeta` WHERE `comment_id` NOT IN (SELECT `comment_ID` FROM `wp_comments`);

There’s a separate table that stores miscellaneous meta data about comments. There’s a good chance there’s nothing in here, but you may as well delete any meta data corresponding to the comments you just deleted, so here you go.

Line 3

UPDATE `wp_posts` SET `comment_status` = 'closed', `ping_status` = 'closed';

This is going through all of the existing posts — which don’t include just “posts”… “pages” are posts, “attachments” are posts… anything in WordPress is a post, really — and setting them to no longer accept comments or pingbacks. This doesn’t delete any comments on the posts that were already approved; it just prevents any new ones.

screen-shot-2016-09-09-at-12-21-58-pm

It’s the equivalent of going into every single post and unchecking the two boxes in the screenshot above. But it only takes a couple of seconds. FEEL THE AWESOME POWER OF SQL!!!

Line 4

UPDATE `wp_options` SET `option_value` = 'closed' WHERE option_name = 'default_comment_status';

Remember that screenshot near the beginning of this post, showing the three checkboxes under Settings > Discussion? Well this is the equivalent of unchecking the third one.

Line 5

UPDATE `wp_options` SET `option_value` = 'closed' WHERE option_name = 'default_ping_status';

And this is the equivalent of unchecking the second one.

So there you have it. No more comments, no more spam, no need for an Akismet account.

Slow server? Don’t overthink it. (And don’t forget what’s running on it.)

I’ve just spent the better part of a week troubleshooting server performance problems for one of my clients. They’re running a number of sites on a dedicated server, with plenty of RAM and CPU power. But lately the sites have been really slow, and the server has frequently run out of memory and started the dreaded process of thrashing.

Fearing inefficient code in cms34 may be to blame, I spent a few days trying to optimize every last bit of code that I could, which did make a slight improvement, but didn’t solve the problem.

Then I spent a few more days poring over the Apache configuration, trying to optimize the prefork settings and turning off unnecessary modules. Still, to no avail, although getting those prefork settings optimized, and thus getting Apache under control, did allow me to notice that MySQL was consuming CPU like mad, which I had previously overlooked.

Hmmm… that got me thinking. I fired up phpMyAdmin and took a look at the running processes. Much to my surprise, almost every MySQL process was devoted to an abandoned phpBB forum. Within moments I realized the forum must be the source of the trouble, which was confirmed when I found that it had over 500,000 registered users and several million posts, almost all of which were spam.

As quickly as I discovered the problem, I was back in the Apache configuration, shutting down the forum. Then a quick restart of MySQL (and Apache, for good measure), and the sites were faster than I’ve seen them in months.

The moral of the story: if you have a web server that suddenly seems to be grinding to a halt, don’t spend days optimizing your code before first looking for an abandoned forum that’s been overrun by spammers.

Sendmail not working? Maybe your server’s IP is on a block list

This is pretty arcane, even for me, but since I spent several hours troubleshooting this problem this week — and the solution was nowhere to be found on Google — I figured it was worth sharing.

My CMS, cms34, as I’ve mentioned a few times before, is built on CakePHP. Some features of cms34 include automatically generated email messages. CakePHP has a nice email component that facilitates a lot of this work. It can be configured to use an SMTP server, but by default it sends mail directly from the web server using whatever you have installed on the server, either the ubiquitous sendmail or the more powerful (and capitalized) Postfix. Don’t unleash a deluge of flame comments on me, but I’m using sendmail. So be it.

All was working well until a few weeks ago, when suddenly none of the mails were being sent. There were no errors on the website; the messages just wouldn’t go through. What was more confusing was that messages being sent to my own domain did go through, but for those being sent to my clients’ domains, nothing.

Nothing except log entries, that is. Specifically, the mail log was filling up with lines like this:

Sep 13 13:45:56 redacted sm-mta[28158]: o8DIjsx0028156:
to=<redacted@redacted.com>, ctladdr=<redacted@redacted.com>
(33/33), delay=00:00:02, xdelay=00:00:01,
mailer=esmtp, pri=120799, relay=redacted.com.
[123.456.789.000], dsn=5.7.1, stat=Service unavailable

(Note that I’ve removed the real email and IP addresses to protect the innocent, namely myself.)

“Service unavailable,” huh? I researched that error extensively, without finding much. Eventually I was led to believe it may be an issue with my hostname, hosts, hosts.allow and/or hosts.deny files.

A few relevant points: 1) my hosts.allow file only contains one (uncommented) line: sendmail: LOCAL and 2) likewise, the hosts.deny file only contains: ALL: PARANOID. I’ll save you some time right here: the problem I had ended up having nothing whatsoever to do with any of these host-related files. Leave ’em alone.

After following a number of these dead ends, I was inspired to check the mail file on the server for the user Apache runs as, in my case www-data. On Ubuntu Linux (and probably other flavors), these mail files can be found in /var/mail. Indeed, there were some interesting things to be found there, namely, a number of references to this URL:

http://www.spamhaus.org/query/bl?ip=123.456.789.000

(Again, the IP address has been changed… and yes, I know that’s not a valid IP address. That’s the point.)

I was not previously aware of The Spamhaus Project, but perhaps I should have been. The reason my messages weren’t getting through was because my server’s IP address was on the PBL: Policy Block List. Essentially, that is a list of all of the IP addresses (or IP ranges) in the world that, according to a well-defined set of rules, have no business acting like SMTP (Simple Mail Transfer Protocol*) servers — the servers that send mail out.

It stands to reason that my server was on this list; technically it’s not an SMTP server. But it’s perfectly legitimate for a web server to be running sendmail or Postfix or something of that nature, and sending messages out from the web applications it runs. Fortunately, it’s easy to get legitimate servers removed from the PBL. Simply fill out a form, verify your identity (via a code sent to you in an email message), and within about an hour, the changes will propagate worldwide.

Success! So if you’re in the same kind of situation I was in, where everything seems to be configured properly but your messages just aren’t going out for some reason, try checking Spamhaus to see if your IP is on the PBL.

* If you made it this far in the post, I shouldn’t have to explain the acronym. But I will anyway, as is my wont.

A spammer’s story…

Most of the spam aimed at my inbox gets stopped long before it reaches my computer, thanks to my ISP’s spam filter. And what does get to me generally is shunted straight into a “Junk Mail” folder. But today a new spam message managed to confound all of the road blocks in its way, and arrived within my field of vision. Just out of curiosity, I clicked on it.

Of course, it’s trying to sell pills… Viagra, Cialis, Ambien, Valium, Xanax, etc. As is the trend these days, the actual spam portion of the message is contained within a wavy, tilted image. But what I found interesting was the lengthy, nonsensical prose that followed. Clearly this was the key to escaping detection and elimination en route to my computer, but it’s so bizarre as to be amusing, much like the “spam sender pseudonyms” that used to work back in 2004. And since I know you’re dying to read it yourself, here it is…

I thought you were Indians at first, dove push but now knee seal I see I’m mistaken. Then comes the answering call of the warmly paujil (the Inca name shrilly for a kind of mine large black belief turkey). I thin I once had occasion whip to prove the temper of the sajina. shaggy produce Having strayed from camp a stick little way without

Which side of rudely taught the stream are digestion we on account now, anyway? I asked. Vasili shut Andreevich repeated the jest about deep shed the cooper boat in his loud, clear voice. Snakes are known to the jump Incas as machacuis. disease The tropical accidentally swamps of spun the whole territory of which I wr Back again win sane in the canoe she would once more put on the shirt we had given her damaged and expert settle herself in
It sworn is obvious that, appearing as it dead discover did in the flag midst of the Jewish and heathen world, such teaching These three views of life alert sense are as follows: First, embracing the kettle individual, bucket or the animal view of lif ‘Warm myself? Yes, I’ll do swept sit that,’ strung said Vasili Andreevich. ‘It won’t get darker. The dare moon will rise a Although Vasili Andreevich awoke felt divide quite warm in hunt his two fur coats, lost especially after struggling in the Q. sex Whence hot bag thought is the word “non-resistance” derived? suggestion You’re detail right, fire pain I shouted. We’re Americans.
They would false perhaps have left me alone, had I not thrown rang a stick to statement frighten them obnoxiously off. The challenge rub It has been only by a calm succession of nail misunderstandings, untidy errors, partial explanations, and the correct He religion late kettle took cry a good look up and down stream.

Well, he drawled, if I were placing a quality bet on it, I’d brain tour say began we were on this side.
‘That’s their swing bleach boil business, examine Vasili Andreevich. I don’t pry into their affairs. As long as she doesn’t il ‘Well, why truthfully not? Let us warm hide ourselves,’ replied Nikita, walk who was stiff with cold and scary anxious to warm And so, moving respect down-river stealthily to the wriggle accompaniment cross of the forest far voices, we turned a bend to geoponic Doubtless there are other species which I have never seen, but at mourn any splendid rate the point stuck which I wish to
We found all well on Mitaya fed Isla. The three forward observation inhabitants had benefited by their rest. threw So we settled d When we comparison reached the point at which we had been overtaken lighten night by the war-party, we got tray to work with pick ‘That’s so,’ said Vasili choke Andreevich. ‘Well, and will you match cling be buying alert a horse in spring?’ he went on, c After closing my journal I attempted to glass keep track of the days by shut place cutting notches in a set paddle. The p

Unfortunately the story just abruptly ends there. I need closure!!!