Tag Archives: Web Server

My strange solution to Apache not starting on Ubuntu Linux server with SSLEngine on… (YMMV)

The situation: I’m running a web server on Ubuntu Linux using Apache 2. I have two sites on the server that need SSL. I obtained a second IP address (since you can only have one SSL certificate per IP address) and configured Apache accordingly. I was able to get regular old port 80 non-SSL pages to load just fine on virtual hosts configured to use both IP addresses.

I created my key files, got the certificates from the CA (GeoTrust, in this case), all that business. Put the files in the right places, configured the Apache files, all that jazz. Made sure mod_ssl was enabled, yes. All of that. Trust me, I did it. Don’t bother asking. And yet, whenever I tried to run Apache with SSL configured… nothing.

And I mean… nothing.

I’d restart Apache at the command line, and nothing. No error messages of any kind. But Apache wasn’t running. I checked all of the log files (and I mean all of the log files), nothing. DOA.

Eventually I tracked down the culprit as the SSLEngine on line in the Apache config file. With it in there, Apache wouldn’t start. Comment it out, Apache starts up just fine, but of course you don’t have SSL.

I’m using the arrangement of Apache config files as they’re installed in a default Ubuntu build. That means /etc/apache2/httpd.conf is actually empty, and most of its usual contents are in /etc/apache2/apache2.conf, with a few other settings dispersed into a number of adjacent files. There are some critical settings in /etc/apache2/ports.conf and then everything else is in the individual config files I’ve created for each site on the server, stored in the /etc/apache2/sites-available directory with symbolic links for the active ones in /etc/apache2/sites-enabled.

Well… that turned out to be the problem. I’m not sure why it matters, but I was putting the VirtualHost configurations for the SSL sites in the respective sites’ existing configuration files. But no… all of the SSL-related (port 443) <VirtualHost> blocks needed to be put in the 000-default file. That made all the difference.

Well, almost all the difference. My private key files are encrypted with pass phrases, and Apache needed me to enter them when starting up. But, funny thing… it didn’t ask me for them all right away. I had to fiddle around with starting and stopping it a couple of times (which I bothered to do because it still wasn’t running), but eventually it did ask me to enter the pass phrase for both sites, and after I did that, everything is working. Both SSL sites, all of my non-SSL sites, it all works.

I’m a little concerned that Apache is going to require manual input of these pass phrases again whenever it restarts (e.g. if the server reboots). I hope not, but for now I am at least able to move forward knowing it works at all.

Sendmail not working? Maybe your server’s IP is on a block list

This is pretty arcane, even for me, but since I spent several hours troubleshooting this problem this week — and the solution was nowhere to be found on Google — I figured it was worth sharing.

My CMS, cms34, as I’ve mentioned a few times before, is built on CakePHP. Some features of cms34 include automatically generated email messages. CakePHP has a nice email component that facilitates a lot of this work. It can be configured to use an SMTP server, but by default it sends mail directly from the web server using whatever you have installed on the server, either the ubiquitous sendmail or the more powerful (and capitalized) Postfix. Don’t unleash a deluge of flame comments on me, but I’m using sendmail. So be it.

All was working well until a few weeks ago, when suddenly none of the mails were being sent. There were no errors on the website; the messages just wouldn’t go through. What was more confusing was that messages being sent to my own domain did go through, but for those being sent to my clients’ domains, nothing.

Nothing except log entries, that is. Specifically, the mail log was filling up with lines like this:

Sep 13 13:45:56 redacted sm-mta[28158]: o8DIjsx0028156:
to=<redacted@redacted.com>, ctladdr=<redacted@redacted.com>
(33/33), delay=00:00:02, xdelay=00:00:01,
mailer=esmtp, pri=120799, relay=redacted.com.
[123.456.789.000], dsn=5.7.1, stat=Service unavailable

(Note that I’ve removed the real email and IP addresses to protect the innocent, namely myself.)

“Service unavailable,” huh? I researched that error extensively, without finding much. Eventually I was led to believe it may be an issue with my hostname, hosts, hosts.allow and/or hosts.deny files.

A few relevant points: 1) my hosts.allow file only contains one (uncommented) line: sendmail: LOCAL and 2) likewise, the hosts.deny file only contains: ALL: PARANOID. I’ll save you some time right here: the problem I had ended up having nothing whatsoever to do with any of these host-related files. Leave ’em alone.

After following a number of these dead ends, I was inspired to check the mail file on the server for the user Apache runs as, in my case www-data. On Ubuntu Linux (and probably other flavors), these mail files can be found in /var/mail. Indeed, there were some interesting things to be found there, namely, a number of references to this URL:

http://www.spamhaus.org/query/bl?ip=123.456.789.000

(Again, the IP address has been changed… and yes, I know that’s not a valid IP address. That’s the point.)

I was not previously aware of The Spamhaus Project, but perhaps I should have been. The reason my messages weren’t getting through was because my server’s IP address was on the PBL: Policy Block List. Essentially, that is a list of all of the IP addresses (or IP ranges) in the world that, according to a well-defined set of rules, have no business acting like SMTP (Simple Mail Transfer Protocol*) servers — the servers that send mail out.

It stands to reason that my server was on this list; technically it’s not an SMTP server. But it’s perfectly legitimate for a web server to be running sendmail or Postfix or something of that nature, and sending messages out from the web applications it runs. Fortunately, it’s easy to get legitimate servers removed from the PBL. Simply fill out a form, verify your identity (via a code sent to you in an email message), and within about an hour, the changes will propagate worldwide.

Success! So if you’re in the same kind of situation I was in, where everything seems to be configured properly but your messages just aren’t going out for some reason, try checking Spamhaus to see if your IP is on the PBL.

* If you made it this far in the post, I shouldn’t have to explain the acronym. But I will anyway, as is my wont.

Barack Obama: the open-source candidate?

Now we’re speaking close to my heart. Granted, I’m a freeloader in the open source world. I have yet to contribute a single line of code to an open source project. (OK, I guess that’s not entirely true: I did write a WordPress plugin. Sweet. I’m in the club! Sort of.) But I have wholly embraced open source software in my work. PHP FTW, baby! (Uh yeah… whatever.)

These days the only thing I’m a more enthusiastic and outspoken proponent of than open source software is Barack Obama. So I’m surprised it took me so long to research what he’s running his website on.

Linux PWS/1.3.28

*Whew* Glad to see it’s Linux. But what the heck is PWS? I was at a loss. Then I found this blog talking about the very same issue. And suddenly it made sense why I didn’t recognize the acronym. I never would have considered Microsoft’s Personal Web Server to be the web server of choice running on a Linux server. I am still scratching my head at it. The whole VM thing seems the only logical explanation, except that there’s no logic to explain it. At least it’s not so transparently ass-backwards as John McCain’s configuration:

Linux Microsoft-IIS/6.0

And the inexplicable:

Linux ECAcc (lhr)

Interestingly, though, a Google search for “ECAcc (lhr)” reveals a link to a Digg post entitled John McCain Owns VoteForTheMILF.com. Stay classy, San Diego.

Let’s be clear: I think the idea of running a web site under Windows in a virtual machine on a Linux box is the most incomprehensible, mind-bogglingly stupid arrangement you’d ever bother with. I’d have to guess that the sites were developed to run in a Windows environment, but when it came time to deal with practical server and network capacity issues, load balancing and whatnot, some sysadmin made the (probably prudent) decision to load balance on Linux boxes instead of Windows, but since the site was tied to some feeble Windows technology, they couldn’t just move it over to Linux wholesale.

But let’s take this a step further. Back in late spring I received an email from Barack Obama’s IT director soliciting applicants for web developer positions with the campaign. Even though the job was in Boston, I figured it would be insane not to apply, so I submitted my resume. (I never heard back, for what it’s worth.) And it’s from this that I happen to know that the campaign was specifically seeking PHP developers. Rock on.

With that in mind, the whole Windows-on-Linux-through-VM arrangement made even less sense. Why would they develop the site in PHP, run it on a Windows server (definitely not the optimal arrangement for a PHP-based app, though it certainly will work), and then VM that Windows environment on a Linux box, instead of just gearing the PHP app for a Linux server in the first place? And that’s when I remembered that just earlier in the day I had been looking at taxcut.barackobama.com. Of course! Separate third-level domains are all over Obama’s site. Let’s check the configuration on that domain. Now that’s much better:

Linux Apache/1.3.34 (Debian) mod_gzip/1.3.26.1a AuthMySQL/4.3.9-2 PHP/5.2.0-8+etch10

And I think it explains a lot. Campaigns start off small. Obama had to register barackobama.com and put something up there ages ago, long before he was the Democratic nominee and the hugely successful fundraiser he became along the way. So that original site, www.barackobama.com, was probably developed on a Windows box in someone’s proverbial basement, probably when was running for the U.S. Senate or maybe even the Illinois Senate. But as the campaign has grown, its websites (plural) have grown as well, and in a decidedly open-source direction. There’s some good stuff in there. Debian (which could mean Ubuntu, too… I haven’t checked the signature on Ubuntu’s Apache package to see if it’s split from its Debian roots), PHP (and a reasonably up-to-date version at that), MySQL, etc.

It’s kind of fun to do this kind of research, as long as you don’t mind being distracted along the way, because there are plenty of weird sources of distraction.

Aside from the aforementioned MILF site (classy), and the somewhat interesting fact that searching on “PWS/1.3.28” brings back as its first result a reference to Obama’s hosting, I discovered that for some reason the page title on John McCain’s official store is “Independent Online Stores.” OK. No one looks at title bars. And even fewer web developers look at <title> tags. I know that from experience. But of course that’s just a transitional landing page, announcing that McCain wares are not actually sold by the campaign, but by independent, for-profit companies, and buying these items doesn’t translate into money going back into the campaign. Huh. I can’t quite wrap my brain around that, but I’m a lifelong, union-loving Democrat, so I guess I wasn’t meant to. The only thing that comes to mind is that maybe it has to be that way, legally, now that he’s accepted public campaign financing. Anyway, the first McCain store link I found, which as they state is apparently an independent operation not affiliated with the campaign, is, not surprisingly, running:

Windows Server 2008 Microsoft-IIS/7.0

I also found that the company that hosts some of Obama’s pages also hosts a site for the American Model Yachting Association. Really? Model yachting? That exists?