Noted for future reference: Fixing a slow-to-boot Linux server

I have a few Ubuntu Linux VPSes that were originally spun up on the then-latest-and-greatest 16.04 LTS. Over the past year I’ve been belatedly upgrading them to 20.04 LTS. Almost without exception, all now have a really irritating flaw: where they used to reboot almost instantly — making me capriciously run OS updates involving reboots at any time of day, even on servers with a bunch of sites on them, since it only meant a blip of about 5 to 7 seconds — they were now consistently taking 2 to 5 minutes to reboot. Yikes!

Poking around, I learned that cloud-init was timing out, causing that delay, but since systems administration is just a small sliver of the work I do, I never had a chance to investigate why it was happening or really what cloud-init was even for. I just resigned myself to having to do those reboots in the middle of the night when no one would notice.

Well, I finally decided I needed to get an answer, and I found it. If I’m reading this correctly, cloud-init is really only needed during the initial creation of the VPS, and can safely be disabled after that. So, let’s do it!

touch /etc/cloud/cloud-init.disabled

I’m pleased to say, it works perfectly. I ran it on a test mirror of my biggest server and it worked, so I then applied it to the live server and… capriciously rebooted it, right in the middle of the day!

Shush. It worked.


Update (August 3, 2022): Mayyyyybe it’s a bit more complicated than what I described above. I just went into another server I had previously updated to 20.04, and I just went ahead and pre-emptively ran this before an update that required reboot. After the reboot, I could not connect to the server at all, other than through Digital Ocean’s direct access console. Thank goodness for that. It did not initially occur to me that this change might be why I couldn’t connect, but after trying a few other fixes without success, I just went in and deleted this new /etc/cloud/cloud-init.disabled file and rebooted, and everything came up just fine… and without any kind of delay on boot. Weird.

A lament for a lost video gaming era

For a long time I have been lamenting why computer games aren’t like the ones I loved to play in the late ’90s/early 2000s… games like SimCity 2000 and Age of Empires. Even the über-nerdy version of Scrabble I had on my computer back then, with tournament rules and rankings, etc.

Oh, the descendants of those games certainly exist, but they have lost all of the things that made them interesting to me. And there are no new games that really capture that spirit effectively anymore. (Even the new ones that ostensibly try to evoke that spirit… don’t. At least not for a purist like me.)

Finally today I realized why. Because back then the video game market was way smaller than it is now, and it only catered to hardcore nerds like me. These days, it’s so much bigger, and so much broader, that there’s (comparatively) no money to be made on the types of games I liked back then.

And the real tragedy for me is that I can’t even play those games I loved anymore, because I don’t have any hardware that can play them. There’s emulation, but these games seem to exist in a technological gap. Emulators are great for even older games, mostly console games, but I haven’t really been able to find a decent way to emulate these games that required more computing power. Then again… maybe I just haven’t been trying hard enough.

How to purge fake/bot WooCommerce customer accounts directly in the MySQL database

DANGER! If you don’t know the havoc one can wreak with a DELETE statement in MySQL, stop right here. I take no responsibility for what you might do with the information that follows.


Bots like to create fake customer accounts on WooCommerce (WordPress) sites, apparently. What they’re attempting to do, I don’t know. But if you don’t stay on top of things, you might find you have thousands of fake customer accounts in your site. Chances are they haven’t, won’t, and can’t actually cause any damage, but they’re cluttering things up, and any unnecessary user account in a WordPress database represents a potential future security risk.

On a particular client’s heavy-traffic WooCommerce site, I discovered that over the course of the site’s 7-year lifespan it had accumulated nearly 8,000 such accounts, and I wanted to be rid of them.

After carefully exploring the data surrounding a few of these obviously fake accounts, I determined a pattern, and came up with a fairly cautious set of conditions that, to me, indicated a customer was fake:

  1. They had the customer role.
  2. Their user account had nothing in the First Name or Last Name fields.
  3. Likewise, their user account had nothing in the Billing First Name or Billing Last Name fields. (If you’re feeling extra draconian, you might skip this one.)
  4. They had never placed an order while logged in — their user ID did not have an _order_count entry in the wp_usermeta table. Which is perhaps an obvious condition because…
  5. They had never logged in at all — their user ID did not have a wfls-last-login entry in the wp_usermeta table. This condition will only apply if your site uses WordFence.

You can test all of those conditions with a single SQL query:

SELECT DISTINCT `user_id`
FROM `wp_usermeta`
WHERE
`user_id` IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'wp_capabilities' AND `meta_value` = 'a:1:{s:8:"customer";b:1;}') AND
`user_id` IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'first_name' AND `meta_value` = '') AND
`user_id` IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'last_name' AND `meta_value` = '') AND
`user_id` NOT IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'billing_first_name') AND
`user_id` NOT IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'billing_last_name') AND
`user_id` NOT IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = '_order_count' AND meta_value > 0) AND
`user_id` NOT IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'wfls-last-login');

You may want to spot check some of these IDs in the wp_users table, or directly in the site admin, just to be sure everything looks right. Then you can turn the above into a subquery that will delete all of the matching users. Be sure to make a full backup of your database before doing this!

DELETE FROM `wp_users` WHERE `ID` IN (
SELECT DISTINCT `user_id`
FROM `wp_usermeta`
WHERE
`user_id` IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'wp_capabilities' AND `meta_value` = 'a:1:{s:8:"customer";b:1;}') AND
`user_id` IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'first_name' AND `meta_value` = '') AND
`user_id` IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'last_name' AND `meta_value` = '') AND
`user_id` NOT IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'billing_first_name') AND
`user_id` NOT IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'billing_last_name') AND
`user_id` NOT IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = '_order_count' AND meta_value > 0) AND
`user_id` NOT IN (SELECT DISTINCT `user_id` FROM `wp_usermeta` WHERE `meta_key` = 'wfls-last-login')
);

OK… so the users are gone. But each one has a bunch of records in the wp_usermeta table. Now that we’ve gotten rid of the users themselves, it’s easy to purge their associated meta data:

DELETE FROM `wp_usermeta` WHERE `user_id` NOT IN (SELECT `ID` FROM `wp_users`);

The things I make don’t exist

Like most “creatives,” I have an impulse — maybe a compulsion — to make things. I’ve often reflected on the nature of this impulse, wondering why I simply feel in my bones that I need to do this.

I think in some ways it’s about mortality. Our existence is temporary and fleeting. We want to leave a mark on the world. Something that says “I was here.” The impulse to leave a mark often takes a dark turn, but even when it doesn’t, the fragility of our modern world calls into question the extreme vanity of thinking that somehow your existence matters enough that you need to construct monuments to yourself.

But in my case, there’s an even weirder element to the impulse. Nothing I make exists. That is, it doesn’t exist as physical objects. The closest I come, I guess, is if I print out a piece of sheet music I’ve composed. But unlike a painter, a sculptor, a knitter, a builder… the things I make are even more fleeting and ethereal than my own existence.

Sound waves in the air. Momentary flashes of pixels on a screen. The flow of electrons inside computer chips. None of it is real outside of the moments someone experiences it. And worse, most of it is dependent upon the right hardware and software continuing to operate to manifest those moments again in the future. And all of it depends on electricity.

Intermittently through the years I have run an online shop where I sell merchandise featuring my designs, all created on a computer. I just launched a new one. But I don’t make the products. I certainly am not stitching together the t-shirts or firing the coffee mugs in a kiln or… doing whatever it takes to physically create a sticker. I’m not buying the blank merchandise and printing my designs on it. I’m not even warehousing stock. Everything I sell is print-on-demand, and the “photos” of the merchandise on the site are just composites created by the fulfillment vendor’s software.

That gave me the idea for this t-shirt. It doesn’t exist. And as long as no one ever orders one, it will stay that way.

I don’t have a good desk chair

I don’t have a good desk chair.

I can’t sit in my not-good desk chair for very long. It’s not really a desk chair, even, other than that it’s a chair, and it’s at my desk.

It’s just an $80 IKEA side chair, “walnut,” which I feel probably needs to be in quotes since it’s from IKEA. At least it is solid wood, not a hardened slurry of glue and sawdust covered in a “veneer” that’s actually just a paper sticker with wood grain printed on it. Which is a factual description of some IKEA furniture (that does exist in the house of a couple who are on the verge of 50). This chair is undoubtedly real wood.

But a wooden chair is not a good desk chair.

It is not a good desk chair even when it has a memory foam cushion on it. Not even when that memory foam cushion is on top of a silicone honeycomb cushion, although now we are getting somewhere. (But where is that? One of the Amazon reviews of the cushions praised its effectiveness in the wheelchair of its buyer’s nonagenarian grandfather.)

I don’t have a good desk chair.

But do I really want a good desk chair? Over the past year I have converted my workstation to a fully portable setup. I’ve worked with a laptop as my sole computer for over a decade, and now my second monitor is an iPad. I can bring my two-screen setup anywhere.

For years we’ve been told that sitting at a desk all day long shortens our lives. I hate sitting at a desk all day anyway. And years of working in offices, where someone else paid for the series of various “real” (and surely quite expensive) desk chairs I used, taught me that there is no such thing as a good desk chair, at least not for me.

I do not sit with perfect posture. I do not sit in one position. I do not sit back in my chair, so lumbar supports are pointless. Except when I do sit back in my chair, in which case I am usually sprawled low, and the lumbar supports are again pointless.

But sitting perfectly in a precision engineered desk chair all day is still sitting at a desk all day. And when you do that, YOU DIE.

I don’t want a good desk chair.

And I really don’t want a standing desk.

(Neither do my feet, especially during one of my bouts of plantar fasciitis, such as the one I’ve been enduring over the past month as I write this.)

I just want to move around.

I am self-employed, and I have a mobile workstation. So I spend part of the day in my not-good desk chair. I spend part of the day in a lounge chair (IKEA) in our bedroom, with my feet up and the laptop living up to its name. Or that midcentury modern chair (a rare splurge from West Elm) in the living room, again with my feet up and the laptop laptop. Or — now that the weather is finally nice — sitting under the big umbrella at the cafe table we have on our deck, as long as I can find the perfect spot where its metal mesh top is unwarped and my laptop doesn’t wobble with every keystroke. Or in one of the Adirondacks* on the deck, in the afternoon when they’re shaded. Or sometimes even standing (!) with my laptop on the countertop of the pass through between our kitchen and dining room.

I’ve got options that don’t involve a good desk chair.

If I had a good desk chair, I would feel compelled to use it all the time. And then… YOU DIE.


*That would be a molded plastic Adirondack, of course. From Target. We are in our late 40s and still nearly every piece of furniture we own is from either IKEA or Target.