I spent 5 hours troubleshooting this WordPress problem so you don’t have to (starring: WooCommerce Action Scheduler)

Sorry for that “clickbaity” headline. I added the parenthetical so it might be at least marginally useful. Since my WordPress-related posts are always about how I solved a particularly weird or obscure WP issue, I usually consider their titles carefully. “What would I have googled to find a solution to this problem?” But honestly, I spent 5 hours on this yesterday partly because I wasn’t sure what to google. (And I use lowercase “google” as a generic for “conduct an Internet search”; I normally use DuckDuckGo.)

OK, so here’s the situation. This particular site is — among my normally very-low-traffic clients — one of the busiest I work on. It’s a WooCommerce site with hundreds of products and 20+ daily orders. (Yeah, 20+ orders a day is not huge, but on the scale I normally deal with, it’s a lot.)

This site runs on its own virtual private server, with 8 GB RAM and 4 vCPUs. Pretty substantial for a single site. And yet, for weeks it has been maxing out RAM and CPU resources. Not to the point where the site was in crisis mode that demanded my immediate attention, but it was frustratingly slow. Just slightly below the threshold of me dropping work on other new projects to try to fix this. (At this point I feel obliged to note that I did not actually build or maintain this site for its first couple of years of existence, so I don’t know its inner workings as well as I normally would. I just know it’s running way too many plugins and desperately needs some TLC I have not had time to give it.)

Yesterday things finally got to the breaking point. For me, at least. The client had contacted me about an unrelated issue, but as I was dealing with that, I got frustrated by seeing all of this inexplicable resource usage, so I had to address it.

As it happens, this post is actually a bit of a sequel to my last post, about getting Apache’s mod_status and mod_rewrite to play nicely on a WordPress site. About three weeks ago I finally got mod_status working on this site, and had planned to come back, when I had a chance, to investigate this issue.

If you are not familiar with mod_status, you should check it out. Apache is generally a bit of a “black box” but this lets you see exactly what’s happening with each thread — the requested URL, the requesting IP address, connection time, resource usage, etc.

I noticed an absurd number of threads were coming from the localhost and were requesting wp-admin/admin-ajax.php with a query string referencing WooCommerce’s Action Scheduler. But what to do with that information?

I’ll admit, this is where I wasted a bunch of time in fruitless searches, because I don’t know a lot about Action Scheduler. I read a few threads on the WordPress support forums and StackOverflow that kind of danced around the problem I was having but never really got at it.

Eventually I ended up in phpMyAdmin, scrutinizing the wp_actionscheduler_actions table, and trying to figure out where all of the wc_facebook_regenerate_feed actions were coming from. I used my old favorite Search-Replace-DB to try to find any instances of “facebook” in the database. (This was an utter failure, for reasons I can’t explain. But that failure was critical to why this took me so long to resolve.)

I went to Tools > Scheduled Actions in WP admin and discovered there were over 200,000 actions, although there were only about 70,000 (only!) showing up in wp_actionscheduler_actions. Mystery!

I went to wp_actionscheduler_actions again, saw that those wc_facebook_regenerate_feed actions had all been scheduled weeks ago, and decided to just chuck out the lot. I truncated the table, but within seconds it started filling up again with hundreds of wc_facebook_regenerate_feed actions, with the same weeks-old scheduled dates. Where were they coming from???

What was especially maddening to me about all of this was that I had already, weeks ago, determined that the plugin that had created these — Facebook for WooCommerce — had been causing some kind of trouble, and I had deactivated it. Yesterday I even went so far as to delete it. I scoured the theme code for references to Facebook. I looked in the file system for stray files that might be responsible. And as I mentioned before, I tried to search the database for any references to Facebook. I was getting nowhere.

Eventually I realized Search-Replace-DB was having problems, so I dove into phpMyAdmin directly and started searching individual fields, in individual tables, for “Facebook”. And that’s where I finally figured it out.

WordPress puts everything in wp_posts, and that’s a problem.

I’ve complained over the years about the database architecture in WordPress. Having built multiple custom CMSes in the years prior to when I finally, fully embraced WordPress in 2014, I have a fair bit of experience designing databases. And two things I learned in that experience were: 1) clearly define what your data tables are for, and 2) indexes make databases efficient. WordPress is awesome for many things, but it has far, far outgrown its original conception as blogging software. Custom Post Types and Custom Fields make it super-flexible, but shoving everything they create into wp_posts and wp_postmeta can create a disastrous situation.

Case in point, WooCommerce scheduled actions. In earlier iterations, those were custom posts! (As are, still, WooCommerce orders, which is totally f***ed up, if you ask me.) At some point Woo or Automattic realized scheduled actions don’t belong in wp_posts, so they created four new tables specifically for managing them. Plugins that use scheduled actions had to create new scheduled actions for migrating the old wp_posts scheduled actions into the new tables.

And that’s where I found myself. Through some curious set of circumstances with this particular site, which probably at some point included someone other than me disabling WP-Cron to try to fix some other problem, 200,000+ scheduled actions from the Facebook for WooCommerce plugin (in the wp_posts table) got queued up for migration to the new tables. And as quickly as I was deleting them from the new tables, Action Scheduler (which runs once a minute!!!) was dutifully refilling them.

(And obviously they were never actually running… perhaps because I had deactivated the plugin? Or because they were simply timing out? Who knows? But here’s something I see as a flaw with Action Scheduler: it should check to see if the plugin that scheduled an action is currently active, and if not, purge the action immediately.)

At last here was the fix. I had to run this SQL query in phpMyAdmin. (Proceed with caution! Don’t just use this code… look in your database for exactly what is causing problems and adjust accordingly.)

DELETE FROM `wp_posts` WHERE `post_title` = 'wc_facebook_regenerate_feed';

Note: I’m doing this from memory — and a glance back at my browser history from yesterday. I didn’t keep notes on exactly what the title was.

For a more generalized — and drastic — approach, you could also do this:

DELETE FROM `wp_posts` WHERE `post_type` = 'scheduled-action';

I’ll just conclude here with a nice little graph of the site’s CPU and RAM usage over the past 24 hours. It was 6 PM when I finally figured this out!

How to get Apache’s mod_status and mod_rewrite to play nicely on a WordPress site

Apache’s mod_status can be very handy for monitoring exactly what’s going on inside of Apache on a busy website, but it can be a bit difficult to set up, if your site runs something like WordPress that also relies heavily on Apache’s mod_rewrite.

Specifically even though I had set up mod_status according to the official instructions, and specifically had also added the code to the virtual hosts, I still found that trying to access a site’s /server-status URL was just redirecting me to the WordPress 404 error page.

Here’s the fix. Maybe there’s a “better” way, but this worked for me. I just needed to hijack the rewrite rules in the site’s .htaccess file.

If you’ve already got IP or Auth based access restrictions configured in the virtual host, you probably don’t need the RewriteCond line, but I prefer to err on the side of caution. I used my VPN’s IP address (masked as 9’s here, which of course is not a valid IP address)… you’ll want to fill in whatever IP address(es) you want to allow in.

RewriteEngine on
RewriteCond %{REMOTE_ADDR} ^(999\.999\.999\.999)$
RewriteRule ^server-status$ – [L]

Put this before the WordPress rewrite rules, or it won’t do any good. And of course this is missing the <IfModule mod_rewrite.c> wrapper you probably should include, but if you’re doing this you already know mod_rewrite is enabled, so I don’t bother.

Scott’s Vegetable Fried Rice

Even before COVID-19 hit, back when we were getting takeout at least once or twice a week, this was a staple meal I’d cook… uh… almost as often as we were getting takeout. Now I make it two or three times a week, because it’s reliable, satisfying, and I have it down cold.

Any time I realize I have a recipe down cold, and that my own technique for it has probably deviated somewhat from whatever recipe I originally followed when I needed to follow a recipe, it feels like I should write down my own recipe. For posterity, or whatever. Anyway, here it is!

I’ve adapted this recipe slightly for the time we’re living in. Normally we always buy fresh produce, but as stay-at-home orders set in, and I wasn’t sure what the future held, grocery-shopping-wise, I bought some bags of frozen vegetables. Frozen vegetables don’t take as long to cook as fresh, so that affects the timing and sequence of adding vegetables to the stir fry.

Scott’s Vegetable Fried Rice

Makes… a lot. But you’ll eat a lot of it, so it’s pointless to try to say how many servings it is!

Ingredients

All quantities are approximate. Use as much or as little as you want, and feel free to omit or substitute vegetables.

4 c or more cooked jasmine rice
1 package extra firm tofu
1 medium onion, rough chopped
1 c chopped or sliced carrots
1 c cauliflower florets, fresh or frozen
1 c broccoli florets, fresh or frozen
1 c chopped Chinese, Savoy or green cabbage
1 c sliced button mushrooms
1 can baby corn, drained and rinsed (optional)
1 c fresh pea pods or frozen peas
1-2 cloves minced garlic
2 eggs, beaten
2-3 diced scallions
peanut oil
sesame oil
soy sauce or soy paste
rice wine vinegar
Sriracha sauce (optional)

Preparation

  1. Cook the rice in a rice cooker or otherwise according to package instructions. (Any kind of long-grain white or brown rice will work in this, but we prefer the taste and texture of white jasmine rice.) You can also use leftover rice! Note this is 4 cups cooked. It only takes about 2 cups of dry rice to make 4 cups cooked.

  2. Drain the tofu and press it to remove excess moisture. (We wrap ours in a kitchen towel, between two plates, with a heavy can placed on top, for about 10-15 minutes.) Cut the tofu into 48 cubes. (That’s 2 x 4 x 6.)

  3. In a large bowl, mix about 2 tbsp each of the soy sauce, sesame oil, rice wine vinegar and Sriracha until well blended. Add in the tofu cubes and toss very gently (they’ll break apart otherwise), until thoroughly coated. You can do this up to a few hours in advance so they’ll marinate, but I never plan ahead enough. Not a vegetarian or just hate tofu? Skip it! Or, chicken or shrimp will also work very well in this recipe. (Marinate the meat in the same mixture and stir fry it separately before the vegetables, then set aside until the end.)

  4. Spread the marinated tofu cubes on a baking sheet lined with parchment paper, and bake at 400ºF for about 10-15 minutes. The timing isn’t super-critical; you just want them to get a bit crusty on the outside. You can also deep-fry them but I prefer baking.

  5. While the tofu is in the oven you can do the rest of these steps. We’ll start with a typical stir-fry of the vegetables. This means adding in each type of vegetable every couple of minutes, so you need to approach them in order of how long they take to cook. In a large skillet or wok over medium-high heat, add 2-3 tbsp of peanut oil and a generous splash of sesame oil. When the oil is hot, add the onions and carrots. If you’re using fresh cauliflower, add it now as well. Stir fry for a couple of minutes, until the vegetables start to soften and change color. If the onion starts to turn brown, turn the heat down a bit or add a splash of water.

  6. Add the broccoli, if it’s fresh, and stir fry for a couple of minutes, again until it starts to soften and change color. If your broccoli is frozen, go right to the next step. If you’re using fresh pea pods, add them now as well.

  7. Add the cabbage and any frozen vegetables except peas. You know the drill… a couple of minutes, etc.

  8. Add the mushrooms and baby corn, if using. Stir fry until the mushrooms have released their moisture and are starting to darken.

  9. Add the frozen peas and the minced garlic, and stir fry for no more than 2 minutes. Add a splash of rice wine vinegar and stir well, then add about 2 tbsp of soy sauce or paste, and Sriracha to taste. Stir well and then remove all contents of the skillet to a large bowl. (I use the same bowl I tossed the tofu in, so the vegetables absorb whatever is left of the marinade.)

  10. Wipe the skillet with a paper towel if there’s a lot of residue, or just leave it as-is. Return to the heat and pour in about 4 tbsp of peanut oil and another splash of sesame oil. This should heat up very quickly.

  11. Add your rice to the skillet, breaking up any chunks, and spread it around in an even layer. Cook for a few minutes, turning occasionally, so that some grains get a bit crisp and brown, but not burnt.

  12. Push the rice to the sides of the skillet in a ring, so there’s a large opening in the center. Add a touch more peanut oil, then pour in the beaten eggs. Let sit for a few seconds and then stir and break up as the eggs cook. (Pretend you’re making scrambled eggs for breakfast.) Cook the eggs until they’re no longer runny, but don’t overcook — we’re not done yet, and they will cook more.

  13. Stir the cooked eggs and rice together until the egg is thoroughly mixed through. Reduce the heat to low. Add a generous amount of soy sauce or paste — at least 2-3 tbsp, and stir until well blended.

  14. Add your vegetables to the skillet with the rice and egg and stir to blend together.

  15. Take the tofu out of the oven if you haven’t already, and add it to the skillet. Stir again to blend.

  16. Remove the skillet from the heat, and stir in the diced scallions.

Optional “Dipping” Sauce

We always make this fried rice with frozen vegetable potstickers. I make a sauce for dipping the potstickers using equal parts soy sauce, soy paste, rice wine vinegar, sesame oil and Sriracha. It’s good to make a bunch of this, because it is also great drizzled over the fried rice in your bowl!

Wait… Soy Paste?

I had never heard of soy paste before, but I got turned on to it a few years ago. It’s brewed in a similar way to the soy sauce we know well in America, but it’s thicker — almost the consistency of Hershey’s chocolate syrup — and it has a deeper and less salty flavor. You can pretty much use it interchangeably with soy sauce, but bear in mind that because it’s thicker, you might need to add a little bit of water to your skillet with it so it doesn’t just burn to the bottom. When you’re pouring it on top while serving though, keep it thick!

My favorite kind is Kimlan, which I pick up at United Noodles.

How to get (Mac) Safari to stop showing Favorites and Siri Suggestions panel when clicking the address bar

Not only do I never use it, I scarcely ever even noticed it before, but in recent versions of Safari on the Mac, whenever you click the address bar, a large panel pops out under it, displaying Favorites and Siri Suggestions.

Like I said, I scarcely noticed it before, but I did notice it the other day when I was doing a screen sharing session with a client. Luckily I don’t have anything embarrassing in my browsing history for it to reveal, but my recent activity did convince Siri that I would be extremely interested in links pertaining to the Minnesota Twins. Which I was not at that particular time.

But then it occurred to me… I never — I mean never — click on anything in that panel, but it does add to the visual “noise” of my daily browsing activities. So there’s got to be a way to get rid of it.

Yes, there is. I hunted around for it, so you don’t have to.

First, let’s turn off Siri Suggestions. This is an insidious pathogen that has metastasized throughout macOS, so if you’re like me and never use anything Siri-related, let’s kill it everywhere.

Open up System Preferences and click on Siri

Then in the Siri preferences, click the Siri Suggestions and Privacy button.

I went through the full list of apps and unchecked all the boxes, but at the very least you’ll want to uncheck Show Siri Suggestions in App under Safari:

Now if you go to Safari and click on the address bar, you’ll see Siri Suggestions is gone, but the panel with Favorites still shows up. You can get rid of it entirely by going to Safari > Preferences and clicking the Search tab. (Which… uh… is that really the way its icon is supposed to look???)

Uncheck Show Favorites under Smart Search Field… and maybe everything else. Then put on your tinfoil hat, sit back, and relax!

How to sort empty values last in WordPress

For the past several days I’ve been hammering my head against a conundrum: how to get WordPress to sort a set of posts in ascending order, but with empty values at the end of the list instead of the beginning.

This seems like it should be a simple option in the query. But MySQL doesn’t offer a straightforward way to do this. There are some fairly simple MySQL tricks that will accomplish it, but there’s no way to apply those tricks within the context of WP_Query because they require manipulating either the SELECT or ORDER BY portions of the SQL query in ways WP_Query doesn’t allow. (I mean, you can write custom SQL for WP_Query, but if you’re trying to alter the output of the main query, good luck.)

I tried everything I could possibly think of yesterday with the pre_get_posts hook, but it all went nowhere, other than discovering a very weird quirk of MySQL that I don’t fully understand and won’t bother explaining here.

Sleep on it

I woke up this morning with an idea! I resigned myself to the fact that this ordering can’t happen before the query runs, but I should be able to write a pretty simple function to do it after the query has run.

Bear one key thing in mind: This is not going to work properly with paginated results. I mean, it’ll sort of work. The empty values will get sorted to the end of the list, but they’ll stay on the same “page” they were on before the query was run. In other words, they’ll be sorted to the bottom of page one, not of the last page. Anyway… consider this most useful in cases where you’re setting posts_per_page to -1 or some arbitrarily large number (e.g. 999).

The function

This simple (and highly compact) function accepts a field name (and a boolean for whether or not it’s a custom field [meta data]), then takes the array of posts in the main query ($wp_query), splits them into two separate arrays — one with the non-empty values for your selected field, one with the empty values — and then merges those arrays back together, with all of the non-empty values first. (Other than shifting empties to the back, it retains the same post order from the original query.)

function sort_empty_last($field, $is_meta=false) {
  global $wp_query;
  if (!$wp_query->is_main_query()) { return; }
  $not_empty = $empty = array();
  foreach ((array)$wp_query->posts as $post) {
    $field_value = !empty($is_meta) ? get_post_meta($post->ID, $field) : $post->{$field};
    if (empty(implode((array)$field_value))) { $empty[] = $post; }
    else { $not_empty[] = $post; }
  }
  $wp_query->posts = array_merge($not_empty,$empty);
}

Calling the function

As I said, this function is designed to work directly on the main query. You just need to call the function right before if (have_posts()) in any archive template where you want it to apply. Because of the way it works — especially the posts_per_page consideration — I thought calling it directly in the template was the most clear-cut way to work with it. Here’s an example of the first few lines of a really basic archive template that uses it, looking for a custom field (meta data) called deadline:

<?php

get_header();

sort_empty_last('deadline', true);

if (have_posts()) {