iCloud Drive: Don’t do what I did!

Being deeply immersed in the Apple ecosystem, a couple of years ago I made a decision:

I’ll move all of my work files onto iCloud Drive!

I work (as in, write code and edit image files) mainly on my Mac. But I was seduced by the possibility of accessing all of my work files in a pinch on my iPad (which I still had at the time) or even my iPhone. Plus, since my files would be in “The Cloud”, I could even access them from another computer (or from my Mac when booted into Windows) if I needed to, by logging into my iCloud account from a web browser.

It seemed… so obvious. So perfect.

Umm… maybe not.

For the past two years, I have been constantly fighting with iCloud Drive. One of its signature features is that it can manage disk use on your Mac automatically, so as your hard drive fills up, it deletes files you haven’t used in a while, keeping them safely in the cloud while freeing up disk space on your Mac. And with my MacBook Pro sporting a (meager?) 256 GB hard drive, with 40-odd GB allocated for a Windows partition, and over 60 GB occupied by Logic Pro X sound samples, my drive is filling up constantly.

While this is great in principle, it is completely unworkable in practice for three interrelated reasons:

  1. If you have a large amount of data in play here (for me, it’s in the vicinity of 100 GB), iCloud Drive may get to a point where it is constantly transferring data. If you’re not on a gigabit fiber connection, this can both use up all of your Internet bandwidth and take ages.
  2. Because #1 is taking place constantly, if you do find yourself needing to grab one of those files that has been deleted locally (as indicated in the Finder by a cloud icon with a down-pointing arrow), you may find yourself waiting several minutes for the file to become available, even if it’s small (as in, under 1 MB).
  3. In an effort to make this all appear seamless to the user, the Finder represents cloud-only files as… regular files. But they’re actually just pointers with a hidden .icloud filename extension… as you’ll find if you ever try to perform Finder actions inside another program, such as syncing files to a web server using Transmit.

All of this might be tolerable if Apple gave you any control whatsoever over which files get deleted locally. But they don’t.

It gets worse.

What’s worse than getting stuck in this situation? Trying to get out of it. It’s like quicksand… the more you struggle against it, the faster and deeper you sink into it.

I made the decision earlier this week to extricate myself from the iCloud Drive nightmare, by buying a 250 GB SanDisk external SSD. First off, a little unpaid plug for this drive… it is awesome. It’s super light and small, seemingly at least as fast as the internal SSD in my MacBook Pro (in that I am able to transfer multiple gigabytes of data in seconds), and it even looks cool. I’m going to be using it all the time, so I’m actually considering putting adhesive Velcro on both it and the top of my MacBook Pro so I can keep it permanently attached. (Which says a lot about how much my regard for Apple has fallen lately — in the past I would never have sullied the exterior of an Apple laptop with something adhesive.)

So anyway, external SSD acquired, my goal was to start transferring my files from iCloud Drive over to the SSD.

Uh… good luck with that.

Because I’m now at a point where I have more than double the amount of data stored on iCloud Drive as I have available space locally, a majority of my files are now only in “The Cloud.” Ugh. Which means waiting for all of that data transfer stuff to happen. If only I could, somehow, bypass this broken process, I thought.

There has to be a way.

tl;dr Nope.

So, here’s the thing. I’ve been using iCloud Drive for the bulk of my cloud-based file storage, but I do use other services as well. I have Google Drive. I have Dropbox. I know how they work.

Specifically, I know that you can, y’know, like, select a folder and download the entire thing as a zip file.

So I thought to myself, I’ll just go to iCloud in a web browser and do that! Download the whole friggin’ thing as a big zip file, or maybe a few zip files, and be done with it.

Nope.

For whatever reason, iCloud doesn’t let you do that. Probably because of the whole seamless “It just works” Kool-Aid drinking song everyone in Apple land has been singing. (Myself included, mostly.)

You can only download individual files, not folders, from the iCloud web interface. It does let you select multiple files at once, but only within one folder.

Check out this delightful thread full of know-it-all asshats whose response to a legitimate question — why doesn’t Apple allow this thing that every competing service does? — is to challenge the validity of the question and the intelligence of the questioner. (That thread is now closed so I’ve just opened my own new thread on the topic. Watch this space for trolls!)

There. Are. Plenty. Of. Reasons. A. Person. Might. Have. For. Needing. To. Do. Something. That. You. Have. Not. Previously. Considered. Stop challenging their premises and answer their question, or shut the hell up.

Whew.

I ended up “solving” the problem by resigning myself to the fact that it wouldn’t be completely solved. So instead I took the drastic approach of temporarily logging out of iCloud completely, just so I could strand the files I did have saved locally, and copied them to the SSD.

Then I logged back into iCloud Drive and tried to get it to stop syncing my files by unchecking the Desktop and Documents Folders option.

The only problem is, I didn’t have my work files in those folders. I had them in a separate top-level folder in the iCloud Drive that I created myself. Because, you know, you can do that and it didn’t seem like a crazy idea or anything.

It was.

I discovered this morning that even though I had done all of this and tried to purge the nightmare of constant iCloud Drive syncing from my Mac life, once I had logged back into iCloud, the Mac went right back to quietly, constantly, syncing that iCloud Drive data on my Mac. As I type this, I have a Finder window open to my iCloud Drive and in the status bar it says “downloading 120,079 items (36.14 GB of 48.66 GB)”. Fun!

So, my new plan for today is to watch that window, and as the little cloud icons next to individual folders goes away, I’m copying those folders to my SSD and then deleting them from iCloud. My assumption is that as I do this, I am freeing up more local space and iCloud will continue to download the remaining items, and eventually I’ll have everything transferred over.

But please… do yourself a favor and don’t do what I did. iCloud Drive is not suitable for professional use.

How to get iWork (Pages, Numbers, Keynote) apps to stop defaulting to iCloud when saving

Ever since I semi-fully embraced iCloud, I’ve found that the iWork apps — Pages, Numbers and Keynote — always default to wanting to save every new document in iCloud, which I never — well, OK, almost never — want to do. It’s fine that it’s an option, but I want the default to be saving to my local hard drive (which, actually, means saving to my Dropbox account).

It didn’t take much effort to find this thread on Apple’s support forums, but the first suggested solution — turning off “Documents and Data” in System Preferences → iCloud — seemed draconian. With this option you can never sync your documents to iCloud.

A little further down the thread I found the “real” solution, courtesy of “Bernie_uk”, which was important enough for me to want to share here.

It requires opening up Terminal, but it’s not too scary. You just have to run this command:

defaults write NSGlobalDomain NSDocumentSaveNewDocumentsToCloud -bool false

This command doesn’t turn off anything in iCloud; it just tells the system that your default should be saving files to disk, not to iCloud. Note that since this is a global setting, it will affect not just iWork, but any other apps that use iCloud’s “Documents and Data” syncing. (I guess.)

New adventures in hi-fi… er, iTunes Match

As successful as iTunes has been in transforming both the music industry and the music listening experience, it has, from the beginning, been hamstrung by restrictions imposed by the outmoded, fearful major record labels.

Little by little, Apple has whittled away at those restrictions while managing to create a hugely successful business — iTunes has for several years been the largest music retailer in the world. First there was iTunes Plus: a boost in quality and a victory for users with the elimination of DRM copy restrictions. And now we have the real game changer: iTunes Match.

For $25 per year, you can now store your music “in the cloud.” iCloud, to be specific. That annual subscription allows you to create a centralized, comprehensive library of all of your digital music on Apple’s servers, and accessible from any of your computers and iOS devices. No more worrying about limited disk space or struggling with syncing issues. It just works.

In principle.

In practice? Well, I put iTunes Match to the test today. My music library poses a few unique challenges to this new service:

  • My library consists of over 18,000 songs, and more than 140 GB of data.
  • My main computer is a MacBook Air with a 128 GB hard drive, so I keep a “master” library on an external hard drive and a day-to-day library on the internal hard drive.
  • My main iOS device is a 32 GB iPhone 4, which has been syncing with the “day-to-day” iTunes library on my Mac’s internal hard drive.
  • SLP and I have our own iTunes accounts but have long desired to have a single shared music library.
  • My music library consists mostly of non-iTunes Plus tracks: a mix of DRM-laden 128 kbps iTunes tracks, ripped CDs, and tons of MP3s downloaded from Amazon.com.

With these factors in play, I had some specific goals for iTunes Match, roughly in this order:

  1. Move my “master” library from an external hard drive that sits on my desk, into iCloud where all of our devices can access it.
  2. Free up precious storage space on my MacBook Air and iPhone.
  3. Upgrade old DRMed 128 kbps iTunes tracks to higher-quality, DRM-free, 256 kbps versions.
  4. Consolidate SLP’s purchased iTunes music (around 600 songs) with my main library.
  5. Clean up duplicate tracks.

Spoiler alert: I pretty much knew going into it that the last of those items was going to get worse before it got better. But there were still plenty of surprises (good, bad and ugly) along the way.

The journey of 1,000 miles (or 18,000 songs) begins with a single step

I began my iTunes Match journey about two weeks ago, as soon as iTunes Match became available to the public. (For what it’s worth, I’m registered as an iOS developer, so I had access to the beta, but was never able to get it working properly.) The first goal was to get all of my music loaded into the system, and for the most part that went fine. Which is to say, it went… and went… and went… and w…e…n…t… fine. Loading the 4,000 or so songs I kept on my internal hard drive was fairly inconsequential. The process completed in a couple of hours while I went about my work that day. But then when the time came to fire up the external drive and load the remaining 14,000 or so songs… hmm. How can I put this? I guess the plus side was that I could leave it unattended and sleep, because it took three nights (overnight) to finish.

At that point I left things alone for a while, as I was too busy at the time to devote an entire day to organizing and cleaning up my music library. I did, however, get to play around with the overall iTunes Match experience for a week or so, and I discovered the following:

The good:

  • Having my complete music library at my fingertips on any device is amazing.
  • Streaming works great on the Mac, iPhone and Apple TV. Just pick a song and within a few seconds it starts playing.

The bad:

  • Browsing can be slow, sometimes painfully so, with a large library. This is especially a problem on the iPhone.
  • Cover art is often missing. I haven’t yet determined if it’s just not being downloaded, or if it’s not attached to the albums in iCloud, either.

The ugly:

  • Syncing an iOS device with iTunes on your computer can become a real mess. It’s hard to delete anything: like Michael Myers, no matter how many times you shoot him or stab him or stick a hanger in his eye, he just keeps getting up and coming back to get you. OK, bad analogy. But it almost feels that way.
  • Sometimes you don’t really want to remember just how many songs by Edison Lighthouse, England Dan and John Ford Coley, or Peppermint Trolley Company you own. It would be nice to have more filtering options than: a) just what’s on your device, or b) the whole shebang.

The big day arrives

Today I finally decided that I could afford to put off almost all of my real work for an entire day and devote my attention singularly to the task of getting iTunes Match fully synced, and SLP’s music fully integrated into the main library. To be honest, however, it’s not just today. I began the process at around 8:00 last night, worked until just after midnight, resumed from 7:00 to 8:30 this morning, then worked on it straight from 10:30 AM to 3:30 PM and again from about 7 PM to 10 PM, when I began writing this post. That’s 13 1/2 hours total, or approximately 2.66 seconds per each of the 18,266 tracks in my library. YMMV, as they say, but I’d guess it’s reasonable, if you’re trying to budget some time, to assume that you’ll need about 3 seconds times the number of tracks in your library. (And I’m still not really done.)

I took some notes today as I was going about things. Here are some pertinent observations:

Some things were just plain gone. I’m pretty sure this was the fault of my own carelessness in keeping my various pre-iTunes Match libraries in sync, but it’s worth noting that two conspicuous omissions in my library were The King Is Dead by The Decemberists and The King of Limbs by Radiohead. Coincidence?! I think… well, actually, yes, I do think it was probably just a coincidence. Luckily I was able to track down backups of both of those albums, but now I wonder what else is missing that I’m forgetting about.

“Matched” tracks are hit-and-miss. I’m sure Apple is relying on some very powerful algorithms to analyze each track in your library, in order to determine whether or not it matches a track that already exists on iTunes. It’s clearly not just relying on title-and-artist matching like the longstanding (and semi-useless) “Display Duplicates” option. One of the big selling points of iTunes Match is that if your music is available on iTunes, even if you didn’t buy it there, you’ll get the (usually higher-quality) iTunes version instead of the original version in your library, saving you time and saving Apple server space, as well. (Macworld’s Jason Snell has written an excellent tutorial on how to upgrade your tracks.)

The algorithms aren’t perfect, however, and I was annoyed to discover numerous cases where all but one or two tracks of an album were “matched” and could be replaced with 256 kbps iTunes versions, but the other tracks were rejected, for reasons unknown, and were stuck with the inferior quality versions I had to begin with.

Duplicates are a mess. Apple has done a lot to try to make it easy for you to find and weed out duplicate tracks, but you still have to do it. I appreciate that they don’t just assume which tracks you will or won’t want and automatically delete things capriciously, but I still wish there were a more efficient way to trim the excess.

Cloud symbols and error messages could use some clarification. Neven Mrgan has a great summary of the icons and his interpretations of their meanings, but I encountered too many dialog boxes today with useless statements like “This item is not eligible for iCloud” or “The track could not be downloaded because an unknown error occurred.”

If you’re trying to consolidate tracks from two separate iTunes accounts into a single library, you’re on your own. While the 10-device limit on DRMed iTunes tracks, and iTunes’ ability to be authorized for multiple accounts on a single device, allows for this kind of consolidation, Apple has not gone out of its way to support such activities. In my situation, I was dealing with a large number of SLP’s iTunes purchases that were no longer on any of our devices. I happened to have a spare Mac in my office with an empty iTunes library, so I logged into SLP’s iTunes account on that Mac and used the “Purchased” link in the iTunes Store to re-download all of her music in prep for eventual syncing with the main library.

But it wasn’t that easy. At first, a bunch of the songs wouldn’t download. I realized it was because they were still DRM versions, and that I needed to pay another $25 for an iTunes Match subscription on SLP’s iTunes account to get them. Even then, there were a number of weird issues with tracks being unavailable. Strangely, it seemed that in some cases, if I already had some of those tracks in the master iTunes library, and had already downloaded 256 kbps versions of them, it would not allow me to download them on this second computer. This leads me to believe that there is some hidden mechanism whereby Apple does still keep track of even the DRM-free tracks that have been downloaded, and if they’ve been “transferred” (as it were) to another user’s library, they become unavailable to the original user. This is just a guess, but it seems to fit my experience. (On a related side note, since this second computer was not yet authorized with SLP’s iTunes account, I needed to authorize it — which was triggered by attempting to play a song — before iTunes Match would work properly.)

What if your music is no longer available in the iTunes store? I’m sure this is one of the most commonly asked questions about iTunes Match, and I’m sure Apple has given very reassuring scripted answers, but it still remains as perhaps the biggest risk you take in trusting your music to the cloud. Tracks that iTunes fails to match and has to upload should be no problem, but once you’re relying solely on a “matched” track — or, for that matter, a “purchased” or “protected” track — you’re at the mercy of Apple and the record labels keeping the music available. I initially noted this as merely a point to ponder, but during the process of integrating libraries I encountered the problem firsthand. SLP had an album that was DRMed 128 kbps, but which is no longer available in the iTunes Store, at all. Luckily I had it copied to my master library already, or I wouldn’t have even known it existed. As it was, I was stuck with an album of low audio quality and that iTunes refused to load into iTunes Match. (It was “ineligible.”)

I hit upon a hokey workaround solution, one that is flawed mainly in that it results in further compression/degradation of the sound quality of the tracks, but at least it’s a way to get the music into iTunes Match. I burned a CD of the album, then re-ripped that CD back into iTunes, DRM free. (That’s the old school way of circumventing iTunes DRM, circa 2004.) It worked, but of course I’ll always know that the sound quality is sub-128 kbps. (Not that it matters much to me, as it’s an album I’ll probably never listen to.) This led me to a related discovery…

Burning a CD of DRMed tracks, re-ripping it, and uploading the results to iTunes Match will not get you “matched” DRM-free 256 kbps versions. Granted, my sample size here is pretty small — two tracks — but I suspect this is deliberate (if it’s possible). In addition to the aforementioned unavailable album, I found two other tracks from SLP’s library that stubbornly refused to load into iTunes Match, even though the rest of the tracks from the albums they were on were recognized and “matched” with no problem. So I burned them onto a CD, re-ripped the CD, and loaded the tracks into iTunes Match. No match. Just the further-compressed versions based on the original DRMed 128 kbps tracks.

Corrupted files? Are you kidding me? I had been wondering what might happen if files got corrupted, either during upload or download. Unfortunately, I found out. Just another meaningless error message with no real indication of a solution. I’m a few thousand tracks into the “upgrade” process so far, and to this point I’ve had four songs fail to download due to an “unknown error.” The behavior is the same in most cases: the song appears to download several times in quick succession. As soon as the progress bar gets to the end, it starts over again. After maybe 5 attempts, it stops with an error number (sometimes err = -100000, sometimes err = 11111). I think it may be necessary to contact iTunes customer support to resolve the issue, but I want to wait until I’ve finished downloading all of my music, to see if it happens with any other songs first.

So, is it all worth it?

I still have a nagging fear that some kind of catastrophic data loss is just around the corner, but so far I am inclined to say that iTunes Match definitely is worth it. It was delayed by a few weeks and still seems like it may have been rushed out the door, but I am hopeful that most of the current glitches and usability issues will be resolved over time. It would be nice if it “just worked,” as we Apple fanbois are so frequently inclined to say, but knowing the complexity of the task at hand, it’s a nearly superhuman achievement, even flawed as it is today.

At the moment I still have almost 3000 low-quality tracks that are eligible for an upgrade (using Jason Snell’s smart playlists), not to mention countless duplicates to weed out and a few other stray errors (in my nightmares, clouds have exclamation points) to contend with. But I think the biggest testament to the magnitude of Apple’s accomplishment is that it’s actually gotten me excited about “the cloud,” something I’ve looked upon disdainfully for years.

Is iCloud deleting your iCal events? Here’s a possible solution

Like many Apple enthusiasts, I spent much of the day yesterday updating software. Mac OS X 10.7.2, iTunes 10.5, iOS 5, and… iCloud. I’ve been relying on MobileMe for a little over a year to keep my mail, notes and calendars (mostly) in sync. I was not an “early adopter” with MobileMe, so I escaped the first-day glitches that promted Steve Jobs to declare the system’s launch “not our finest hour.”

Less than a day into my experience with iCloud, I’d have to say that this launch also is not Apple’s “finest hour.” There have been numerous complaints today about iCloud mail outages (following what I have observed as several days of flaky MobileMe mail performance). But without a doubt the biggest issue for me personally has been related to iCal.

After completing the iCloud transition yesterday, to my dismay I discovered that all of my iCal events were duplicated! My MobileMe account and my iCloud account were both showing up, with all of the same events. Now, in retrospect, the correct thing to do would probably have been to go to Preferences > Accounts and just delete the MobileMe account from my iCal configuration. But is that what I did? Why, no, of course not! I proceeded to delete all of my individual MobileMe calendars. That appeared to do the trick. The iCloud calendars were still there, and every event was just showing up once.

But then this morning I sat down at my computer and discovered — to my horror — that everything was gone. At some point yesterday, when I wasn’t looking, MobileMe and iCloud synced up, and deleted all of my events.

Time Machine to the rescue!

I opened up my Time Machine backup from yesterday afternoon… sometime just before I had made the iCloud transition. I drilled down to [home]/Library/Calendars. (Note that Library is now a hidden folder, but I have my system set to show hidden files and folders*.) I found the multitude of .ics files that represent each individual calendar event, and dragged them into iCal. At first, things seemed great… until I noticed that one by one, the events started disappearing from my calendar again! Apparently iCloud didn’t like having these events show up in the calendar in this way — probably because it recognized them as being events I had “deleted” yesterday — so it “helpfully” removed them again.

AAAAARGH!!! How am I supposed to get these events back into iCal when iCloud just deletes them as soon as they’re added?! Then it hit me… you don’t have to put events into iCloud calendars.

iCal also allows you to created local calendars (“On My Mac”). My solution was to — temporarily — create new “On My Mac” calendars, add the events to those calendars, then export those calendars and import them back into the iCloud calendars. (Then the “On My Mac” calendars can be deleted.) It worked!

Here are step-by-step instructions to do what I did, in case you’ve found yourself in the same conundrum.

1. Find the old calendars in your Time Machine backup. You could open Time Machine to do this, but I like to just explore the disk in the Finder. (The remaining instructions assume you’re taking my approach.) The most important thing is to determine the date and time when your last “good” iCal backup would be. Drill down into that backup to your home directory (that would be something like [drive name]/Users/[username]), and then to Library/Calendars. (Remember that Library may be hidden; if so, see the footnote below.) You’ll see one or more weirdly-named folders. Each of these represents a separate calendar in iCal. Inside each is a directory called Events, and inside that are all of the events on that calendar, each with a filename ending in .ics. If you have more than one calendar folder, you can tell which calendar this is by selecting one of the events in the Finder; its icon will show its date and title. Keep this folder open; you’ll need to come back to it in a later step.

2. Create a new “On My Mac” calendar in iCal. Go to File > New Calendar > On My Mac. Call this calendar whatever you want. If you have multiple calendars, like I do, you’ll need to repeat this process for each of them separately (to keep your events from all getting jumbled together in one calendar).

3. Set the new “On My Mac” calendar as the default calendar. This can be found under iCal > Preferences > General > Default Calendar. When you drag events into iCal, it automatically assigns them all to the default calendar, so this is a pretty important step. Reassigning the events to a new calendar once they’ve already been imported can be a pain.

4. Drag all of the backed-up events into iCal. Go back to the Time Machine backup window you left open in step 1, select all of the .ics files, and drag them into the iCal window. Depending on how many there are, it may take a while for them all to load. Once they’re in, proceed to the next step.

5. Export the “On My Mac” calendar. It can be tricky to make sure you’re getting iCal to export the correct events. Click the Calendars button in the upper left of the iCal window (on the brown “binding” of the cutely skeuomorphic interface), find the “On My Mac” calendar that you’ve added all of the events to, and right-click (Control-click) that calendar to get a contextual menu. Click Export... and follow the prompts. I recommend saving the exported file to your desktop.

6. Set the appropriate iCloud calendar as the default calendar. This is a repeat of step 3, but this time you’re changing it to the iCloud calendar you want the events to be loaded into.

7. Import the exported calendar file into the iCloud calendar. Go to File > Import > Import... and locate the file you created in step 5.

8. Delete the “On My Mac” calendar. Once you’ve completed the import (and have confirmed that the events are not disappearing), you can safely delete the “On My Mac” calendar you created. Click the Calendars button in the brown “binding” again, right-click (Control-click) the “On My Mac” calendar, and select Delete from the contextual menu.

9. There’s no step 9!

* To get your system to show hidden files and folders, open up Terminal and type this: defaults write com.apple.Finder AppleShowAllFiles TRUE then hit Return, type this: killall Finder and hit Return again.