09 Jun Cloudy Days
I used to be the technical director of an international broadcast facilities group, handling everything from live television to films and radio. This meant I spent a surprising amount of time on-air either filling in for missing DJs, becoming a technology journalist and eventually hosting drive-time programmes. One thing led to another; while still the group Technical Director, I became a DJ both in clubs and on-air. Somewhat addicted to music and with 6 studios available, I got into music production. All of this meant that I received thousands of white-label and promotional tracks and made many hundreds of recordings – everything from commercials to live events and, of course, commercial popular music tracks.
In the ‘good-old-days’, I used to keep my recordings in a wide variety of physical media ‘repositories’: I had a tape library, and a CD storage system, and shelf upon shelf of vinyl. Then along came digital audio tape, mini-discs – and my pride and joy a 1.7GB Micropolis AVLT hard-drive: at £2,500 it was an expensive beast, but it allowed me to edit encoded video without needing to book time in one of our editing suites (although in reality, it didn’t as the computers were too slow to render broadcast-quality video sensibly; but it was an early indicator of how digital technologies would dominate media production).
The downside of having all these varying formats and storage solutions meant that I had to maintain the hardware necessary to play each of these formats. Who, these days has access to a Sony R-DAT player, or even a mini-disc unit. With the old reel-to-reel tapes starting to degrade, I also had to consider how best to preserve these (I hesitate to say ‘ancient relics’ but instead choose …) original and extremely rare recordings so that they didn’t simply disintegrate into a heap of ‘rusty dust’ (I’m sure I met her once).
This meant transcoding all of this material into a digital format; but not just any digital format, I had to choose one that would persist. The problem with data formats being that, as technology and algorithms improve, the older formats cease to exist (have you tried opening a Visicalc spreadsheet recently?). As the underlying standards change, software manufacturers stop supporting the older file formats.
I spent months encoding this material – I had to restore, renovate and recalibrate my old reel-to-reel decks, find a way to recover graphics and animations from some of my favourites but long-deceased SGI kit, and listen to some truly awful studio sessions that I had forgotten, but also knew I should keep to torture future generations. For audio, I chose lossless audio recording formats (I equate audio and image compression to tape hiss – the more times you load and edit compressed data, the more noise you introduce). My workflow for this being to record/capture from the original source into dedicated sub-directories. Once imported, I then copied this information into my standard media library – iTunes. I’ve done this for years – a raw copy on one large drive, a copy held in iTunes for ease of access, and the ability to air-play around the house.
With that background, I can finally get to the point of this blog. Apple introduced iMusic – a cloud-based mechanism to store and play music over all my devices: iPhone, iPad, laptop, main machine, studio machine … It seemed like a good idea to enable this service as it would allow me to get access to my entire 760GB library wherever I was … At least that’s what the hype suggested.
A month ago, I tried to play a rare recording I had made many years ago, but iTunes reported that the original media was missing and helpfully prompted me to find it. With a mild panic setting in, I realised that by enabling iMusic, my version of iTunes had casually deleted, replaced (with completely random music), or corrupted thousands of tracks. I contacted Apple Support – their response – “have you got it backed up?”. In all, iTunes/iMusic seemed to have trashed over 5,700 tracks (these being the ones I know about). Thankfully, I had the original files that I had created when making the digital versions and could use the command-line to identify and reinstate most of the corrupted versions with the originals.
Apple’s response was, in my opinion, simply useless: comments such as “iTunes won’t delete the originals …” didn’t wash – I don’t randomly delete files in my library or rename them to long hex-strings (remember this catchy little tune?).
iMusic won’t sync long files or those greater than 200MB, so these most of my early recordings won’t play on all my devices – the marketing hype really didn’t meet my expectations as a customer. While not overly concerned by selected tracks not available on my ‘phone – I am extremely annoyed that some rare dance-track edits turned into hip-hop from an artist I’d never heard of. If this was a customer database I view it as tantamount to randomly changing details of my customers to those belonging to another company that just happens to share the same hosting service.
WannaCry hit over the last month, and I draw strong parallels between what iTunes/iMusic (allegedly) did to my library and this particular breed of malware. WannaCry encrypted data, leaving strange names and unusable data; network shares and synchronisation services ensured that any online shared resources also became encrypted. The difference with WannaCry being that if you paid a fee, you stood a chance of recovering your data. My subscription to Apple didn’t give me a recovery option!
I am thankful that I never deleted my old files – there was no way I could determine which files had become corrupted, let alone find them in a time-centric back-up (on which date did which file disappear … Find each file in turn, recover and repeat) – it would have taken me years to restore my data. Perhaps then, by sharing my experience, others may avoid similar pain:
- Keep raw format periodic backups – especially for critical (or sentimental) data
- Validate your data on a regular basis to ensure corruption hasn’t crept in
- Rely only on your own backup integrity – especially when using cloud services
- Assume that cloud storage services will replicate corruption as it would any valid data
- Make backups and keep at least one backup offline to negate the effects of damage synchronisation
- Make sure you adopt formats that will stick around
- Storage formats and structures will change so ensure you migrate from old formats before decommissioning systems
The old broadcast adage “keep all the masters and only use copies” saved me in the same way as offline data backups saved many from the effects of WannaCry. iCloud sync dutifully trashed my cloud backups because it simply reflected what iTunes had (allegedly) done to my music. I can’t emphasise this enough: if you’re going to have big digital repositories synchronised with cloud services, make sure you keep the offline backups safe and keep something in place for when your subscription stops, the service crashes, or the software ceases to exist.
Now if only I could just listen to 0000000000122FB0D-000000000DAB4F02 one more time …