My Favorite IT Disasters!

I’m an IT guy, and I’ve seen my share of colossal failures in the workplace over the years. Recently there have been some “IT disaster” threads at Ars Technica and Reddit which got me thinking about my own disaster stories. Here are four of my favorite. Note that all but the last one come from the same job at a third-party IT company I used to work for.

THE RAID ARRAY FROM HELL

I was sent to a real estate firm to swap out a failing drive in a RAID 5 array. Thanks to the LEDs on each drive, I quickly spotted the drive I was to replace. I opened the RAID utility on the server to make extra-sure I was replacing the correct drive. The software verified that yes, the drive with the blinking LED is failing. I removed the old drive, put the new drive into the slide, and placed it in the array. The software recognized the new disk and asked if I wanted to rebuild the array. I clicked yes, and for the next 20 seconds or so everything seemed normal. But then the server BSOD’d. When I tried to reboot it I got the dreaded “SYSTEM DISK NOT FOUND” error message.

Come to find out, this server was one of the first my boss built himself after he started his company. For reasons only he knows, he installed Windows 2000 Server on to the RAID 5 array itself. Now this isn’t a “disaster” per se. The RAID software should have been able to rebuild itself without taking down the entire array. But installing an operating system onto a RAID 5 array is just something I’ve never seen done, ever. I’ve only worked with small and medium-sized businesses (SMB). In an SMB environment, you’d typically install Windows Server onto a regular hard drive or possibly a RAID 1 array. You then create the RAID 5 array as a separate disk to store vital data. And you do it this way because the operating system files just aren’t that valuable, and installing Windows on a standard (or RAID 1) drive is significantly less complicated (as a general IT rule, the fewer points of failure or complexity the better). If you have no idea what I’m talking about, imagine installing Windows on a regular hard drive, and putting all your important data on a heavy-duty, “guaranteed to never fail” external hard drive. If the Windows drive dies, it’s no big thing to go to Best Buy, get a new hard drive, reinstall Windows, then reinstall the external drive, right? Same theory, different implementation. And this real estate agency had tried to become as paperless as possible, so everything was on the server… which was now dead.

The icing on the cake was that the owner, an attorney with zero sense of humor but a giant sense of ego, flipped out because… “[my boss at the IT company] told me that we didn’t need backups because of this RAID thing!” I tried explaining that RAID is not a backup, just a way to make hard drives more fault tolerant. But he seemed to be of the opinion that my boss told him otherwise. Which put me in a pickle. Anyone who’s worked in IT knows that you can say one thing, even in as simple English as possible, and clients hear another. So it’s possible that my boss said no such thing, but the client interpreted it as such. On the other hand, I knew my boss would tell clients anything he felt they wanted to hear to make a sale. Perhaps my boss was afraid that the client wouldn’t sign the contract if he added a $1,200 tape drive into the mix. Maybe my boss was planning to sell him some kind of tape or online backup later on. Whatever the case, I had a dead server and a highly pissed off attorney to deal with. And it wasn’t pretty. I took the server back to the office and rebuilt it from scratch – not installing Windows Server on the RAID 5 array this time. My boss claimed to have recovered more that half the data off the old array… but the recovery software only pulled the file names; the actual files themselves were just a bunch of binary gibberish. So the firm started over from scratch.

LESSONS FROM THIS ORDEAL: RAID is not a backup. Don’t lie to clients, and make them understand, no matter what you have to do, what they’re signing up for.

Continue reading “My Favorite IT Disasters!”

RETRO TECH: Microsoft Mira

Poor Microsoft can never seem to get a break. Any time they mimic an existing product (like say, their Zune to Apple’s iPod) critics say they’re just copying someone else’s work. But when they do come up with something cool, no one seems to buy it. That’s exactly the case with Mira, a “Smart Display” device:

mira

Here’s how it worked: the tablet computer ran Microsoft Windows CE for Smart Displays (thankfully shortened to just “Smart Display OS”). It was nearly instant-on, and it would automagically connect to a desktop PC running Windows XP via Remote Desktop. So you could be sitting at your desk working on something and suddenly decide to go sit on the downstairs sofa, or the back deck, or the big comfy bed. You’d bring your Smart Display with you and BOOM! in seconds you have your desktop on the screen, and can continue what you were doing.

While it was a really cool idea – hell, I still like the idea of a Smart Display… imagine a 19″ model with today’s thin hardware that could sit on a stand like a regular monitor until you wanted to leave your desk – the actual implementation of the device left a lot to be desired.

For one thing, the 802.11b Wi-Fi of the day simply wasn’t fast enough to allow wireless videos, and Remote Desktop didn’t have any video optimizations at the time. The touchscreen tech was subpar at the time. The battery life wasn’t nearly as good as a modern iPad or Android tablet. The tablet was as thick and heavy as a notebook, but was useless without a desktop PC to “mate” with. What’s worse, Microsoft desperately wanted vendors to sell them in the $500 range, but devices were introduced at between $1,000 and $1,500… at at time when a decent notebook with far more functionality could be had for $600.

But the funniest thing about the devices was Microsoft’s own licensing issues. Because only Windows XP Professional (or higher) allowed Remote Desktop connections, millions of consumers running Windows XP Home were out of luck. But even if you were lucky enough to run XP Pro, that OS only allowed a single session, meaning that once you connected to your computer with the Smart Display, the desktop would be locked and no one else could use it until you shut down the Smart Display device. And because of XP’s RDP limitations, only one device could connect to a PC at a time. So Mom, Dad, Johnny and Susie couldn’t use their Smart Displays at the same time… unless they had individual computers to connect to.

Of course, Microsoft probably could have fixed the RDP\licensing issues if Smart Displays really took off. But they didn’t. They were released in early 2003 and discontinued in December of that year.

RETRO TECH: Philips Velo 1

One of the hottest gadgets of Christmas 1997 was the Philips Velo 1. It was a Handheld PC (sometimes called a “palm top”) and it was absolutely tiny:

philips_velo_1

When closed, this bad boy was smaller than most paperback books. It ran a Philips MP3910 chip at a blistering 36.864 MHz (yes, that was sarcasm). It had 4 whole megabytes of RAM and 8 whole megabytes of storage space. It rocked a 480×240 monochrome monitor, had a built-in 14.4kbit/s modem and connected to your computer using RS-232.

It was, compared to even the cheapest Chinese knockoff electronic organizers of today, absolutely awful. But it was also bad ass! I could whip this puppy out at a bar or restaurant and people would crane their necks to get a peek at it. And connecting to the Internet via dial-up on this thing made a lot of people (even me) absolutely giddy! Seriously – it almost seemed like something out of a James Bond movie!

I once worked at place where I had a very specific job; it was made very clear to me by the management that if I had nothing to do, I was supposed to sit at my desk and do nothing. As a contractor, I was not allowed to have an email address. The company’s firewall prevented access to (literally) 98% of the Internet, and I was “strongly discouraged” from bringing in books or magazines to pass the time. So I’d sneak in my Velo 1 and connect to the ‘Net via dial-up. I’d check email and surf a few sites… but the main thing I’d do was talk to this 19 year-old Israeli girl I’d met on ICQ (and no, it wasn’t “that kind” of chatting; she just seemed to be online ALL THE TIME).

While I loved my Velo 1, the main problem with the thing is that it ran Windows CE 1.0. Just typing that made me wince! (Get it? Windows CE? WinCE? Wince?) That OS was a complete disaster. As you can tell by the above picture, the operating system looked just like Windows 95 or Windows 98… which would have been great, except Start Menus and taskbars and system trays are a horrible idea on a device with a 5.1″ screen. And the device, for some godawful reason, supported multitasking, which meant that you’d sometimes have to use the built-in stylus to manually move windows around… on a 5.1″ screen. I was able to eventually upgrade it to (IIRC) Windows CE 2.1, which was slightly better. But still, there just weren’t a lot of apps for WinCE out there, and many of the ones that did exist weren’t that great. And synching the device via serial port seemed to take FOREVER, even though the amount of data being transferred wasn’t all that much.

It’s really amazing that this device was almost “cutting edge” in its day… but less than three years later Compaq would release the insanely popular iPaq 3630. The iPaq had a vivid color screen, a 206 MHz processor, supported Wi-Fi via CF card, had a vast array of accessories (including a folding keyboard, which I used to take notes in meetings), and syncing with a desktop PC didn’t totally suck, either. The Velo 1 seemed like 1950s black and white TV, while the iPaq seemed like a late 1980s color TV. But a mere 30 months separated the two products!

YouTube and Aspect Ratios

It drives me nuts: you search and search for a video on YouTube… but when you finally find it, some idiot has encoded it in the wrong aspect ratio:

yt_ar_01s

You can (sort of) work around this if you have the ever-popular VLC video player installed on your computer. Just open VLC and choose Media > Open Network Stream, paste the YouTube URL into the “Network URL” box and press “Play”:

yt_ar_02

When the video appears, keep pressing the “A” key until the correct aspect ratio is displayed:

yt_ar_03

This isn’t perfect, but it’ll work in a pinch!

RETRO TECH: IBM Simon

There’s not a single definition of what a “smartphone” actually is. However, most tech historians consider the IBM Simon to be the very first smartphone:

IBM_Simon

It was developed by IBM in 1994 and was the first device to combine a mobile phone with PDA-like features. A concept model was shown off at COMDEX in 1993 and generated a lot of interest, even making the front page of USA Today’s “Money” section the next day. The device allowed users to make phone calls, send and receive faxes, emails and pages and had comprehensive address book and calendar features. It lacked a dedicated number pad, relying instead on a touch screen interface. Perhaps most amazingly of all, the device was able to do all this using DOS!

BellSouth got an exclusive for the device, and sold it for $899 with a two-year contract ($1,305 adjusted for inflation) or $1,099 without a contract ($1,595). The carrier sold 50,000 units before discontinuing the phone in February 1995.

But the funniest thing about it was that it was the first phone to come with the ability to run third-party apps. An Atlanta company called PDA Dimensions developed “DispatchIt”, the only app ever developed for the platform. The app required a desktop PC client ($2,999) and a phone client ($299). Adjusted for inflation, the desktop client and one Simon client would cost $4,788 today! Not surprisingly, PDA Dimensions sold exactly two copies of the software.

Removing Wallpaper over RDP

If there’s one thing in this world I hate it’s the default wallpaper that large OEMs like Dell and HP include with their server products. It’s bad enough that you have to spend $6,000 (or more) on a server, but now, every time you log in remotely, you have to see this:

dell_rdp_sm

It’s not just that it’s a visual annoyance. Loading that wallpaper takes time and bandwidth: the Dell wallpaper is around 1.3MB, and having it load every single time you log on to a server gets old.

The thing is, in most cases the OEMs have configured the servers to display the wallpaper, even if you don’t have a wallpaper configured at the console session. Thankfully, it’s pretty easy to remove this. Just open REGEDIT and go to

HKEY_USERS\.DEFAULT\Control Panel\Desktop

Look for a key called WALLPAPER in the right-pane. If you’d like to remove the wallpaper entirely, just delete the WALLPAPER key. If you’d prefer to just change the wallpaper, change the value of the key to the location of your preferred wallpaper. For example, the default on Dell servers is \windows\system32\DELLWALL.BMP. You can change this to whatever you’d like, no reboot is necessary. Once you’ve changed the key, the next time you RDP into the server the wallpaper will be gone, replaced with the default color background.

Firefox Crashes

OK, I’m probably the last person on earth to find out about this, but if Firefox is crashing on you, restart the browser in safe mode by holding down the SHIFT key when clicking the Firefox icon (if necessary – I find that Firefox rarely crashes when displaying the “Session Recovery” page; just open a new tab with Session Recovery open). The next step is to type about:crashes in the address bar and press enter. If you’ve submitted your crash reports to Mozilla (and you have been doing that, haven’t you?) you’ll see a list of your crash reports:

about_crashes_01

Reports are listed by the most recent, so in most cases you’ll want to click on the one at the top of the column. When you do, a new page will open, connect to Mozilla and download the report:

(click to enlarge)

It might take a few minutes for the report to be retrieved and downloaded. But when it does, you’ll find a wealth of information about the crash, hopefully enough to get you headed towards a fix.

Firefox recently started acting weird on my computer… mostly by crashing every 8 hours (at first), then every 15 minutes, then every 5 minutes, then every 45 seconds. Poking around the ‘Net for a solution, I found out about “about:crashes”, and my crash reports couldn’t have made it any clearer: adblockvideo.dll, part of the AdBlockVideo extension I’d recently installed, was causing the crash. I disabled the extension, and it’s been smooth sailing ever since!

Win7: Changing the “Shut Down” Button

OK, this might be the lamest tip ever, but did you know that you can change the default setting for the “Shut Down” button in the Windows 7 Start Menu? Just right-click the Shut Down button, select “Properties” and choose whichever action you’d prefer from the drop-down box:

win_7_power

Kind of nifty, especially if you use a pre-Windows key keyboard like myself:

win_7_power2

IE and the Default Save Folder

If you’re one of the 14 people still using Internet Explorer, you might have noticed an annoying new “feature” that Microsoft introduced in IE 8 and continued in IE 9: when you save images from different tabs, IE always defaults to the user’s “Pictures” folder instead of the previously used folder.

For example, say you’re at an imageboard like Reddit, and you have opened a bunch of funny images you want to save in to different tabs. You right-click the first image and choose “Save Picture As…” and IE opens the save dialog box in the Pictures folder. You navigate to your c:\MyStuff\pictures\reddit\funny folder and save the image. You then go to the next tab, right-click that image and choose save… but instead of the previous location, IE has defaulted back to the “Pictures” folder. So you have to navigate to the your preferred location. And then repeat the process for all the other pictures in open tabs.

This happens because Microsoft used a method it calls “Loosely Coupled IE” (LCIE) to spawn new instances of IE every time you open a new tab. In other words, every time you open a new IE tab, Windows treats it as a completely new instance of IE. Which is why new tabs revert to default locations, but saving multiple images from the same tab don’t exhibit this behavior. And the reason Microsoft did this is for stability: if one of your open tabs crashes, only that particular instance of IE is killed, not the other instances of IE. So LCIE makes IE much more stable generally, but adds a huge annoyance for saving images.

Can you change this behavior? Yes, but you need to read this first: Trust me, it’s not the usual techno-babble yadda yadda yadda.

You can force IE to run as a single process, but this has two important ramifications: for one thing, all instances of IE will run as a single process, so if one tab crashes, all tabs will crash. Also, forcing IE to run as a single process disables Protected Mode in Windows Vista and Windows 7, because IE is running as a single process, instead of separate instances which Windows can run at different integrity levels.

Continue reading “IE and the Default Save Folder”

Win7: Task Scheduler and Weekdays

In Windows XP, it was easy to configure Task Scheduler to run a certain event on weekdays only:

shutdown2

Windows 7, however, doesn’t have a “weekdays” option, just “Daily”, “Weekly”, “Monthly”, “One time”, “When the computer starts”, “When I log on”, or “When a specific event is logged”:

w7_ts_01

So… how do you configure tasks to only run on weekdays? It’s totally counter-intuitive, but you choose “Weekly” and then choose which day(s) you want the task to run on:

w7_ts_02

It doesn’t make a lot of sense at first, but new method is actually far more powerful than the old “XP way”. You could, for instance, easily create a task that only runs on Mondays, Wednesdays and Fridays using this method, whereas in XP you’d have to create three individual daily tasks that run on Monday, Wednesday and Friday. Or if you had a business that was open every day but Sunday, you could easily schedule a backup to run Monday – Saturday with this method.

Play around with it – I think you’d like it better in the end!