Ryan Tate - Update for Friday, March 11, 2005

Greenspun's review helps illustrate the sorry state of newspaper product reviews. Take, for example, Walter Mossberg of the Wall Street Journal, who according to the New Yorker makes close to $500,000 per year. Mossberg is easily the dean of technology reviewers, and has written some helpful reviews, including one that rightfully sent many customers to MusicMatch, an underdog maker of MP3 players. But his review of Internet telephone provider Vonage is a letdown.

Greenspun notes that Vonage is among the most expensive Internet telephone providers. Mossberg doesn't. Greenspun notes that Vonage has limited service in Europe. Mossberg doesn't. Greenspun also tries to use the Vonage as his primary telephone number and uncovers all sorts of customer service problems in the process. For some reason, maybe because his review was written 10 months earlier, Mossberg doesn't.

I hate to pick on Mossberg, who seems usually to work harder and know more technically than other reviewers. There is a broader problem with newspaper product reviews. The problem is the process.

For one, greedy publishers set insane deadlines, so the reporter reviews the product in an absurdly artificial span of time. For example, I know a video game reviewer who had one weekend to play two or three different titles. The review was due Monday. (Of course, this is for illustrative purposes only, I'm sure *her* review won't suck! Especially if she's reading.)

Look at Greenspun's review. He spent 10 months wrestling with Internet telephone service. Back in 1999, I wrote an online review of an Internet Service Provider after six months. Newspaper editors simply will not give this kind of time.

Newspaper product reviews also suck because the reporter isn't a tough customer. He's an artificial customer, one who half the time hasn't paid for the product in question. And the writer doesn't review the product because he honestly wants it. He reviews it because he's the guy who does the reviews. Or because he got three of them for free in the mail. Or because his editor had an idea about how cellphone mp3-playing videogame handhelds are the wave of the future.

I recently bought some outlining software for my PalmOne handheld comuter called ShadowPlan. This little bitty piece of software has its own Yahoo discussion group with over 20,000 messages! These people spend weeks arguing passionately over the optimal way to implement David Allen's Getting Things Done in this little outliner. They complain about default folder locations, Save dialog boxes, syncing, the desktop software, how files are launched, importing files, linking the outliner to the other Palm software -- the list is nearly endless.

Newspaper reviewers, in contrast, miss the little details that are so critical to the effectiveness of a product. I'm not going to say Mossberg was wrong to recommend the iPod. But it took a regular user of the product like Dave Winer to point out that it's a difficult platform for audio books and unintuitive when you hook it up to your car.

The biggest tipoff that newspaper reviews suck is that they so often sound like promotional material for the service or product in question. For example, in Mossberg's Internet telephone review, he spends just half a sentence trying to figure out why anyone would switch to Internet telephones. Maybe because they hate the phone company? Bzzzt, wrong. If a phone user is not buying Internet access from the phone company, they are buying it from the cable company. And if they hate the phone company, they *really* hate the cable company.

Anyway, after the half a sentence thinking about people, Mossberg starts with the product pitch. Saves you money! Unprecedented flexibility! Easy setup! Call anywhere in the world! Three plans available! They'll throw in a free gift! The kind of stuff you'd see on a brochure, press release or promotional website.

Then Mossberg gets into the product flaws, something you won't see in any sales literature. And I think that's great, and it's what he gets paid the big bucks for, and he doesn't do a criminally bad job of it.

But Greenspun's whole approach is different. He's relentlessly focused on what an Acutal Person Actually Using the Product would want.

He gives some great reasons to switch to Internet phones -- cheap international calling, cheap or free long distance, save some money, one phone number for multiple phones, and so on. His pitch, actually, is more effective than what you would get on the Vonage home page, probably because he is thinking about customer needs rather than how to make lots of money.

And then Greenspun gets into the nitty gritty of actually using the product -- transferring your number, making those promised international calls, getting the thing to forward to your telephone. Does it help that Greenspun actually wants to make this service work so he can keep using it, that he's not just trying to write up a quick review and hit another deadline? Absolutely.

More updates