Do Reviews Matter?
Here is a very interesting thread over at EDCF regarding the relative merits of the Kershaw Cryo. Some know-it-all chimes in with his big mouthed opinion (that was me, of course) and things kinda went south from there. Kripto's point on page three with regard to my last substantive comment was well-placed (and well-deserved) and it got me thinking--do reviews matter?
The answer is no, not in the long run. But then again, depending on how you define "the long run" lots of things cease to matter. Even in the short term, I think reviews don't really matter all that much. Doctor saves a kid's life in the ER--that matters. Innocent man exonerated by DNA--that matters. Compared to those things, reviews don't matter.
But, if you are in the market to buy something, then knowledgeable and systematic assessments of things can help you out. In that sense, reviews do matter. If you are the kind of person that wants to make every penny go as far as it can in terms of getting you more of what you want, then why not spend a few minutes, read a review, and get another perspective?
Reviews and Objectivity
Daniel Patrick Moynihan (a Democrat that worked for two Democratic, JFK and LBJ, and two Republican Presidents, Nixon and Ford, and crossed party lines when voting) famously said: "people are entitled to their own opinions, but not their own facts." The problem is, in today's age of perpetual cable news, there are no facts of agreement anymore. Every fact is surrounded by and contextualized by a nebula of quasi-facts and depending on where you get your facts from the cloud surrounding it takes on a different partisan feel. If you are a liberal you get your facts from MSNBC. If you are a conservative you get your facts from Fox News. If you are hipster that only wants to know enough about the news to make snarky, funny comments at a party you get your facts from Jon Stewart. This proliferation of "facts" makes it hard to determine what the heck is going on.
The Internet and forums boards have the same effect. Everyone has an equal voice and those willing to type the most seem to be the loudest. But the truth is, in this morass of opinion, there are still undeniable facts. These facts should form the basis of any review and I do my very best to tailor what I write to represent those facts.
Reviews cannot be objective. If they were, they would read like spec sheets. "Light A produces 300 lumens on high, 100 on medium and 1 on low." Reviews, at their heart, require analysis and analysis requires both facts and opinion. Ours is an age of relativity, in part because of the lack of facts of agreement, but also because anyone can get access to a soap box. I did. I paid $10 got a domain name and started this blog. I have no expertise. I have no engineering background. I do try to write clearly, have an honest review process, and ground my opinions in fact, but other than that there is no secret. I don't have special knowledge and I have very limited special access (it is true that companies send me products to review and BladeHQ is kind enough to do the same), but my opinion is no better than anyone else's opinion.
What I hope to do is to set my opinion apart by writing well, being transparent, and being systematic. I hope that by hewing as close to the relevant facts as possible, you can understand my opinion, and hopefully agree with it. All reviews are a form of persuasive writing and in writing as in film, it is always better to show and not tell. I could tell you something is a piece of crap, or I could show you all of the things that led me to that conclusion and you yourself could form that opinion on your own. I hope I do the latter, because, after all, it is vastly more persuasive to lay things out and have you put them together in the same way. It is like a mini science experiment. Repeatability demonstrates reliability.
This, of course, begs another question--whose reviews matters? For me, there is a sensible answer. Three things count for me when I am trying to figure out whose opinion to listen to:
1. Does the person have a lot of relevant experience?
2. Does the person have tastes and objectives similar to mine?
3. Does the person arrive at their opinions in a fair, systematic, and transparent way?
The closer the person is to matching all three of those things the more likely I am to value their opinion. NOTE: all of the reviewers in this article can be found on the links page of this site.
Experience
If you could, wouldn't you want to test out every single product in the class of products your looking at and decide based on that which one you want to buy? Unfortunately, none of us can test every knife or light out there for a month, decide, based on that, and buy what we want. There is not enough time nor do most of us have enough cash. Instead, reviews and reviewers allow us to short circuit the product deluge and present us with Cliff's Notes versions of those month-long trial periods.
Better reviewers have a wide range of experiences with products. The more experience the better. A guy like Selfbuilt is a perfect example of this. Selfbuilt's reputation among the flashaholics around the world is second to none. Companies send him review copies of virtually every light being produced and the end result is a MASSIVE library of reviews. If there is a production light released, chances are Selfbuilt will get his hands on it.
Over time that experience creates a certain basis of knowledge. You learn what works and what doesn't, what's important and what's just marketing hype. While I have no where near the library of reviews Selfbuilt does, I do see this refinement-of-knowledge process happening in my own reviews. After 30 folding knife reviews, I have a pretty good sense of how steels perform. Now sometimes performance varies, like I have seen with 154CM, and that is something that is challenging for reviews. But if you are consumer and you don't have access to lots and lots of review samples, that challenge can be virtually insurmountable. When I bought and used by Sequel, with its lemon batch of 154 CM, I had such a negative experience that it turned me off to the steel and Benchmade for more than two years. Having used more knives with this steel (the blade on the Skeletool CX and the Mini Grip) has shown me that the experience was just an aberration, but without more data points, one random bad experience taints a person's assessment of things.
The more experience the better because with that experience comes a certain kind of knowledge, maybe not expertise exactly, but something like more representative knowledge. This more representative knowledge is based on an increased sample size and with a larger sample size the knowledge one has becomes more accurate and less likely to be polluted by random good or bad.
Roger Ebert has watched probably something like 20,000 movies. Does that, alone, make him a good reviewer? Nope. But 20,000 movies can't make him worse. In a reviewer, the more experience the better.
Similar Tastes
Nutnfancy is probably the most important single reviewer of gear and EDC stuff. He has a massive audience, great access to stuff, a systematic approach, and a pretty transparent set of tests and standards. My big issue with him is that I don't think he places enough emphasis on quality. For him there seems to be a threshold and once met, nothing else, performance or quality-wise matters. I understand that approach. It is very helpful. I still religiously watch his reviews. But for me, he has less sway because his tastes and objectives diverge pretty significantly from my own.
Compare Nutnfancy then to a reviewer like Oromoto. Oromoto's channel is populated with exotic and flashy custom blades. Its rare for a knife to be in front of his camera with a price tag less than a kilobuck. I can appreciate his tastes. He certainly places an emphasis on quality and uniqueness, but again, I find something lacking. Sometimes the best knife is the one that has been constantly and thorough vetted for 20 or so years, regardless of whether it has super conductor bolsters. It may not cause heart palpitations in bling fans, but remember, knives are, at root designed to cut stuff.
Both Nutnfancy and Oromoto are excellent reviewers but their tastes vary a bit from mine. My tastes synch up more with a reviewer like Spydercollector. Its not just his love of Spyderco knives that does it but is appreciation of small, well built knives and knives that are strictly designed for utility tasks. He also a clean aesthetic sensibility in his photography, presentation, and preferences.
Fair, Systematic, and Transparent
This is perhaps the most difficult thing to figure out about a reviewer. What do they do to test their review samples? How do they get them? What do they do with them afterwards? How do they assess them prior to the review.
I can't speak to others, but with me it is pretty simple. Most of my review samples either come from the company that made them, BladeHQ, or I purchased them or got them in trade. I always note in the review where they come from so that you know. I also have a strict policy of not benefiting in any way from the site, let alone from the reviews. I give away every single item that I have been allowed to keep and that tradition will continue. Furthermore, I don't benefit from the AdSense money or the BladeHQ commissions. Those funds all go towards other giveaways. Right now I have given away about $600 worth of gear and I hope to keep adding to that number. In terms of tests, I always discuss what I did in a given review, what I cut, how I carried and used the item, how it performed in various tasks. I do not write reviews without having experience with the product, usually about two weeks of exclusive carry. In terms of assessment, I created the scoring system so that the method used in a review is clear. You want to figure out what I liked and didn't like, you can read the review itself or just check the scores in each category.
My hope is that the reviewing system I use is as clear, fair, systematic and transparent as possible. I hope that you never have a suspicion that something got an unfairly high score because of its source or its brand.
Conclusion
All of this is a long winded way of saying that reviews matter, to the extent that you care about how you spend your money. They also matter to the extent that they are done correctly, match your tastes, and are based on relevant experience.
The Cryo thread is a perfect case in point about how to process reviews. Lots of people aren't as finnicky as I am about fit and finish. I get that. A lot of people aren't willing to pay $300 for a knife no matter how good it is. I also get that. I understand, as well, that a lot of this is preference. Some folks like heavy knives, some don't.
But in this morass of opinions, there are facts. The fact is that the Cryo is very heavy for its size. I also believe that few people want a heavy knife. I think, in fact, they want a sturdy knife and that usually requires a good deal of bulk, so for these folks heavy isn't an issue, its a side effect of sturdy. But if you went to these folks and said you can have a sturdy knife that weighs 3.0 ounces or one that weighs 4.5 ounces, all other things being equal, no one would choose the heavy knife. The facts are also unavoidable when it comes to the awful thumb studs. They do not work as blade stops, so that is not a reason to have them. They snag on the pocket because of their size, making them less than optimal. Finally, they are just inferior as a deployment method on the knife when compared to the flipper. The thumb studs stink. That last part is an opinion, but all of the supporting elements for that conclusion are facts.
Disagreeing over the quality of gear is part of the appeal of tools and finding what works for you. But objective assessment also has a part. This is not all opinion. Realizing that can help you save money and get more of what you want. I don't think people are dumb because they like the Cryo. I think Kershaw could have done better. That's it. No personal judgments or slights.
In the end reviews help. It is always nice to get some insights from people with experience and knowledge, whether it about buying a car or purchasing a flashlight.
The answer is no, not in the long run. But then again, depending on how you define "the long run" lots of things cease to matter. Even in the short term, I think reviews don't really matter all that much. Doctor saves a kid's life in the ER--that matters. Innocent man exonerated by DNA--that matters. Compared to those things, reviews don't matter.
But, if you are in the market to buy something, then knowledgeable and systematic assessments of things can help you out. In that sense, reviews do matter. If you are the kind of person that wants to make every penny go as far as it can in terms of getting you more of what you want, then why not spend a few minutes, read a review, and get another perspective?
Reviews and Objectivity
Daniel Patrick Moynihan (a Democrat that worked for two Democratic, JFK and LBJ, and two Republican Presidents, Nixon and Ford, and crossed party lines when voting) famously said: "people are entitled to their own opinions, but not their own facts." The problem is, in today's age of perpetual cable news, there are no facts of agreement anymore. Every fact is surrounded by and contextualized by a nebula of quasi-facts and depending on where you get your facts from the cloud surrounding it takes on a different partisan feel. If you are a liberal you get your facts from MSNBC. If you are a conservative you get your facts from Fox News. If you are hipster that only wants to know enough about the news to make snarky, funny comments at a party you get your facts from Jon Stewart. This proliferation of "facts" makes it hard to determine what the heck is going on.
The Internet and forums boards have the same effect. Everyone has an equal voice and those willing to type the most seem to be the loudest. But the truth is, in this morass of opinion, there are still undeniable facts. These facts should form the basis of any review and I do my very best to tailor what I write to represent those facts.
Reviews cannot be objective. If they were, they would read like spec sheets. "Light A produces 300 lumens on high, 100 on medium and 1 on low." Reviews, at their heart, require analysis and analysis requires both facts and opinion. Ours is an age of relativity, in part because of the lack of facts of agreement, but also because anyone can get access to a soap box. I did. I paid $10 got a domain name and started this blog. I have no expertise. I have no engineering background. I do try to write clearly, have an honest review process, and ground my opinions in fact, but other than that there is no secret. I don't have special knowledge and I have very limited special access (it is true that companies send me products to review and BladeHQ is kind enough to do the same), but my opinion is no better than anyone else's opinion.
What I hope to do is to set my opinion apart by writing well, being transparent, and being systematic. I hope that by hewing as close to the relevant facts as possible, you can understand my opinion, and hopefully agree with it. All reviews are a form of persuasive writing and in writing as in film, it is always better to show and not tell. I could tell you something is a piece of crap, or I could show you all of the things that led me to that conclusion and you yourself could form that opinion on your own. I hope I do the latter, because, after all, it is vastly more persuasive to lay things out and have you put them together in the same way. It is like a mini science experiment. Repeatability demonstrates reliability.
This, of course, begs another question--whose reviews matters? For me, there is a sensible answer. Three things count for me when I am trying to figure out whose opinion to listen to:
1. Does the person have a lot of relevant experience?
2. Does the person have tastes and objectives similar to mine?
3. Does the person arrive at their opinions in a fair, systematic, and transparent way?
The closer the person is to matching all three of those things the more likely I am to value their opinion. NOTE: all of the reviewers in this article can be found on the links page of this site.
Experience
If you could, wouldn't you want to test out every single product in the class of products your looking at and decide based on that which one you want to buy? Unfortunately, none of us can test every knife or light out there for a month, decide, based on that, and buy what we want. There is not enough time nor do most of us have enough cash. Instead, reviews and reviewers allow us to short circuit the product deluge and present us with Cliff's Notes versions of those month-long trial periods.
Better reviewers have a wide range of experiences with products. The more experience the better. A guy like Selfbuilt is a perfect example of this. Selfbuilt's reputation among the flashaholics around the world is second to none. Companies send him review copies of virtually every light being produced and the end result is a MASSIVE library of reviews. If there is a production light released, chances are Selfbuilt will get his hands on it.
Over time that experience creates a certain basis of knowledge. You learn what works and what doesn't, what's important and what's just marketing hype. While I have no where near the library of reviews Selfbuilt does, I do see this refinement-of-knowledge process happening in my own reviews. After 30 folding knife reviews, I have a pretty good sense of how steels perform. Now sometimes performance varies, like I have seen with 154CM, and that is something that is challenging for reviews. But if you are consumer and you don't have access to lots and lots of review samples, that challenge can be virtually insurmountable. When I bought and used by Sequel, with its lemon batch of 154 CM, I had such a negative experience that it turned me off to the steel and Benchmade for more than two years. Having used more knives with this steel (the blade on the Skeletool CX and the Mini Grip) has shown me that the experience was just an aberration, but without more data points, one random bad experience taints a person's assessment of things.
The more experience the better because with that experience comes a certain kind of knowledge, maybe not expertise exactly, but something like more representative knowledge. This more representative knowledge is based on an increased sample size and with a larger sample size the knowledge one has becomes more accurate and less likely to be polluted by random good or bad.
Roger Ebert has watched probably something like 20,000 movies. Does that, alone, make him a good reviewer? Nope. But 20,000 movies can't make him worse. In a reviewer, the more experience the better.
Similar Tastes
Nutnfancy is probably the most important single reviewer of gear and EDC stuff. He has a massive audience, great access to stuff, a systematic approach, and a pretty transparent set of tests and standards. My big issue with him is that I don't think he places enough emphasis on quality. For him there seems to be a threshold and once met, nothing else, performance or quality-wise matters. I understand that approach. It is very helpful. I still religiously watch his reviews. But for me, he has less sway because his tastes and objectives diverge pretty significantly from my own.
Compare Nutnfancy then to a reviewer like Oromoto. Oromoto's channel is populated with exotic and flashy custom blades. Its rare for a knife to be in front of his camera with a price tag less than a kilobuck. I can appreciate his tastes. He certainly places an emphasis on quality and uniqueness, but again, I find something lacking. Sometimes the best knife is the one that has been constantly and thorough vetted for 20 or so years, regardless of whether it has super conductor bolsters. It may not cause heart palpitations in bling fans, but remember, knives are, at root designed to cut stuff.
Both Nutnfancy and Oromoto are excellent reviewers but their tastes vary a bit from mine. My tastes synch up more with a reviewer like Spydercollector. Its not just his love of Spyderco knives that does it but is appreciation of small, well built knives and knives that are strictly designed for utility tasks. He also a clean aesthetic sensibility in his photography, presentation, and preferences.
Fair, Systematic, and Transparent
This is perhaps the most difficult thing to figure out about a reviewer. What do they do to test their review samples? How do they get them? What do they do with them afterwards? How do they assess them prior to the review.
I can't speak to others, but with me it is pretty simple. Most of my review samples either come from the company that made them, BladeHQ, or I purchased them or got them in trade. I always note in the review where they come from so that you know. I also have a strict policy of not benefiting in any way from the site, let alone from the reviews. I give away every single item that I have been allowed to keep and that tradition will continue. Furthermore, I don't benefit from the AdSense money or the BladeHQ commissions. Those funds all go towards other giveaways. Right now I have given away about $600 worth of gear and I hope to keep adding to that number. In terms of tests, I always discuss what I did in a given review, what I cut, how I carried and used the item, how it performed in various tasks. I do not write reviews without having experience with the product, usually about two weeks of exclusive carry. In terms of assessment, I created the scoring system so that the method used in a review is clear. You want to figure out what I liked and didn't like, you can read the review itself or just check the scores in each category.
My hope is that the reviewing system I use is as clear, fair, systematic and transparent as possible. I hope that you never have a suspicion that something got an unfairly high score because of its source or its brand.
Conclusion
All of this is a long winded way of saying that reviews matter, to the extent that you care about how you spend your money. They also matter to the extent that they are done correctly, match your tastes, and are based on relevant experience.
The Cryo thread is a perfect case in point about how to process reviews. Lots of people aren't as finnicky as I am about fit and finish. I get that. A lot of people aren't willing to pay $300 for a knife no matter how good it is. I also get that. I understand, as well, that a lot of this is preference. Some folks like heavy knives, some don't.
But in this morass of opinions, there are facts. The fact is that the Cryo is very heavy for its size. I also believe that few people want a heavy knife. I think, in fact, they want a sturdy knife and that usually requires a good deal of bulk, so for these folks heavy isn't an issue, its a side effect of sturdy. But if you went to these folks and said you can have a sturdy knife that weighs 3.0 ounces or one that weighs 4.5 ounces, all other things being equal, no one would choose the heavy knife. The facts are also unavoidable when it comes to the awful thumb studs. They do not work as blade stops, so that is not a reason to have them. They snag on the pocket because of their size, making them less than optimal. Finally, they are just inferior as a deployment method on the knife when compared to the flipper. The thumb studs stink. That last part is an opinion, but all of the supporting elements for that conclusion are facts.
Disagreeing over the quality of gear is part of the appeal of tools and finding what works for you. But objective assessment also has a part. This is not all opinion. Realizing that can help you save money and get more of what you want. I don't think people are dumb because they like the Cryo. I think Kershaw could have done better. That's it. No personal judgments or slights.
In the end reviews help. It is always nice to get some insights from people with experience and knowledge, whether it about buying a car or purchasing a flashlight.