How We Test Products at SPY.com
SPY.com is a discovery and reviews platform, connecting you with the latest in the world of products. We focus on men’s grooming and lifestyle, technology and gadgets, and health and fitness. We bring you under-the-radar releases and top-rated picks from across the globe. Over the past few years, the SPY team has tested thousands of products, and we have a simple goal: to help you shop smart. We will never recommend a product that we wouldn’t buy ourselves, and we love finding awesome new gear to share with SPY’s readers. So how do we do that?
The SPY editorial team includes product testers, commerce journalists, and product reviewers with decades of combined experience testing products. From our office in New York and Los Angeles, or from our own homes, we’re always trying out new products for SPY readers.
To find the best products, we scour the web, carefully read customer reviews, compare product specifications, and talk to experts. When writing about the best face wash for men, we’ll consult dermatologists in addition to relying on our grooming writers’ expertise in this subject area. Most importantly, we have implemented a rigorous product testing system to ensure SPY readers are getting the best possible product recommendations.
To be as transparent as possible, we wanted to outline our product testing process in detail. You can also refer to our About Us page for more information on our editorial process.
The Testing Process
Our editors and writers hand-select every product we feature on SPY. Our coverage includes new product releases, trends and news that’s relevant to our readers, in addition to product reviews based on in-depth hands-on testing. Brands cannot pay SPY for positive reviews, and every review and shopping guide on our site is based purely on our editorial recommendations.
Though we can’t try every product we cover, the SPY team has rapidly ramped up our product testing efforts. That includes bringing on a reviews editor and new team members to expand our hands-on testing capacity.
If you’ve ever been curious to take a peek behind the shopping guide and see how we chose a specific product, these are the steps we take when writing our reviews and review roundups.
Step 1: Research
We start by conducting market research to learn more about the overall product category, what the top-performing brands are, which products people have questions about and which are the biggest players in the space.
Depending on the product category, the number of products we test can vary wildly. For a product with thousands of choices, like men’s underwear, we narrowed down our list of test products to about 100. For adjustable dumbbells, that number was closer to 15. After we have our list of products we need to test to determine rankings, we start calling in samples.
Part 2: Calling in Sample Units
In order to conduct reviews, brands will sometimes send out sample units. These products are typically free for publishers like SPY. In return, brands typically request that we publish a review, but we make no guarantees about coverage. We have tested many products that we ultimately decided not to feature on the site. In some cases, sample units are not available. In those cases, we will purchase products for our reviewers to evaluate.
Part 3: Creating Judgement Criteria
As the sample units are arriving, we create a list of review criteria. This is a detailed list of specifications and features we think are relevant to overall product quality. The criteria will vary widely depending on the specific product category. We create internal spreadsheets to centralize our criteria, helping to reduce bias when products are evaluated by different testers.
Part 4: Testing
Now we get to the fun part: testing. SPY has about a dozen experienced product testers producing reviews at present. We spend as much time as possible with each product, using it in a variety of contexts and taking note of everything.
When testing a product, our reviewers evaluate and rate how it performs according to the criteria we’ve set. This can be a time-consuming process, and sometimes it can take three months or longer to fully produce a review like this. Reviewing products in this way allows us to fairly compare how similar products perform. We log detailed notes about how well the product performs in the given criteria.
Using the criteria we’ve established, we then evaluate the products and award them a numeric score. This is then used to help finalize the rankings in a review. This allows us to create rankings that are as objective as reasonably possible. Many of our reviews will feature a section that details how we tested that particular product category.
Part 5: Re-Testing and Updating
The last step in our testing process happens weeks, or even months, after our initial testing round. We regularly update our reviews and shopping guides with the latest products and new thoughts on products we’ve already tested. Sometimes our experience of a product changes after spending more time using it, and so we need to adjust our rankings accordingly. Likewise, if there’s a new product that’s garnered a lot of attention, we’ll test it to see how it fits into our existing ranking.
How We Test Products FAQ
1. How Do We Get Products For Testing?
We use a number of different channels to acquire products for testing. The most common is requesting sample units from brands, who, more often than not, are more than willing to send them. If a product isn’t available as a sample unit, we’ll purchase it ourselves using a designated section of our site’s budget.
2. How Long Do We Test Products?
The amount of time we test a product largely depends on the nature of the product itself. More expensive, high-ticket items require longer testing because we want to ensure they’re worth the investment.
Most of the time, we spend 1-3 weeks with a product, testing all of the features and compiling our data into a spreadsheet. If a product is a big release, unique, or otherwise noteworthy, we’ll often dedicate an entire article to reviewing it. In other cases, we’ll test many different products for a review of the best products in that category.
Sometimes, we will update a review as we have new experiences with the product. When this happens, we will note in the article that the review has been updated with new information.
3. What Do We Do With Products After We’re Done Testing?
It depends on the product and the brand. Some brands request the unit be returned within a certain window, or after we’re done testing, while others encourage us to keep or donate the products after we’re done.
Some of our editors have kept sample units, and continue to test them. We frequently upgrade reviews after 3, 6, 9 and even 12 months of use, to give our readers an even more in-depth view of a product.
4. Do You Use a Number Scale?
Yes, we have an internal scoring system based on a 100-point scale. This lets us objectively compare products to one another and informs our overall reviews. We keep our scoring system private, but we always disclose the criteria we use when reviewing a product.
5. What Do You Do If You Don’t Like a Product, Or Have a Bad Experience?
Some product review websites have a “no bad reviews” policy, which is meant to keep the website on good terms with brands. At SPY, we will always be honest about our experiences with products, good and bad. Typically, when we don’t like a product, we simply don’t include it in our shopping guides. However, for especially notable products, we do occasionally publish negative or mixed reviews. Our guiding principle is to promote products we fully believe in and love.
6. How do you make your money?
SPY earns money when our readers trust our recommendations enough to purchase a product. When readers click a link on SPY and go on to purchase a product at sites like Amazon or Target, we may earn a commission on that purchase. This is standard practice for commerce publishers. Our full affiliate disclaimer appears on all of our content. It reads:
Our editors independently select the products we recommend. We may earn a commission on items bought through our links. Why Trust SPY?
In addition, we run advertising and sponsored content campaigns. We always label sponsored content using a “SPONSORED” label, and ads are always clearly marked. We do not accept money for reviews or guest posts under any circumstances.