Choosing a direction: Learning how to test product ideas and designs

Why do we test?

We don't know what we don't know and it's risky to push product out into the wild with no sense of how it will perform. We've all been part of some project, big or small, that failed. It belly-flopped into your customer pool like a punch to the gut. How do we build a process to avoid that outcome? Testing helps us get there.

I test to validate assumptions and remove opinions from the process. My whole team has ideas about the right solutions but none of us know how our ideas impact others until we get feedback from real users. The discussion should never be "I like" or "I think.” Decisions should be made with "The data says…"

I test to understand the problem from my user’s perspective. I am not my user and I don't know how they feel, what they think, or what they do unless I seek it out. I'm not so good at reading their minds from afar.

I test to learn about the “hidden” things I didn’t know. Humans are irrational and unpredictable. There are always surprises that I can't anticipate, but would love to solve before we build and ship. It's the Rumsfeldian, “Unknown Unknowns.”

I test to measure the impact of my proposed solutions. If I have a benchmark, I can measure whether I'm having a positive, negligible, or negative impact for my users. Why continue to put effort into something that isn't getting the job done when you could learn how to improve on it?

I test to create alignment. The best solution in the world is useless if the team is not in agreement on what it is and how to implement it. By being data-driven I can rally the team around a single vision.

I test because it reduces the cost and time of delivering a successful product.

Planning a test

Contrary to popular belief, the testing is the easy part. What most people forget is the planning part.

“If I had an hour to solve a problem, I'd spend 55 minutes thinking about the problem and five minutes thinking about solutions."

— Einstein

Why: Before you test, make sure you sit down and think about why you are testing. Define your goals. Think a few weeks down the road and what you plan to present to your team about testing findings. What questions do you need to answer? What are the disagreements that need a solution? Are you trying to assess if a user understands the product concept or understands the UI?

Who: The second big question is choosing your audience. Do not recruit by demographics alone! You need to understand the behaviors of your target audience so that you can find the right people. What are the behaviors of the user you are looking for. What level knowledge is useful. Crownpeak makes a CMS. I once recruited for Marketers and ended up speaking with Events and Social professionals instead of digital marketers who use a CMS. Think carefully about finding the right people.

"You can learn valuable things by asking the right people the wrong questions. If you are talking to the wrong people it doesn't matter what you ask."


— Erika Hall
Author of Just Enough Research

Data and goals: The last big question is around what data you need to capture to meet your goals and how you plan on capturing it. Preparing a note-taking sheet and an analysis document before you start is a sure-fire way to save time and get better data. What kind of data, you ask? Here are some examples.

• Trends - Similar patterns across participants
• Positive/Negative outcomes - What worked well and what didn't
• Priorities - Findings that are higher risk, higher impact, higher value
• Big Picture - Can you decipher symptoms from a root cause
• Quotes - "We all love quotes!" - Ari Weissman
• Opportunities - New feature ideas or product opportunities 

How do I get better at testing?

The key to getting better at testing? Testing more frequently. The only way to test more frequently is if you build testing into your delivery process. If you have it in your project plan, you will do it. Otherwise, it's always a nice to have.

"To many people, conducting user research is a bit like cleaning up the garage: they know it would be a good thing to do and would bring some real benefits, yet somehow they never quite get round to it. "

— Arin Bhowmick
Vice President, Design at IBM

Testing should be in your project plan. It should have an owner and a consistent cadence. That way your entire organization understands what to expect, and expectations should drive action.

Keep a backlog of problems to study. If you run out of problems to solve you are probably not looking hard enough. One way to grow the research backlog is to join in the design and dev process or to meet with customer success or support. Engage your teammates to help understand where there are opportunities for deeper understanding, and which of those are highest priority.

Develop a pool of users. It's much easier to recruit from a defined group who have interest in helping you improve your product. Maybe it's a product council or a Beta test group. Maybe you choose a segment of your user base monthly to engage with. Recruiting sucks. The more you can plan and automate the process the easier testing is.

Engage your team. Testing cannot live on an island. It is an organizational activity. Think of all the activities I've mentioned above.

• Choosing goals and audience
• Creating or reviewing a discussion guide
• Observing and note taking during a test
• Analysis and prioritization results
• Building a backlog

Each of the steps I’ve outlined above is infinitely easier if you invite your team to help you along the way. That's the path to creating great products.