Launched five years ago, Evernote has tried it all when it comes to testing. The company, which provides note-taking software across a variety of mobile and desktop platforms, has conducted expensive focus-group testing and currently relies on thousands of beta testers as well as a third-party testing vendor in order to tweak its applications to maximize user engagement.
 |
|
Evernote continues to tweak its apps, including its Android app. |
"We use testing in an iterative way," said Philip Constantinou, head of products for Evernote, in an interview with FierceDeveloper. "We want to make sure they (the user) can find that new functionality and use it."
Retention is a key metric for Evernote. The company provides a free note-recording service that currently counts around 41 million users, but the company's goal is to entice those users to upgrade to the paid version of Evernote, which offers offline note support and other premium features. By constantly testing and refining its apps, Evernote is able to encourage a greater portion of its user base to upgrade to continue to use the free version of the app and potentially upgrade to the paid version of the company's product--late last year Evernote said it counted 1.5 million paying customers.
So how exactly does Evernote conduct its testing? Constantinou said the company initially relied on focus groups, which he said could cost as much as $40,000 a session. Constantinou explained that, in a focus group, a small crowd of users would run through Evernote's apps and the company would monitor each user's every step in order to locate trouble spots.
"We actually did a fairly extensive study where they (the testers) hooked up devices to their computers at home, and we could watch them use the product and record their usage, but it was fairly involved," Constantinou said. "The subjects in the study got paid a fair amount of money, and we also had to have a coordinator who ran that study."
Constantinou explained that this process was ultimately inefficient because the company could quickly locate trouble spots but it couldn't issue a fix fast enough to test the result. Thus, the $40,000 the company spent to locate one difficult area in the app couldn't be used to also test its solution--or find out whether there was another trouble spot after it.
So Constantinou said Evernote moved on to corralling thousands of beta testers for its various apps, including its iOS and Android apps. He said the company continues to push new designs and features onto these beta testers in order to see what works and what doesn't. On Android, for example, Constantinou said Evernote issues weekly releases to 1,000 beta testers who then provide feedback on the company's tweaks, which then informs the company on how it should proceed with its commercial updates.
But Constantinou said Evernote is currently employing a third testing option that he said has resulted in a significant improvement in the company's testing process. Evernote uses a testing service from UserTesting.com that Constantinou said has resulted in a 15 percent increase across the board in Evernote's user retention statistics. The company conducts five or six tests per product per week via UserTesting.com--a total of around 20-30 per week--and the company can make tweaks based on the results of each of those tests.
"The cost of running these tests is fairly small, so we push it out to five users, we see the response, we tweak it a bit, we push out another version. So it really fits into our rapid development process," Constantinou said, explaining that the company views the effort as "iterative usability testing."
Evernote is one of a number of UserTesting.com customers that dips into UserTesting.com's pool of around 1 million testers. These testers, which fall into a range of demographics and test desktop, tablet and mobile products, typically follow a handful of steps and record their response to each step. The result is a video that companies like Evernote can use to find what works and what doesn't about their app or website.
"This was especially informative on Android because of the variety of Android devices in the market and the various ways people use them," Constantinou said. "Through UserTesting.com, we discover devices and fix related ergonomics issues before we release anything to the public. Also, UserTesting.com lets us try out experimental versions of our applications, which is something we could not do with traditional beta testers."
Related Articles:
Evernote: Google Keep isn't a threat
App testing industry flourishes in chaotic Android landscape
Samsung devices considered best for testing Android apps