The first experiment
Because experimentation education was needed, Michele and her counterparts at Conversion decided to push for “simpler” tests at the outset. This thought process reflects Conversion’s prioritization framework: PIE.
PIE is a framework that enables organizations to prioritize where (and what) to test first based on three factors: Potential, Importance, and Ease. Potential is how much improvement can actually be made in a given area. Importance is how valuable a page or funnel is (e.g. how much paid traffic does it receive; how visible is it, etc.). And finally, ease is how easy it is to test in a certain area, both technically and politically.
Given the level of understanding around “CRO” at Envoy, political Ease played a role when Michele and her Conversion team were determining where to run Envoy’s first experiment. The team identified the site-wide navigation as an important, high-impact area for testing, where we could run a simple experiment that required minimal engineering work and easily fit within Design standards.
“We came in knowing that the site had just undergone a full redesign so we knew that certain areas would be politically sensitive. From a data perspective, the site-wide navigation made perfect sense to kick things off. It receives the highest amount of traffic on the site and has high potential for driving results. We wanted to start with the nav in order to prove the worth of experimentation, while encountering the least amount of pushback from the different teams involved.””
— Neil Lim, Experimentation Strategist at Conversion
The experiment details—site-wide navigation
As soon as Envoy signed on with Conversion, the team began the Explore phase. Explore is the first phase in our Infinity Optimization Process, wherein we gather as much information as possible about an organization and its customers. Explore insights come from customer research, analytics, stakeholder interviews, persuasion principles, business objectives, the experiment archive, and more.
Explore is a continuous process. The insights gathered inform every experiment we run, and insights that are uncovered when an experiment concludes are fed back into the Explore phase—hence the infinity loop.
Conversion’s Infinity Optimization Process.
During Explore with Envoy, Strategist Neil Lim collected user research data and digital analytics data. The team also conducted several LIFT analyses, all of which revealed several potential friction points in the site-wide navigation.
- CLARITY: There is no easy way to see the features for both Envoy’s Visitor and Delivery products or what distinguishes them from each other.
- CLARITY: It isn’t clear that Envoy Visitors and Envoy Deliveries are two exclusive products.
- CLARITY: Do the value proposition points in the “Why Envoy” dropdown apply to both products or only one?
- ANXIETY/DISTRACTION: The value proposition points in the dropdown are easily missed if no one hovers over the “Why Envoy” menu.
After much discussion around these points, the team proposed several hypotheses. Ultimately, we decided to focus on a small change in the first variation: Renaming the “Why Envoy” dropdown to “Solutions”. According to Envoy’s user research, the term “solutions”—specifically “visitor management solutions”—was important to Envoy’s visitors. It was a term they were using.
With this variation, we wanted to make it clear that the value proposition points hidden in the “Why Envoy” drop down were actually solutions to visitors’ queries.
Hypothesis: Changing the “Why Envoy” label to “Solutions” will communicate that the key value proposition points found within this dropdown are the benefits users can derive from using Envoy’s products, increasing their motivation to sign up for a trial, resulting in more sign ups.
Variation A.
With variation B, we wanted to explore a slightly different angle. Signing up for an Envoy account is free and easy, a potential value proposition point that was only mentioned towards the bottom of the page—we wanted to see if emphasizing this value point in the navigation would increase sign ups.
This experiment idea was also bolstered by a past experiment we’d run with a SaaS client, which resulted in an almost 9% uplift in signups:
We had previously tested this idea with another SaaS client, and had seen a strong positive result.
As a clarity point, we also wanted to test spacing out the links in the navigation. The overall hypothesis for variation B was:
Leveraging key value proposition points around the call-to-action and reducing the number of actions users need to take to complete signing up for an account will reinforce the fact that Envoy is risk-free, resulting in more visitors signing up.
Variation B.
In this variation, we shifted the primary navigation links to the left and added an email capture field into the nav.
Ultimately, variation A resulted in +16.53% increase in signups at 99% statistical confidence—a huge first win for Envoy. Variation B had a flat result, which goes to show that best practices mean nothing if they aren’t validated within your specific context.
Interesting insights:
In variation A, the team observed greater engagement with all key navigation links and reduced engagement with the “Sign Up” call-to-action. Visitors were exploring other pages on the site to a much higher degree, particularly the Visitor Management product page—which is Envoy’s most valuable product. While fewer visitors clicked on the CTA within the nav, sign ups increased drastically.
This indicates that Envoy visitors may need more education around the platform itself before they are willing to sign up for an account. Based on these results, future experiments should focus on either educating visitors about the benefits of the platform or making it easier for visitors to access this type of content.
Socializing the first experiment win
When Conversion Strategist, Neil Lim, presented this result to the Envoy team, they were pumped. So pumped, in fact, that Alex hardcoded the winning nav experience right there on the call.
Then Michele ran (literally) to tell Russ. She explained the experiment and the result verbally—16.5% uplift!—and promised to follow-up with a more comprehensive results analysis, provided by her counterparts at Conversion.
As promised, Michele packaged the results analysis to share in writing with Russ. Russ then took those results and socialized them to Envoy’s CEO as well as the rest of the Executive team. It was important to Michele that Russ socialize this first experiment to really highlight the team’s success, reinforcing the excitement and enthusiasm around the experimentation program in general.
This was the perfect first win. We were able to show how even small optimizations could lead to significant returns. It set us up to start feeling more confident and excited about the bigger opportunities ahead of us.
— Michele Pendergast
The next experiment
Off the heels of this first win, Michele could feel the momentum. She was starting to get that much sought after organizational buy-in. For the next experiment, the team decided to tackle the homepage proper. But given the visibility and importance of this page, Michele wanted to make sure to do things the right way—the collaborative way.
So, she brought Envoy’s Lead Designer, Amy Devereux, into the conversation early to ensure that the experiment variations would 1) make sense visually, and 2) align with her vision for the website.
The experiment details—the homepage
In conversations with the Envoy team about the homepage, one question that surfaced was around credibility: Namely, would highlighting Envoy’s customer logos impact sign-ups? These logos were potentially a major value proposition point—Envoy’s customers include large credible companies. However, they weren’t getting a lot of exposure in their original position on the homepage.
Conversion strategists also considered evidence from psychological principles that would support moving the logos higher on the page. Namely the Bandwagon Effect, which states that we tend to put our trust in something because we believe many other people have put their trust in it first. There is safety in numbers.
So, the following hypothesis was put forward: Increasing the prominence of brands that use Envoy will alleviate any anxiety associated with signing up for an account, while also enhancing the credibility of Envoy which will ultimately result in increasing trial sign-ups.
Control vs. Variation A
Alternatively, with variation B, the team wanted to test emphasizing Envoy use cases. This idea came out of the initial LIFT analysis. From a value proposition perspective, the information contained in the use cases section is useful and beneficial, but may have been missed because it is in between two large content blocks.
There was also click data from FullStory which indicated the use cases section was not receiving any clicks, re-validating the lack of exposure of this section. The Conversion team also wanted to use this variation to funnel Envoy’s visitors to a single product: Visitor Management. This reflected an overall business objective.
For variation B, the hypothesis was:
Communicating the benefits of using Envoy and directing visitors to a single primary product will encourage visitors to learn more about the platform, ultimately resulting in increasing sign ups.
Control vs. Variation B
Both variations were winners, with variation A resulting in a 12% lift in sign ups and variation B resulting in an 8% lift in sign ups.
Interesting insight:
Given that most visitors may be unfamiliar with modern workplace technologies like Envoy, exposing them to prominent customers may have resulted in:
- Spurring their curiosity about the products and driving them learn more on key Envoy pages
- Helping to alleviate anxiety involved in signing up for a trial after seeing prominent companies using the platform (Bandwagon Effect)
The future of experimentation at Envoy
With two strong experiment results under her belt, Michele sees blue skies ahead for experimentation at Envoy. From her point of view, there are two main priorities to keep momentum going:
- Continue to socialize experimentation at the Executive level, and
- Level-up Envoy’s approach to rigorous experimentation
Sharing these initial experiment results has already led to increased organizational buy-in, and opened new doors for the experimentation program. The team is hoping to expand to program to test on the sign up funnel as well as other areas and products for Envoy.
“I’ll probably continue to share every experiment result with my VP as soon as I get it—I get really excited and want him to be excited, too,” laughs Michele. “But moving forward, I will want to be more strategic about how I bundle and share results within the organization at-large.”
Michele is building a communications plan that will allow her to spread the word as effectively as possible. She has started providing in-depth updates (hypotheses, experiment designs + rationale, results) in weekly Marketing Team meetings and rolling up high-level updates (experiment hypotheses + results) post-experiment into Envoy’s weekly leadership meetings.
As for the overall experimentation strategy, the team wants to focus on getting into more strategic and higher impact experiments. Where the first few experiments were focused at addressing high-impact, low-hanging fruit, subsequent experiments will focus on improving customer lifetime value and trial conversion rates for Envoy.
These are the metrics that will really drive sustainable growth and revenue for the company.
Everyone at Envoy is here to help elevate the workplace experience. We have a shared vision for the future. No matter your department, we’re all rowing in the same direction. WiderFunnel [Conversion] helped us kick off an inclusive growth experimentation program so we could play to all of our strengths, getting us a little closer to that future, one test at a time.
— Michele Pendergast
The Envoy team has a shared vision for the future — and experimentation is a big part of it.