Usability Testing Toptal's Onboarding Experience

June, 2022 | Christopher Nguyen

Contents

Intro
Experimental Design
Usability Session
Analysis
Results
Impact

Key points covered in this document

How my team used role playing to usability test Toptal’s entire onboarding experience
How I design scripts and take notes during sessions that capture critical user moments
How I use videos as a visual communication tool to bring insights to life

Overview

Toptal is a premium freelancing platform that connects startups with pre-vetted technology talent. In order for a talent to be accepted into Toptal's network they have to undergo a rigorous screening process, where roughly 97% of talent who apply get rejected during screening. Hiring managers who hire from Toptal are given a "white-glove" service where Toptal hand picks talent specifically for their project. This premium service extends from initial sign-up to hiring the talent.

For this project, my role was to lead the research efforts aimed at improving Toptal's client conversion rates. The lead product manager, provided business guidance, and product design built the prototype used for this study. As a first step, we investigated potential friction points within Toptal's onboarding funnel that may be preventing hiring managers from accepting a talent to interview (See Figure 1 for onboarding funnel).

Figure 1 (Onboarding funnel)

Research Goals

The decision to be made for this study was to determine what client onboarding enhancements are needed to increase client conversion rates?

To address this question, we recruited 9 participants who were hiring managers but had not yet used Toptal. We used external recruiting services such as userinterviews.com to find technology hiring managers who had hired talent in the past 3-6 months. Participants had to be involved in their company's hiring process, hire for developers and work within the technology space.

Designing an experiment that incorporates an entire onboarding experience

Before drafting the experimental design we started with evaluating the current onboarding funnel. We conducted a brief heuristic evaluation to get familiarized with the experience and pinpointed potential usability issues. Reviewing the experience prior allowed us to form the evidence needed to decide whether any enhancements to the onboarding funnel would be beneficial.

Evidence Needed

Understand hiring managers onboarding experience
Identify frictions points in the client onboarding experience
Measure how hiring managers feel about onboarding experience

Determining Research Method: After reviewing the onboarding, we came to the decision that it would be useful to gather both behavioral and attitudinal data as we needed to understand what hiring managers were doing and thinking as they were going through the experience. So we took on a semi-structured research approach with the decision to conduct a moderated usability test (Figure 2).

Figure 2 (Research Method Graph)

Incorporating role playing into our usability test

Once we decided on the type of usability test to run, we had to figure out how to incorporate the sales and matching call into the study. Both calls required a human to get onto a call with the hiring manager lead to discuss their project and hiring requirements. Brainstorming on this issue, my team decided that we could simulate the sales and matching call on our own using roleplaying.

Role playing materials: In order to make the role playing a success we needed to choose which sales and matching call to record. We referenced real job descriptions from hiring managers who were looking to hire a front-end developer. We then looked up their sales and matching call and created our own version of the calls for the study. We did this by having myself act as the hiring manager and my product manager acted as the sales rep. and matcher. During the study, we then replayed the sales and matching call to the participant. We asked the participant questions before, during and after each of the calls. This information was used as the basis for the research scenario, task, prototype, and experimental design.

Scenario and Tasks

Scenario: Imagine that you are John Appleseed, a hiring manager of a company called BlueRock and you need to find a front-end developer for one of your projects. You happen to hear about a company called Toptal, a freelancing platform. You land on Toptal’s home page. How would you go about determining if Toptal can help you find a developer for your project?

Task: Please go through the experience as you naturally would do (as if I was not here) and let me know when you are done.

Prototype: Once the research material was developed, we had our designer build out the prototype based on the scenario and tasks. The interactive prototype consisted of the Toptal home page, sign-up, job details, sales call, matching call, and sent talent.

Capturing key moments in usability sessions

We used the think-out-loud approach to capture hiring managers thoughts in conjunction with observing how they navigated the prototype on their own. Once the participants were done with the task they were asked follow-up questions, first based on their behaviors and then what they voiced verbally.

Script (example)

Home page

Observe how they review the homepage, what information are they looking at?

Process

What is your understanding of the hiring process?

Profile

<if they went into profiles> What did you expect when reviewing the profile?

What information are you looking for?

Adding metrics to look out for in the script comes in handy as a reminder if the participant had performed a certain behavior. For example, after giving the scenario and tasks, did the participant do any of the following on their own without being probed?

Did they do any of the following:

Immediately go into the "hire more" CTA button?
Did they review case studies?
Go into review profiles
Attempted to use the search bar in the home page?

Analyzing data for patterns and trends

The process for synthesizing the observations used session notes and video recordings to recall key moments. Observations were tallied up in a spreadsheet, revealing themes and patterns within the data. Table 1 provides an example of an insight drawn from multiple similar observations. It presents hiring manager behavior to seek evidence for Toptal's credibility and claim of having the top 3% of talent across different domains. All observational data including outliers were taken into account to form insights. See example of data spreadsheet in Table 1.

Example data

	Participants	P1	P2	P3	P4	P5	P6	P7	P8	P9	Count
Homepage	Measures/ Questions
	Reviewed case studies on their own	Yes	Yes	No	No	No	Yes	Yes	Yes	Yes	6/9
	Mention they want to know if Toptal has worked with similar clients as themselves (industry, talent, project)	Yes	Yes	-	Yes	-	Yes	-	Yes	-	5/9
Sign-up	Reviewed talent profiles on their own	Yes	Yes	No	Yes	No	Yes	No	Yes	Yes	6/9
	Expect to know talent availability based on their project details	Yes	Yes	-	-	Yes	-	-	-	-	3/9

Table 1 (Data sheet)

Results

The following insights are also shown in Figure 3.

Trust and transparency: Toptal claims to offer the top 3% of talent but hiring managers lack visibility into how talent are vetted and selected for their project, raising trust concerns.
Discoverability and efficiency: Hiring managers are overwhelmed with documentation to complete and review, causing important details, such as signing the legal STA to become less noticeable. Hiring managers also come from larger teams and lack the authorization to review legal documentation on behalf of their organization.
Process comprehension: New hiring managers have to go through multiple calls with Toptal representatives, often having to repeat many details. Additionally, they are unclear about the differing roles and responsibility of the sales and matcher.
Personalization: Hiring managers come in all sorts of shapes and sizes. Some have a technical background and know how to hire tech talent, while others may need to hire multiple candidates at once or require expert assistance in the hiring process.

Figure 3 (Insights)

Using videos to visually communicate insights

In order to effectively communicate insights, we took inspiration from storyboarding and video creation by outlining the research story and from there deciding which insights would benefit from a short video. In Figure 3, observers can first handedly experience how the insight of sales and matching call process was unclear to users. In the video, research pieced together various 1-2 minutes videos of hiring managers struggling to understand how the Toptal representative (sales) and Toptal matcher were different from each other. Other videos worth showing were users missing the mark on signing the legal STA and seeking top 3% of talent validation. Each video consisted roughly 3-4 clips and were used as complimentary visual aids to the key insights. This presentation was shared out to a wider group of teams so the videos were helpful for context building and engagement.

Impact

There were a variety of opportunities uncovered during this study presented in table 2.

Opportunities	Impact
Explore ways to inspire hiring managers to trust our rigorous vetting and hiring process	Improved trust in process (Talent availability, talent screening details)
Help hiring managers prepare for each step of the hiring process (sales, matching, and project scoping)	Easier demonstration of process (Roles and responsibilities and progress tracker)
Investigate ways to make Toptal’s onboarding flow more inclusive rather than exclusive (personalized hiring experience)	More personalization (Hiring flexibility, Multi-hire and Ask an expert services)
Streamline STA signing	Improved discoverability and efficiency (Added legal team in onboarding to sign STA)
Educate on billing and deposit	Billing and deposit education

Table 2 (Opportunities and impact)

Final Thoughts

This study was the first to evaluate Toptal's onboarding experience using qualitative methods. Designing the experiment had to take on a creative approach as a result of incorporating the sales and matching call. Results from this study not only answered many of Toptal's most pressing questions regarding hiring managers' curiosity of Toptal's vetting process and hiring steps but also inspired other product teams outside of client research to further explore new product opportunities. Many of the material used in this research stands as a working template for other Toptal teams to reference for their own studies.

Team

Research (myself) was responsible for project scoping, recruiting, experimental design, moderating sessions, synthesizing insights, reporting and sharing research. I also authored this document
Product was responsible for project scoping and assisting in recording the sales and matching call
Product design was responsible for building the prototype.

Resources used for study: Airtable, userinterviews.com, Google sheets, slides, and documents, Quicktime video, iMovie, Gong, and JIRA