Async test projects in practice

Rasťo, our Head of Android, shared in a prior post how we came to leveraging an async-first test project format for our engineering roles in place of more traditional whiteboard coding.

The async test project helps us better understand candidate behavior in their day-to-day workloads, while leaving room for creativity and providing comparable input into the hiring process. It enriches other data points we gather through screening and structured interviews.

In today’s post, we’ll go deeper into how we put the approach Rasťo outlined into practice in one of the Doist teams, the Integrations team.

The Shape

When candidates enter the test project stage, they receive an invitation to a private GitHub repository that only they and the interviewers can access.

The repository contains a small codebase as a jumping-off point for individual tasks. Giving candidates a brownfield project helps us better simulate real-world conditions and provides better comparability between candidates.

In it, they find the following items:

1. README – introduces the candidate to the repository, how to set it up locally, and the individual tasks to complete.
2. Main branch – contains a code sample forming the basis for all the tasks.
3. Feature branch pull request – contains a work-in-progress feature implementation for the candidate to review and provide feedback.
4. Feature request issue – A feature request for the candidate to implement as a pull request.
5. Bug report issue – A bug for the candidate to triage and resolve as a pull request.

Once we invite candidates to their repository, we ask them to decide a deadline for delivering the test project. As an async company, we understand that our candidates have a life outside our interview project. Communicating about realistic timelines is more important than being able to work under time pressure.

Async Preparedness

One of the key benefits of an async test project that happens wholly within GitHub is the ability to communicate with candidates as we would with our colleagues when discussing our work. Clear and concise communication is critical for async work; understanding the candidate’s communication style is an explicit goal for our hiring process.

We encourage candidates to ask clarifying questions directly on the issues instead of debating through email. Likewise, we might comment on their feature with pull request comments or rebuke the feedback within their review.

One technique we discovered (accidentally) that helps us understand a candidate’s problem resolution skills is to introduce a confusing element into one of the tasks. We did this by mistake when we gave candidates to fix a bug that didn’t exist. Still, we learned so much from how candidates responded that it is, occasionally, helpful to introduce flaws into the tasks on purpose. 🤭

Automation

To rely on the approach above at scale, we automated the creation of the test project:

• Generate a new repository in a specified organization named after the candidate.
• Push sample code to the repository.
• Push feature branches to the repository.
• Generate Pull requests and Issues based on markdown templates.
• Automatically invite the candidate to the repository.

You can see my automation tool built with Node + TypeScript at interviewing-repo-scaffolding . After adding the requisite test tasks via Markdown annotated with Front Matter , you only need to add the sample code and create the feature branch.

GitHub has an excellent API , making most of the above steps a breeze. Generating repos, creating issues and PRs, and finally inviting the candidate is a call away.

For manipulating git to prepare local branches and pushing them to remote, we leverage dugite , lightweight Git bindings for Node.

Once you’ve configured the tool locally, creating new test repos is as easy as inputting the candidate’s GitHub username. 💪

Despite automating most of the process, there is more we’d like to improve. Most significantly, an integration with Workable or Pinpoint would enable us to automate the provisioning of test projects entirely when a candidate enters the test project stage, as well as cleaning up after the candidate concludes the hiring process.

Learning Together

The mix of constraints, interactivity, and freedom that the test project format gives us has benefits beyond providing better assessment output.

As we model the test projects after real problems and codebases, it’s common to learn about new approaches when reviewing candidates’ test projects.

Candidates experienced with our tech stack sometimes show us neat tricks we can imitate. Novice candidates also occasionally point us to new tech stack features we’re unaware of that can make us more efficient.

Conversely, we aim to provide constructive feedback on candidates’ work via discussions within pull requests. We aspire to create mutual value even for candidates that don’t end up progressing in our hiring process. It doesn’t always happen, but when it does, the process becomes more engaging for both sides.

We hope that, if you haven’t already, you will consider giving the test project format a shot and find it as helpful as it has been for us! 🚀