Fixing the live coding exercise

It's become a meme in software engineering: "Interviewing is broken." But what's broken about it? In her "Interview Theater" post, Terra Field identifies a couple fundamental anti-patterns that will make for a broken interview:

It's stressful and adversarial, and therefore excludes folks who don't perform well under pressure.
It challenges the candidate in ways that don't accurately assess their abilities.

These anti-patterns can manifest in a bunch of ways, all of which are covered in Terra's post. I recommend you read the whole thing. It's a great, insightful read. I want to zoom in on one specific practice that she calls out: the live coding exercise.

A live coding exercise often has little to do with what the candidate’s output is like on the job. Instead, they are much more of a measure of the candidate’s anxiety and comfort level in the interview room than their technical ability, and many of us already have a therapist to tell us how anxious we are.

I agree with so much of this! And yet, as a hiring manager, I use an interview structure that involves a live coding exercise. And I think it works pretty well. How can this be?

Apples and oranges

Honestly, it might just be that the domain I'm in, UI engineering, is more amenable to this type of exercise than Terra's domain, which is systems engineering. I don't know! With that caveat established, read on if you're curious about how I optimize my team's live coding exercise to minimize candidate stress and maximize interviewer signal.

Our live coding exercise

We use the live coding exercise as a screen. It's the first interview in which the candidate meets members of our team, and so the goal is to identify whether they possess the bare minimum of requirements to progress to the next round.

Here's the process in five stages.

Design the exercise

A good coding exercise will approximate the work that the team is actually doing. In my team's case, this involves UI engineering with React. The exercise should also:

Have simple, straightforward requirements. For example, ask the candidate to build a very simple UI with a series of increasingly challenging requirements that build upon one another. Each requirement should touch on what your team would consider to be a fundamental aspect of UI engineering, such as form state management, code composability, or string formatting.
Require minimal setup. Don't make the candidate waste time getting things running – help them get to work by giving them a skeleton app with some stubbed functions. Don't think "blank canvas" – think "coloring book".
Require minimum investment. The exercise we use takes 40 minutes. We leave 10 minutes at the beginning for intros and 10 minutes at the end for the candidate to ask us questions. The whole interview takes one hour.

As you iterate on the exercise, go through it yourself. Eventually you'll become too desensitized to identify its rough edges, so ask a coworker to be your guinea pig. As you iterate, eliminate points of confusion, redundant requirements, and any work that won't give you insight into the candidate's abilities. Don't forget to time yourself! When you're done, write down how the exercise will likely progress, step-by-step, using real code examples. This will help you review the exercise and iterate on it in the future, especially if you find candidates tend to approach the exercise differently from what you expect.

Prepare the interviewers

Write a guide for the interviewers. Enable the interviewers to provide psychological safety for the candidate. This is critical if you hope to get any useful information out of the interview. More on this below. You should also ask the interviewers to look for specific criteria to assess the candidate's abilities. Some things I ask interviewers to look for:

Do they know the tools of the trade, in this case JavaScript, HTML, browser dev tools, and Google?
Do they reason well? Verify assumptions?
Do they take the time they need to think about the problem?
Do they explain their thoughts in a way you can understand?
When things don't work, do they react constructively and find a way to move forward?
Do they ask questions when they need help? Do they look things up when they need to?
Do they respond positively if you constructively point out a mistake or ask a question?

Put them in the candidate's shoes. To develop empathy with the candidate, I ask interviewers to take the exercise themselves. For example if they have to look up some syntax at a particular point they can share that experience with a candidate who does the same thing. This kind of genuine, informal conversation puts candidates at ease and helps interviewers maximize the value of the interview.

Prepare the candidate

Uncertainty creates anxiety, so I try to set expectations for the candidate as much as possible. Here's the prompt I ask recruiters to send to candidates in advance of the interview. I've bolded the important bits.

You'll go through a coding exercise with the interviewers. You'll work with one of the interviewers as a pair of engineers working together to solve the exercise. The other interviewer will drive the interview. This is not meant to be a test of your ability to memorize syntax, so we encourage you to approach the exercise as you would approach your everyday work (e.g. by asking questions or looking things up online).

This test involves React, but if you're not familiar with React that's OK. Let the interviewers know and they'll help you by answering any question you have about React syntax. If you'd like to brush up on React ahead of the interview, we recommend Thinking in React and React Hello World.

Please learn about or refresh yourself on the useState hook because we will be working with some state-related logic.

During the interview we’ll send you a zip file containing the exercise. You'll need to use npm or yarn to install its dependencies and run the code locally. You’ll screen share over Zoom as we work through the exercise together.

What's important to me is that the candidate comes into the interview expecting to treat the interviewers as coworkers, not adversaries. I just want to see what it'd be like to work together. I also want to give the candidate time to warm up on the technical material in advance, so we can spend our time on working the problem instead of wrestling with tools.

Conduct the interview

Terra's post explores the disruptive effects stress has on a candidate. And I agree with her. The less stressful an interview is, the more signal we'll get on the candidate's abilities. Which is why I think interviewers must learn how to create psychological safety for candidates in order to interview well.

My go-to for creating psychological safety is: "Don't let the candidate fail." If a candidate is spinning their wheels, they'll just get frustrated and we won't learn anything about their abilities. So just like a good coworker helps you get unstuck and move forward, we help interviewees also get unstuck so we can continue with the exercise and learn more about the candidate. And by ensuring the candidate completes the exercise, we can make sure they leave the interview with good vibes and a positive impression of us.

A coworker pointed out to me that this should feel like a conversation. I love that. Here are some psychological safety tricks we use, infused with this "interview-as-conversation" perspective:

If a candidate falls silent and seems stuck, you can prompt them to think aloud by asking, "What specifically are you trying to solve for right now?" or "What are you thinking about right now?" If you feel like you're interrogating them you can respond to what they say, for example by paraphrasing their answer to make sure you understand.
When the candidate is trying to figure out the path forward and asks for your opinion, get them thinking aloud by asking them back, "Hmm, well what do you think?" or "What options are you considering?" Offer your opinions on their ideas. Or just be encouraging: "That sounds good to me. Let's try it."
Maybe the candidate has already thoroughly defined the options they're considering. You can give a little more structure by saying, "I'm concerned about [insert requirement here]. How would your solution take this into account?" or "How do the pros and cons of these options compare?"
If the candidate is totally stuck, suggest the solution. "Hey, what about...?" or "Have you considered trying...?"

Encourage the candidate, be genuine, and most of all – relax! If you're relaxed, the candidate will be relaxed. Be friendly, positive, respectful, and informal. Treat it as a conversation. The point is to communicate and work together.

Evaluate and debrief

When evaluating the candidate, give more weight to your positive impressions than the negative ones. Did they get stuck in a couple places, but respond well to your questions? Were they able to demonstrate any problem-solving ability at all? Give the candidate the benefit of the doubt as long as there weren't any glaring red flags. Evaluating for the absence of weakness is generally unhelpful, particularly this early in the hiring pipeline.

Have interviewers submit scorecards before the debrief, to avoid biasing one another. Then get together to compare notes on the candidate and on the interview experience itself. How did you do as interviewers? What can you add to your enablement materials to better prepare for next time? How can you improve the quality of the interview structure or the exercise?

On my team, we have interviewers of all experience levels participate in these interviews, so a debrief is particularly helpful for leveling up the interviewing capabilities of our less-experienced interviewers.

Wait, but why?

If you can eliminate anxiety and accurately simulate a work situation, you can learn a lot about what the candidate is capable of... but only on a fundamental level. Remember, this is a screen. The goal is to screen out candidates who might have BS'ed their resume or lack fundamental problem-solving abilities. And yes, this is a low bar.

All we're doing is establishing a baseline of capability. To truly assess candidates you're going to have to learn about their unique strengths and accomplishments in subsequent interview rounds. This is something Terra touches on, too:

...when you’re asking questions to evaluate someone’s practical experience in an area, some of the best questions prompt that person to talk about whatever they want.

And later:

Have a conversation, engineer (or manager) to engineer with me where I’ll tell you what I know and what I can do. We can shoot the shit about Hadoop or Kafka or Kernels or crontabs or wherever the conversation takes us — and you will learn way more about me than you would quizzing me on something that I could find the answer to on cyberciti.biz.

I found myself nodding as I read these parts. In later rounds, we ask candidates to share their experiences with us and go deep on the aspects that they found most interesting and challenging. These are the fun parts of the interview process that I really enjoy, and they give us the clearest picture of how the candidate might add to the team. In fact I broadly agree with Terra's point of view in her blog post, at least on a philosophical level. Maybe we just differ on the implementation details.