Coding Challenges for Screening and Skills Assessment

Design and best practices for two types of recruitment test

Matt Fisher
August 2019

---

Coding challenges for tech recruitment can be a contentious issue. Done badly, candidates will have poor experiences and you will lose talented people. Hiring engineers and managers can also sometimes struggle to understand how to get enough value from coding challenges.

Often, this is because the purpose of that challenge isn't well thought out. A code-based assignment should be designed with a particular purpose in mind. It should also be designed differently depending on where in the recruitment pipeline it sits.

In this post, we explore two defined types of recruiting challenges: simple screening tests and skills assessment tests. We'll cover where they sit in a recruitment pipeline, and what you can expect to get out of them.

Note: I'm writing this from the perspective of an offline take-home test, but very similar considerations apply to online or pairing tests - adjust accordingly. Always bear in mind the overall objective: your recruitment process should be aimed at extracting the best signal-to-noise ratio of your candidate's capabilities whilst being respectful of everyone's time and commitments. A take-home coding challenge is no exception to that; don't alienate your candidates by giving them a poor experience.

Simple Screening

A screening test is an early stage filter to prevent unsuitable candidates from progressing to later stage interviews. Whether you should have one depends on your industry and your pool of applicants. I've had many candidates reach a mid-stage interview that fail hard the first time they are asked to write code. I'm not talking about difficult algorithms either; I'm talking about simple functions to add together the numbers in an array^[1]. Sometimes, an applicant just doesn't have the skills you're looking for. It's not good for anyone to keep those candidates in the process for longer than necessary - it consumes your company's resources and wastes their time. Fail faster is good advice in this scenario.

If you often find yourself rejecting applicants for lacking basic skills, you should find ways to filter them out earlier. It's hard to overstate the impact this can have. Recruitment is a difficult process that crosses multiple company departments and poses unique scheduling difficulties; depending on how far you progress an unsuitable candidate, it can cost you hundreds or thousands of dollars.

A quick coding challenge as a screening test provides a remarkably good filter. The outcome here is an easily definable pass/fail. You're not attempting to judge overall skill level - just ascertain that a minimum level of skill is present. You want to pitch it so that it's almost trivially easy for your skilled applicants. It shouldn't put them off or yield false negatives. As a general rule of thumb, make the challenge too easy and increase the difficulty later, rather than the other way around. It's very easy to fall prey to the curse of knowledge here and unknowingly make a test much harder than it should be.

Tips and Guidelines for Screening Tests

Make it perfectly clear what you are looking for, and what you are not.

At this stage, you should care about:

• Correctness
• Completion
• Simple and straightforward solutions

You should probably should not care about:

• Performance, Big-O complexity
• Adherence to linting rules
• UI polish
• Comprehensive error handling
• Test coverage

Write what you are looking for and what you are not looking for into the instructions of the test. There is nothing to be gained from an applicant having to guess what skills you're looking for them to demonstrate.

Make it short. There are no hard and fast rules, but an expectation that it should take 30 minutes is reasonable. Longer is fine if necessary, but make sure you're doing it for well understood reasons. Consider time limiting the test. If it's a short challenge, it shouldn't be hard to do in a contiguous block of time. Set a limit that allows skilled candidates plenty of time to complete all of the tests comfortably. The limit isn't to add pressure to those who do make it through - it's to make sure that someone isn't spending 20 hours of trial and error to find the correct answers.

Create several smaller tasks taking 5-10 minutes each, rather than one larger task. You'll be able to see a better overview of a candidate's capabilities, and the impact of a single failure on the results is reduced. Aim for very simple FizzBuzz style tasks that you think a competent applicant could easily do. Functions to manipulate built-in data structures are a good source of inspiration. E.g. "write a function to count the number of times the string element 'Horse' appears in an array" or "create a map that has the integers 1-10 as keys, and the square of its keys as values".

Provide a scaffolding project to get them started; you're not timing how long it takes a candidate to set up a new project. Have source files set up and ready to run; mark clearly function bodies that need implementations, or classes that need defining. If you can provide a test suite to allow coders to check they've reached a successful solution, even better. Provide simple and explicit instructions for running the project and/or tests.

Evaluation of Screening Tests

The result you're looking for here should be no more complicated than pass or fail. Write down evaluation criteria so your recruitment pipeline scales well. You might see a lot of submissions for this phase, so making it easy for your recruitment team to dip in and out and evaluate results quickly is important.

If the solutions produce the correct results and are accomplished in a sane manner, you should be looking at a pass.

Skill Assessment Assignments

Further down the pipeline you might want to issue a more in-depth coding challenge to assess the level of an applicant's engineering skills. There are a few reasons this can be a good idea. It can replace awkward on-site coding sessions, and provide a more comfortable environment for the candidate to work in, available at times that are more likely to suit them.

You should be using a this kind of coding challenge to assess a particular set of skills that you have determined are important for the role. Aim to create a challenge that results in a strong hiring signal. You shouldn't be creating a challenge at this stage if your review process is to just glance at it and think "yes, that looks OK. Bring them on-site". Make sure it results in a rounded understanding of a candidate's skillset that gives actionable information for a hiring decision.

Because a skills based assessment challenge is much more in-depth than a screening test, the guidelines for designing a good one are necessarily more fluid.

Skill Assessment Test Guidelines

Most importantly, scope it clearly. There is so much potential for a candidate to spend time on the wrong things. It's perfectly acceptable to consider things like test coverage, linting, well engineered and maintainable code or performance here as long as you are explicit about what you're looking for.

It's tempting to use the phrase "code like you're writing for production" in challenges like this, assuming that is a comprehensive enough guideline. It is not. Even if you have perfectly reasonable expectations, there's a good chance your candidate will have encountered other challenges with unreasonable expectations ("where's the analytics? How are you supposed to understand your users' behavior without it?" or "you haven't provided a metrics endpoint for monitoring - how could I actually deploy this?").

Be very clear about how you're assessing the results; don't make your candidate play guessing games with their limited time. Try to evaluate mostly on the quality and correctness of the code itself, and leave actual production considerations (monitoring, service-to-service architecture, analytics) for a wider in-person discussion.

It's important to set an expectation of how long a candidate should spend on the assignment here too. The actual time required will depend on the scope of the test. What is justifiable depends on how you use the test in the rest of your pipeline. For a candidate you've screened well and has a reasonable chance of being hired, a two to four hour test should be fine. Longer can also be OK, as long as it is done for the right reasons. For example, a test that takes a day's worth of work is asking a lot of your candidate - but if that removes the need for an extra day's worth of on-site interviewing, it is worth considering as an option. As a general rule of thumb, make it as short as you can whilst still gathering the information you need to make a hiring decision.

Always ask some of your existing colleagues to try the test under the same conditions you expect candidates to take it - you'll often find that it will take longer to complete than you expect. Adjust the scope accordingly.

This style of challenge should also be time-limited. Candidates will have different amounts of available time. Allowing a wide variance in scope and time allows some candidates to turn in more impressive results simply because they've spent more time on it. This can bias your process towards young, unattached graduates and away from e.g. senior engineers with young families. You don't want to miss out on some excellent hires this way, so set clear expectations about how much total time should be spent. You'll also want to allow a reasonable period (e.g. a week or two depending on size) for the completed challenge to be turned in so that candidates can work around existing commitments flexibly.

Create an existing example project for your candidate to work from. This lets them understand some of your conventions, styles and patterns, and gives them a strong starting position. Make it representative of your typical project style, but ensure it's fairly lightweight and simple to run. Always provide instructions for running the tests, compilation, etc.

Request that candidates complete several tasks to show you a range of their capabilities. It's useful if these start simple, and increase in the amount of complexity and autonomy the applicant is required to demonstrate. There are many different ways of doing this, but one structure that works well is:

• Fix the cause of a failing test - demonstrates basic fault finding and bug fixing
• Implement new functionality alongside existing similar functionality - e.g. an additional stage in a data processing pipeline or an alternative data storage implementation. This demonstrates the ability to extend existing code in-line effectively with current conventions
• Add a new feature not currently supported by the example project - this demonstrates the ability to refactor existing code cleanly, how to manage information between different areas of the system and to think creatively

Make sure you're using this opportunity to get to know an engineer's skills productively - you should avoid covering the same ground again during in-person interviews later.

Evaluating Skill Assessment Assignments

Make sure your evaluation is in line with the scope of the assignment's instructions; don't expect things you haven't asked for.

Several opinions are usually better than one - get a couple of experienced engineer's eyes on the results. Two engineers will often make more interesting observations in discussion and review than one.

Don't try to over-formalize the review. Marking schemes and a purely objective set of criteria sound nice in theory, but in practice it bogs down reviewers and provides little value. Instead, trust in the judgement of the skilled people you've already hired as to whether a candidate should proceed to the next stage.

Finally, return to the results a candidate has submitted during a later in-person interview stage. It's a great jumping off point for discussing design decisions and more complex considerations further.

---

[1] There's actually a couple of possible reasons for this. One, they're just not particularly good coders or problem solvers. The other is that some people just find interviews very stressful, and can freeze up and panic when asked to complete the simplest of exercises. I've got a lot of compassion for people in that position - it's one of the strongest arguments for take-home tests in the first place.