After an initial screening, based on CV and perhaps a phone interview, I think it’s essential when hiring a developer to include some kind of realistic test of problem solving and programming abilities as part of the recruitment process. This is the point at which candidates are being seriously considered, and as a matter of courtesy, I think it’s important not to waste a candidate’s time; I think it’s more considerate to filter meaningfully at the CV stage, and then again at phone interview, rather than ask every candidate to spend a considerable amount of time on a coding exercise. As a candidate myself, I actually quite enjoy well-constructed coding tests, because they act as a kata; they are small self-contained exercises with a defined end-point. I sometimes re-use coding tests for things like training or mentoring exercises, because these properties can make them a useful teaching tool.
A well-designed technical test should be interesting and challenging enough to meaningfully test a candidate’s ability, but small enough not to take up more time than is needed. It’s nice if the test bears some resemblance to the kinds of problems that are actually being solved in the real codebase, but I always try to design tests that don’t involve infrastructure, sticking instead to pure logic. That said, it’s amazing how often I get a solution put forward that contains an awful lot of infrastructure! I think it’s a good thing for the requirements to be “realistic”, perhaps with some ambiguity or not totally complete. In the real world, requirements are very rarely perfectly defined, and it’s an essential skill as a developer to be able to identify these ambiguities and make reasonable assumptions or work with the team to iron them out. I like to see a readme or similar that describes any assumptions that were made, with a quick explanation, because for me this shows initiative and understanding.
It’s kind of obvious, but the fundamental thing that a solution to a technical test needs to do is solve the test. It’s amazing how often this isn’t the case! I sometimes see solutions that go off on wild tangents, only tenuously linked to the actual question, and these are clear fails. As developers, our job first and foremost is to solve the actual problem, and we have a responsibility to at least try to understand what that is before starting. The other thing that is of top importance to me is simplicity and elegance. It can sometimes be tempting for a candidate to feel like a technical test is an opportunity for them to show off, and create something complex. The most common reason I see real projects failing is because developers do exactly this in the real world, so if I see any sign of this trait while recruiting it’s an instant no. Some examples (by no means exhaustive) are candidates wiring up a full-blown database or web stack when the test asks for an algorithm, candidates using hand-rolled build tooling or using something that needs 3rd party plugins, and pattern abuse. Dependency injection deserves a special mention; it’s widely misunderstood and technical tests often include great examples of how not to apply this technique. If you are asked to write a FizzBuzz console application, and feel the need for an IoC container such as StructureMap, you’re doing something very wrong!
Beyond that, the first thing I look for in a technical test is evidence that a sensible quality control approach – i.e. Test Driven Development in most cases, as we’re dealing with simple algorithm problems – has been followed. TDD has become a buzzword, and it’s very common to see misapplications of the approach – things like tests that contain the same logic as the production code, tests that test trivialities or frameworks, misuse of and over-reliance on mocking tools and so on. If possible, I ask for git repositories as the means of submitting a solution – with tools like GitHub being so commonplace there is little excuse for not being able to do this; the advantage of this is I can see more of how the candidate likes to work. Things like whether they commit little and often, are the commits atomic and reasonable, did they actually test first or just write tests afterwards can all be seen from the git history.
The next criteria is style, which can be a controversial one. Everyone has their own style, and that’s fine, but I’m looking for an attention to readability and a reasonable approach. Each language has its own idioms, and I expect these to be respected; there are de-facto styles and well-known style guides for many languages, and it will certainly raise questions if there is significant deviation from these. Sometimes it’s clear that the solution is not in a developer’s main language, because they have tried to carry across that language’s style. Style is important, because it makes it easier or harder for groups of people to work on the same codebase. Good developers are able to adapt and follow local conventions – but asking good developers to abandon convention for the sake of someone’s whimsy will wind them up!
A good solution to any problem will show appropriate levels of complexity and abstraction. I’m looking for reasonable application of design principles, such as SOLID, or sparing use of appropriate design patterns. A good technical test will not have a single valid solution, but allow candidates to reach a solution that shows their own approach, so it’s nice to see some variety in how people solve a problem as an indication that the problem isn’t too trivial. The range of solutions is often very wide though, ranging from single-component solutions that show no regard for separation of concerns or single responsibility principle, through to over-engineered mazes of abstraction that are seemingly deliberately opaque. There’s quite a big space between these extremes, and the choices that are taken can become interview questions rather than a pass/fail decision. It’s sometimes nice to include a pairing exercise as part of the interview, asking a candidate to pair with the interviewer to extend their original solution. This grants opportunities to ask about style choices, and also is a good acid test of whether the technical test solution is maintainable or easy to extend!