Over the last 8 years, we have built multiple engineering teams, and we have had to interview and assess many candidates applying for roles ranging from junior to senior software engineering, to directors of engineering and to DevOps and SREs (Site Reliability Engineers). We have learnt a lot during this process, and we are constantly looking at improving the process. One of those ways is using technical assessment tools, so in this blog, I’ll review some of the main tools available.

If you’d like to read more about how to improve your interviews overall, I’d recommend two reads: our post on improving DevOps and SRE job interviews. and TripleByte’s post on How to Interview Engineers.

Before I get started, let me say that a lot of the tools in the space overlap significantly, so some of the differentiators are subjective.

CircuitOps

We launched CircuitOps with a focus on helping companies hire DevOps and SREs. The key difference being that we test troubleshooting skills and not coding skills. We did this by launching multi-tier environments with broken components in them, and asking candidates to troubleshoot, prioritise, and fix the issues.

As an example, CircuitOps would launch a scenario with a full kubernetes cluster or multiple app VMs running behind a load balancer. Problems can be introduced anywhere in the stack.

CircuitOps failed in 2020 due to not enough traction. I will write a full post-mortem in a later blog.

The following list is a collection of companies who help with hiring for roles based on coding skills. Note that I have included the funding numbers in this blog because now more than ever due to Covid-19, if you are choosing a tool for a large enterprise to be used for the foreseeable future, it makes sense for it to be financially safer than not.

HackerRank

Launched as InterviewStreet in 2009, and later renamed to HackerRank. Total raised: $58M.

Probably the most well known tool from this list. The company has a dual focus between helping companies assess software engineers (our main focus of this blog) and helping software engineers improve their coding skills and interview preparation via challenges and tutorials. The challenges are run publicly, and contestants get ‘ranked’ based on performance.

They have a well built-out product that is feature rich; these features not only help you create a test via selection in a catalogue, but also help you look at how the questions fared when used by other customers. This includes how many other tests have included the question, how long it took others to complete and how they scored. They also let you create your own questions.

However, all these features come at a steep price. Pricing starts from $3,000 per year for individual plans with 1 user license and 30 candidates a month, and that only includes the standard library questions. Access to the premium questions library starts at $7,200 per year for teams with 2 user licenses and 100 candidates a month. The more expensive ‘Team’ edition also includes other features under a category called ‘projects’, which include ‘role-specific skills through real-world projects’. They also offer enterprise plans that are fully negotiated, see their pricing page for more details.

Subjectively, we found two things that we didn’t like about HackerRank and both are focused on the experience for the candidate taking the test. The questions include a lot of boilerplate code to parse inputs vs just the function that the candidate needs to run. The other is that the questions have a lot of text compared to other tools, so although there is more context, under timed circumstances, it might stress the candidate more.

Here is a screenshot of the sample questions for a Data Scientist:

Codility

Launched in 2009. Total raised: $22M.

The second most well known tool is Codility. They only have one product that focuses on hiring; although they let companies create coding challenges, they do not run coding challenges themselves yet.

Codility covers the features you need to test a candidate’s coding ability. They have a catalogue of questions, and we quite like the way they have written the questions (tho that is subjective). There is a good results page that marks a candidate and how well they have done, which marks the candidate on overall task score, performance of the solution as well as the coding ability. We also like their candidate view, but again this is subjective.

Although they don’t offer a free trial, there is a publicly available version of the app that you don’t need to sign up for here, and a results page here.

One thing we didn’t like is that they hide some of the test cases from candidates when evaluating their code. For example, you can ask a candidate to write a function, and they can choose to run their code against the test cases, but only 3 test cases come back, and 8 tests are done but are hidden. You could definitely argue that this is helpful testing the candidates raw skills without providing hints, but we believe that you will get to see more of the candidate’s abilities by showing these and you will get a greater variety of results.

There is no public pricing listed, but talking to some of their existing customers, their price range starts at around $6,000 per year.

Here is a screenshot of the results page:

Qualified.io

Launched in 2016. Same company and team as Codewars. Total raised: $2.5M.

The team behind Qualified is the team who brought us Codewars, which is an online code learning portal using challenges.

The feature set is more basic than the last two, however, they do cover the table-stakes - tests can be created using predefined questions, programmers can choose their language, and their development environment, and they get assessed with a results page sent back to the hiring manager. There are some more features covering benchmarking stats, comparing how a candidate do compared to others, and detecting plagiarized code too.

They do offer public release notes on what they have been working on, and looking through their release notes, it seems like they are adding new challenges, but not many new features are being worked on, or possibly that they are working on larger features which take longer to release.

There is no public pricing listed, but their CEO in an interview said their customers pay between $6,000 to $17,000 per year.

Filtered.ai

Launched in 2016. Total raised: $120K.

Filtered.ai is a technical assessment tool which also includes a full ATS (applicant tracking system), as they have the different stages that a candidate will go through till they get the job or get rejected.

They offer a catalogue of tests, which include technical and non-technical questions you should ask during interviews. For example, if you select to hire for DevOps, you will get three questions such as ‘What do you find most challenging about being in DevOps role’, ‘why did you choose DevOps as your career’ and one programming question. One of the cool features is that the interview can be done via video responses (HackerRank also offers this, but it’s not as visible as it is in Filtered.ai and it is in the higher price category).

They also ask candidates to verify themselves via social media connections, which ensures you are tracking the same candidate throughout the hiring process. This may or may not be an issue for your company.

On their demo video, it shows that if you pay $1,000 a month you will get the feature rich version of the product. There are also negotiated enterprise plans, which include white-labeling.

Screenshot below:

TripleByte

Launched in 2015. Total raised: $48M.

TripleByte has a completely different approach to hiring technical talent. They take over the hiring process from sourcing to interviewing and to technical assessment of candidates, then they will send you candidates for a final interview with a detailed report on the candidate and how they have done. This means they can test for a wider range of roles and also get a much better sense of who the candidate is. However, this comes at a much higher price to recruit one of their referred candidates, similar to if you were using a recruitment agency who usually charge a percentage of the salary.

Others

CodinGames, HackerEarth, CodeSignal, ByteBoard, Geektastic, Topcoder, Coderbyte… the list goes on with each either focusing on engineering, or hiring in general. I won’t go into the details of the other ones just now, but if you’d like me to cover more of them, then message me and I will add them to this blog.

Conclusion

There are many tools to help you assess software engineers in their coding ability. The majority of these tools have the basics you need to do this, as well their own unique value-add features. If you’d like to hire DevOps and SREs, then you should also test a candidate’s troubleshooting abilities in live scenarios with real incidents.