Since our first acquisition in 2012, we have been interviewing and hiring software engineers, test engineers, designers, DevOps and SREs. We launched AbarCloud, which was a public cloud platform in 2016 and we spent quite a bit of time hiring and training DevOps to help us scale. We needed great troubleshooters, people who can see the tip of an error string and keep pulling till they get to the root cause. But how do we test for this skill during interviews? That’s when the idea of live incident troubleshooting during interviews came to mind.

Fast forward to February 2020 after AbarCloud was acquired. We decided to take that idea and make it into a product - CircuitOps was born. The whole idea behind CircuitOps was to test for troubleshooting abilities using real environments with broken components and asking candidates to diagnose and fix them.

We failed CircuitOps in June 2020, 3 months after starting. There are four main reasons why CircuitOps failed - here is the post mortem.

The job market collapsed due to COVID-19

Starting April 2020 COVID-19 struck hard in Europe and the US and the job market crashed. Candor started a hiring freeze tracker that showed the extent of the damage. Out of the 8000+ companies being tracked, 43% of them had frozen hiring and 11% were doing layoffs. The majority of layoffs were not DevOps or SREs, however, the hiring freeze affected these positions also. Due to cost controls put in place by companies, any new tool purchases also had to go through review, which made the sales cycle much harder and longer.

DevOps and SREs are in high demand

DevOps and SRE positions are high in demand, which means companies did not get many applicants when advertising for these positions. Any candidate who did apply was given personalized, and for senior roles, white-glove treatment. This essentially removed one part of our market which was the ‘pre-filter’ tests.

The only country we heard that bucked this trend was India. When famous companies in India advertise for a role, they could get a few hundred applicants easily even if candidates did not meet the minimum criteria. They needed to filter out the non-qualified candidates quickly and focus on the real candidates.

Some companies did not like this style of test

When it came to companies telling us why they didn’t like this style of test, they had a few reasons… Some preferred to have verbal test interviews with candidates so that they could speak out loud and communicate their thought process. Others said that this style of test put more stress on candidates on top of the already stressful situation of job hunting. Some said a candidate could easily cheat in this style of test if it was sent home as they can have someone help them or even complete it for them.

Troubleshooting is only a part of the job and there are alternatives

We wrote a blog titled The difference between hiring DevOps/SREs and Software Engineers in which we discuss that DevOps and SREs only spend part of their time troubleshooting. CircuitOps fully focused on this skillset, therefore, we found that companies had to go through their entire hiring process and add this tool onto the todo list. The overhead of doing this via another product was hard for them, given there are alternatives such as spinning up an AWS instance and manually breaking something or going through scenarios via discussion (what would you do if…).

Conclusion

We could have pivoted CircuitOps into helping companies hire software engineers, but this space is already crowded so we decided to shut down and move onto other ideas. We may open source some of the cool things we developed during the process, and if we do, we will write a blog about it on this site & Tweet it out.