Interviewing Engineers at Sensu

Sensu’s engineering team has grown considerably since raising our first round of investment. As we built out the engineering team, we realized the need for a consistent and objective interview process. We spent considerable time developing what we believe to be an inclusive interview process that accurately reflects the kind of team we’re trying to build. Our goal is not only to help us decide if you’re right for Sensu, but also to help you decide if Sensu is right for you. The engineering portion of the interview process is two parts: an initial screen and a technical interview.

We want you to get as much out of the interview process as we do so that you can make a well-informed a decision about working with us.

The Screen

During the initial one-hour screening call, we work with you to determine if Sensu is the right “next step” for you in your career. We try to clearly lay out the expectations we have of the engineers we’re hiring and the work ahead of them. We explain the core competencies our team needs to be successful, how the engineering team does work, and what our 6–12 month roadmap looks like. After that, we’ll go over your experience and how it might relate to the work you would do at Sensu.

As much as possible, we leave the decision of moving on up to you. If you move on, you’ll then go through a two-part technical interview made up of a design challenge and a practical coding exercise. We have designed the technical interview to show you how our team operates and to include work that is very much like the work you would be doing at Sensu. We want you to get as much out of the interview process as we do so that you can make a well-informed decision about working with us.

Technical Interview

The Architecture Challenge

Our team regularly discusses design challenges via video conferencing with a shared Google Doc for note-taking. The first part of the technical interview, the architecture challenge, takes you through that process with the exception that members of the panel participate more passively than they normally would. The questions we ask are intended to help explore your line of thinking — not to lead you to the solution we expect.

The challenge comes with a set of requirements, and those requirements could be met in any number of ways. So far, nearly everyone has come up with a unique system of their own design. We want you to leverage your strengths as an engineer to solve a problem. The problem specification is very much like the software you’d be contributing to at Sensu, but its requirements are fundamentally different than ours.

The architecture interview is meant to gauge your experience in the core competencies our team needs to be successful. The Sensu engineering team must:

• Build APIs (particularly REST, since we’re still actively building a REST-ish API)
• Write highly-concurrent multi-threaded applications
• Work with distributed systems
• Have a knowledge of databases and storage
• Be able to scale and operationalize software
• Communicate effectively

We do not expect you to show mastery in all (or even any single one) of these competencies. The goal is to determine the breadth and depth of your experience and what you can bring to the team. We assign an independent grade to each of these competencies.

The Coding Challenge

After the design challenge, we have a practical coding assignment directly related to the design challenge where you’ll be asked to implement a portion of the system you designed. Completing this challenge successfully shows us that you know how to write the kind of software we need you to write — or at least that you can figure out how to independently. We allow a great deal of flexibility in how the coding challenge is administered. You may either participate in a two-hour live-pairing session or choose to do a take-home assessment using your tools and the language of your choice. The problem statement is identical either way, but the requirements are a little different based on which you choose.

Our goal with a coding challenge is to assess your ability to write software that meets specified requirements, how well you know your language and tools, and how well you can communicate your solution. We use the same grading scale for the coding challenge that we do for the architecture interview. The requirements for the take-home assessment are different, because we cannot directly observe how you work.

Pairing, at its core, is a knowledge transfer system and a way to combat the loneliness that can come with working on a fully-distributed team.

We’re a team that pairs regularly — it is uncommon for an engineer to work on their own on an entire issue. Pairing, at its core, is a knowledge transfer system and a way to combat the loneliness that can come with working on a fully-distributed team. If you’re not pairing with someone, we expect that you’re able to effectively transfer your expert knowledge of your solution to the rest of the team. So we expect a take-home assessment to be very well documented.

Avoiding Bias in Grading

We use a mastery-based grading policy with four levels. We have generally defined these levels to be:

• Incomplete/incorrect (0): You failed to complete the assessment or address requirements.
• Beginning (1): You have begun to understand the material — i.e., there were many errors and limited understanding of the concept was demonstrated.
• Developing (2): You grasp key concepts and demonstrate ability with limited difficulty.
• Mastery (3): You consistently grasp, apply and extend key concepts with relative ease; you demonstrate clear, complete, and consistent understanding.
• Exceptional (4): You not only have a mastery of the subject, but it’s quite likely you have done substantial research or written about the topic and are possibly a published author. You are widely recognized as a subject matter expert in your field.

The language we use in a grading system must be specific and lend itself to objectivity. By using descriptors like “beginning” and “developing”, we can describe where in the natural progression of subject-matter comprehension you are. Using active words also reminds us that this is a journey, and that we expect you to progress through these levels in one or more areas during your time at Sensu.

This is better than a highly subjective grading system like “strong no” to “strong yes.” While we understand that some people may feel strongly that we should hire a particular candidate, ultimately we are hiring from a pool of candidates, and we have to be able to eliminate bias toward any particular person. Having a numeric grading system with clear descriptions for each grade allows us to rank candidates and understand where their expertise fits with regard to our team.

Using active words like “beginning” and “developing” also reminds us that this is a journey, and that we expect you to progress through these levels in one or more areas during your time at Sensu.

We have included guidelines for scoring that are specific to our interview questions so that we remove as much subjectivity from the grading process as is possible. All of our scores must be submitted independently by each engineer in the pair immediately following the interview. After feedback is submitted, there’s a QA process where we do a mini-retrospective on the interview. During this time we examine whether or not our grading guidelines are helping us make objective choices when grading the candidates and correct if necessary.

Conclusion

Our interviews are the result of some research done on competency-based education, technical career ladders, and methods for eliminating bias in interview processes. We gathered a lot of information from educators, other companies, and peers; but I want to take a moment to give special thanks to Project Include. The team there has put together a treasure trove of information on inclusion and diversity in the workplace, and their section on hiring is an absolute must-read for any hiring manager.

Technical interviews are a time-consuming endeavor for everyone involved. They’re also necessary. We need as much information as possible to make the best hiring decision we can. Our goal is to fill gaps we’ve identified on our team — to help us learn more in areas we know we’re lacking, and we cannot do that without being able to understand if and how you may help in our endeavor to build a well-rounded, highly functioning distributed team. If you think you can help us with these challenges, or if you’re interested in interviewing for one of our open positions, take a look here.