Hiring Engineering Leaders

This post is about the interview process for an engineering leader. Hiring involves a lot more than the interview part - everything from reviewing resumes, sourcing great candidates, getting them to come onsite, the interview itself, evaluation, offer, and close. In this post, I focus primarily on mechanics, propose a concrete interview process, along with sample questions for the non-technical parts, and propose a template that companies can use to build a custom process suited to their needs. This post is based on my long experience managing engineering teams at Google and Facebook and interviewing numerous amazing managers for various roles. This entire post is available as a coda doc that you can duplicate and use.

Your startup is going great. You have a cool product with hundreds of thousands of users, a strong engineering team, and amazing leaders of design, sales, and other functions. You want to scale your engineering team and are looking to hire your first engineering VP. You chatted over coffee with a few candidates and have convinced them to go through a formal loop. What are the types of interviews in your loop? What are the goals of each interview? Are there some questions that are better than others? I propose to answer these questions, focusing on the non-technical parts of the interview. First, we need to align on what the role is about, on diversity, and on having a great candidate experience.

The Basics: What does a VP of Engineering do?

Martin Casado’s excellent post is the best summary of why you need a VP of Engineering and the value they should provide. A great engineering leader should be able to function across the following axes:

Business / Product impact

1. Ensuring engineering execution is one of their most important jobs. “Execution” refers to repeated, on-time, high-quality delivery of things the engineering team is tasked to build. The leader should be able to balance offense (e.g., growing the product, building features, improving performance) with defense (e.g., reducing the bug load, rewriting systems to pay off bug debt, improving efficiency, and reducing cost).
2. They should provide critical input into the product planning and development process. VPEs are a thought partner for other executive leaders in the company - the CEO, CPO, Head of Sales. They should have an opinion on relative prioritization and technology investments the company makes.
3. Communication is a significant part of a VPE’s job. They must be excellent at communicating upward, downward, and sideways. They need to manage the CEO, so the engineering team doesn’t get thrashed, they need to communicate to the engineering team, to keep them in the loop and aware of everything going on. They need to be able to talk to clients, to the sales team, to the customer support team. They must be a highly versatile communicator, able to juggle a large variety of audiences and contexts.

Team impact.

1. They are responsible for the continued growth and development of the engineering team. Senior engineers need hard challenges in the form of ambiguous but critical problems and leadership opportunities. Junior engineers need mentorship and challenging work appropriate to their level. The leader should be able to balance a team with the appropriate seniority mix so that everyone has opportunities to learn and grow. They are responsible for delivering feedback as well as for letting people go when things are not working out.
2. Hire, hire, hire. If the engineering team is supposed to double or triple over the next 12-24 months, they need to be spending a substantial chunk of their time sourcing, interviewing and closing great talent. They should be able to figure out the org structure 12 and 24 months into the future and then hire toward that. They should be able to build up their bench by grooming or hiring the next set of leaders that can take their place.
3. They must be able to lead the team when times are good and bad. Keeping morale up when things are rough, keeping the team focused on the highest priority deliverables, helping people feel good about doing some of the most challenging, hard, and impactful work they have done - this all falls upon the VPE.

An important note about Bias and Diversity

If you haven’t read Rachel Thomas’s excellent 3 part series on the diversity problem in tech and practical ideas on what to do about it, you really should. Diversity is a big topic for our industry and your company. The rest of this article can wait, please read the series.

The design of the interview process plays a crucial role in increasing the diversity of your company. Essential things to keep in mind:

1. You are biased. I am biased. Everyone is. If you believe otherwise, take the Race IAT. Then take the Managing Bias class to get a little better.
2. Without a corrective process, we tend to hire people like ourselves, we hire for confidence, not competence, and we are likely to tilt the balance in favor of candidates with pedigree.
3. Sponsoring and attending Grace Hopper does not solve your diversity problem. Treating women and under-represented minorities at your company well, and fixing your hiring and promotion processes can build diverse teams.

I recommend a Diverse Slate Approach for leadership hires: implement the Rooney Rule for your company. In short, for any leadership position, you must consider at least one woman and one underrepresented minority candidate.

DSA is hard work. In a difficult hiring environment, this can be especially challenging. Adopting this is a first step, but an important one. It signals that you are willing to do what it takes to build a diverse workforce and are committed to increasing workplace diversity, even with some short term pain.

The Basics: The Candidate Experience

Treat every candidate with respect and gratitude for their time. They could be a current or future client. Even if they are not the right fit, you want them to walk away with a great interview experience. I have personally been on the receiving end of poor interview experiences (hostile interviewers, no food or drink, no time to stretch or reset my brain between interviews), and it was not fun. Make sure candidates stay hydrated (they are going to be talking a ton), are not hungry, and are offered frequent breaks to stretch or use the facilities. One interviewer even offered a walking interview (it was awesome!).

Process Overview

The process comprises of one or two screens to make sure that the candidates pass the sniff test and are sufficiently excited about your company to bring them in for a full loop, followed by a day or two of interviewing, following by sells. Reference checks are essential - I recommend asking candidates for references as soon as you have some reasonable signal and interest so that you can get moving on talking to their references as quickly as possible. I prefer asking candidates for their references and using those rather than doing back-channel references, a common valley practice that is rife with challenges.

1. Screen
2. Onsite 2. Technical 1. Code comprehension. Lightweight coding. 2. Software design. 3. Domain-specific deep dive. Optional. 4. Management 5. People Management. 6. Strategy, Vision, and Execution. 7. XFN Collaboration and Communication. 8. Values
3. References
4. Sell

Screen

Key questions include digging into experience managing teams at various stages of growth, familiarity with your domain, with your company’s tech stack. Don’t forget to spend a significant amount of time selling your company and your vision and getting them excited about coming in. Keep an open mind at this stage and don’t weed out candidates too early. If a candidate has a greater than 25% chance of passing your onsite interview, bring them in for a full loop.

Technical Interviews

The chances are that your company already has a process for technical interviews. I recommend tweaking the senior engineer interview process and using that for the VPE. If you need some help setting up a technical loop, please drop me a line (Twitter @radoshi.

For leadership candidates, I recommend focusing on:

1. Code comprehension over code writing. They need to read and understand the code that the team is writing. They are unlikely to be coding the Next Great Feature.
2. Good software design instincts over specific technologies. They should understand why you need a cache but needn’t go into the finer details about Memcached vs. Redis.
3. Domain-specific deep dive. Product, ML, Infrastructure, DevOps, AdTech – each leader grew up in some background. You’re looking for depth of understanding in their field.

Leadership Interviews

I recommend 3 interviews to understand the candidate’s leadership style.

1. People management.
2. Strategy, Vision, and Execution.
3. XFN Collaboration and Communication.

I found open-ended questions, such as “what is your management philosophy,” to be less useful in distinguishing great answers from mediocre ones. An Situation-Complication-Resolution style, as applied to interviews, is an effective alternative. Setting up specific examples, or asking for specific examples, gives you much firmer ground to dive into details. Continuing with the preceding example, to find out where they are on the directive / supportive spectrum, we could ask the following: “We have an engineer Alex, he’s been here 6 months. We hired him straight from university.” (situation) “He works hard, but writes buggy code and people have to clean up his mess frequently.” (complication). “What would you do?” (resolution). This setup gives you ample fodder to deep dive into how the candidate approaches these sorts of problems and where they lean philosophically.

People Management Interview

The goal of this interview is to determine if the candidate is an exceptional people manager. Can the candidate provide the team with career guidance, help them work through obstacles, and push them forward? Does the candidate have sufficient experience dealing with problems, or are they going to be learning it on the fly? Is the candidate going to be able to hire and scale the team? Are they going to be able to scale themselves as the company grows?

Sample questions

Basic data

1. Management history.
2. Historical and current team sizes.
3. Growth curves - did they build the teams or inherit them?

Management mechanics

How do they keep in touch with their team? 1-1 - what is the purpose, structure, and cadence?

Engineer growth

1. How do they assess an engineer’s strengths and areas of growth?
2. What do they focus on, and what is the process behind making that decision?
3. How do they grow junior engineers? Senior engineers? TLs?
4. How do you give feedback? Give an example where you had to deliver hard feedback to someone?
5. Give an example of someone that wasn’t doing well and what you did to help them improve.
6. Have you ever had to let anyone go? What was the thinking and process behind it?

Manager growth

1. When does a team need a manager?
2. When is someone ready to be a manager? How do they assess this readiness?
3. How is managing a manager different from managing ICs?
4. How is managing Directors / VPs different from managing managers?
5. How do you know when a manager isn’t doing well? How do you debug? How do you help?

Organizational leadership

1. When do they go from an amalgam of engineers to a formal team?
2. What is their ideal team size?
3. How do they think of organizational structure? Is it necessary? At what stage? When does one re-organize the team?
4. What are the mechanics of a reorg? Have they executed a non-trivial one?

Team culture

1. How do they build engineering culture?
2. What is their weekly eng cadence?
3. Do they do architecture reviews (why or why not)? Do they do regular engineering all hands (why or why not?)
4. How do people in the team learn from one another and grow?

Strategy, Vision and Execution Interview

The goal of this interview is to determine if a candidate can set the engineering vision, determine a strategy, and execute on it. They need to understand company strategy and translate that into an engineering strategy. They need to be able to decompose complex, ambiguous directions into concrete projects. Every candidate ha a set of tools and processes they use to keep the trains running, to stay on top of things, and communicate upward, downward, and laterally. At a leadership level, communication is often the primary job, and you want the candidate to be exceptional in this regard. All good projects hit bumps at some point. What happens when things go wrong? How do they detect what’s wrong, debug it, and fix it? The ideal candidate is one that makes bold bets, has seen some of them fail, learned, and did something better next time.

Sample Questions

History

1. Describe a complex project that you ran, how much time did it take, how many people, was it successful, what went well, and what did you learn? (This is intended to be a warm-up, don’t spend the entire interview diving into this).

Inception

1. Construct an ambiguous scenario where a new project is needed. Pretend you’re the CEO and let the candidate ask you questions. Can the candidate quickly get into the heart of the problem you’re asking them to solve (comprehension)?
2. Can they come up with a sufficient first approximation of a solution? Can they decompose the problem into different tracks?
3. How would they set up the engineering team to tackle such a project? How do you manage ongoing maintenance and bugs while tackling projects?
4. How do you deal with morale impact of thrashing the team?

Goals

1. Goals discussion is significant enough to be pulled out into its section.
2. Does the candidate ask about a project’s goals?
3. Can they create reasonable metrics that track success? The discussion can get pretty nuanced but is incredibly important. Without clear goals and metrics, execution becomes an order of magnitude harder.
4. Pick a project they did that was successful. Dive into the project’s goals, measurements of success, relationships to company success, decomposition into smaller projects, and execution of the entire thing.

Project lifetime

1. How do they track progress?
2. How do they communicate progress?
3. How do they know if things are going well or not going so well?
4. If they aren’t going well, what sort of corrective actions would they take?
5. What would they do if they incorrectly estimated time, complexity, or resources?

Conclusion

1. How do they transition a team from one project to another?
2. What do they do if an idea fails?
3. What do they do if the project didn’t complete on time and requires further investment?

XFN Collaboration and Communication Interview

Thanks to Connor Hayes for providing this section.

In this interview, we are trying to assess whether the candidate can communicate in a clear, concise, and structured way. Do they have the right attitude toward working with non-engineering functions? How do they maximize their work through other people? We’re also looking for grit and scrappiness - can they push through tough situations? Can they unblock themselves and get stuff done? Finally, we want someone that is going to be a thought partner for the company’s senior leadership: do they grasp abstract concepts quickly? Will they contribute to strategy and decision making?

Sample Questions

Communication

1. This is observed through the structure of answers to the other topical questions.

Collaboration

1. What is the ideal working model for collaboration amongst various functions like Product, Data Science, and Design?
2. Describe a lousy collaboration situation. How did you handle that? What did you do?
3. Let’s say you were working with a PM who made a decision with which you didn’t agree. What would you do?
4. What if the situation didn’t change, and you still disagreed? How would you escalate?
5. What do you do when a product person gives you an unrealistic deadline for a project?

Grit and scrappiness

1. How did you get to where you are today? Tell me about the times in your life where you had to work the hardest and overcome obstacles.
2. What is the most unfair thing that has happened to you in your career? What did you do about it?
3. Tell me about a time when you were blocked on a product decision and weren’t getting help from Product. How did you push through?
4. What’s the most challenging project you’ve worked on, and how did it go?

Thought partnership

1. Describe a strategic problem that the company is facing. Ask for their input and see how they reason through things.
2. Let’s pretend we are 3 years into the future, and our company has gone out of business. What do you think are some possible causes?
3. How could we account for these risks with what we build today?

Values

Your company likely has a values interview, done by senior leaders, to assess whether the candidate has a similar set of values as the company. This interview typically probes into a candidates work ethic, their motivations, their excitement about the company, about the problem it is solving, and whether they would help take the company to greater heights.

References

Reference checking is an incredibly important part of the interview process. I recommend letting recruiting solicit references from the candidate, but having the hiring manager (CEO) do the reference calls.

The critical thing during reference calls is to elicit details about the candidates working style, strengths, and areas of development.

Conclusion

The goal of this post is to provide a VPE interview process that you can duplicate, customize, and implement to hire the best engineering leader for your company or organization. This post is also available as a structured coda doc.

I would love to hear from you if you use this process, or if you have suggestions on making it better. I’d love to hear about things that I have missed, but have proven incredibly useful for you. Leave a comment or DM @radoshi on Twitter.

{% include about.md %}