Scenario-Driven Interviews

Interviewing for tech jobs is miserable. You do a phone screen and things go well and then you do a 5 or 6 hour interview loop which usually looks a little something like this:

• Tell us about a time someone hurt your feelings and how your ego survived
• How many functional toilets are in Manhattan? ¹
• Show us how you would balance this red black tree, and no google, that’s cheating!
• Pop quiz! What’s the 76th bit in a TCP packet?
• What happens when you type twitter.com/0xdade into your browser? ²
• Draw me an architecture diagram for this product that we’re going to be launching but don’t have anyone to architect so we’re milking our interviewees for ideas.

While the examples that I chose here might be a little too specific for you to believe them, I’m sure anyone who has done any tech interviewing is familiar with the type of question that each represents. Each one is asked by a well-meaning individual who was told that they are responsible for gauging you on some specific part of your performance. They may have had a little corporate mandated training class on how to conduct interviews, if you’re lucky. But even when you’re lucky, those training classes tend to either be one-size-fits-all or one-size-fits-most - maybe they have one specifically for interviewing engineers.

The problem with this approach to interviewing, I think, is that it encourages canned answers for common questions and puts undue pressure on the candidate to study things that they will literally never need if you were to hire them. This is especially true if you’re hiring engineers for specialized roles such as security or quality assurance. Security Engineers and Quality Engineers both have “Engineer” in their title so they should go through the same interview loop as our Software Engineers, right? Well… no.

“But dade, our employees need to be cross-functional and be able to perform under pressure!!!11!one!” Yeah okay boss. If your company culture is such that employees have to solve obscure problems in less than an hour without prior communication about the requirements and planning and preparation time, then let me just grab my things and I’ll see myself out.

The Situational Interview

Situational interviews are those that place you in a situation and typically involve trying to better understand the candidate’s problem solving abilities. If we look at our example interview loop above, the red black tree question and the architecture question would be considered situational interviews because they pose a specific situation for the candidate to solve - an unbalanced red black tree and a business problem that needs a solution. You might think that some of the other example questions are situational because they also involve problem solving, such as the “toilets in manhattan” question. While this type of question does indeed explore a person’s ability to solve problems, there is no situation here in which they actually needed to solve this problem. It’s also not likely to be a problem they’ve ever had to solve before, which means getting more hypothetical answers than concrete answers.

I think situational interviews are generally good. They are certainly better than pop quiz interviews, and I think they also have potential for developing a better behavioral understanding of the candidate than most behavioral interviews. But I also think there’s a lot of room to do situational interviews poorly. Consider the following:

If you know your boss is 100% wrong about something how would you handle it? ³

This is, indeed, a situation. It also teases some insight into the candidate’s behavior. But the problem I have with this question, and hopefully a problem that you have with it as well by the time you’ve finished reading my ramblings, is that it encourages a canned answer. Maybe the candidate doesn’t have a perfectly canned answer for this question, or maybe you’ve made a minor tweak so that they hopefully forget their canned answer, but even still a candidate can easily choose to talk about a time they did (or didn’t but want you to believe they did) their best at this situation. ⁴ Or maybe they say they’ve never had this experience but they would handle it <some optimal way>.

It’s easy to look back at a situation and say “I should have done this differently.” You probably do it nearly every day, whether it was a fight with your spouse, a disagreement with a coworker, an angry exchange between you and another shopper at the grocery store, etc. But just because it’s easy to see things in hindsight doesn’t mean that we’ve actually adopted those modified behaviors. Just because a candidate has a good example of a time they handled this situation, or is able to articulate good ways they would handle the situation, doesn’t mean it is how the candidate will handle the situation.

Understanding what the candidate thinks is the ideal way to handle a situation is great. It helps determine if their ideals are in line with your / the organizations ideals. There’s absolutely value in that. But it misses the day to day behaviors, which I believe to be infinitely more important for a healthy and happy working environment.

The Scenario-Driven Interview ⁵

“dade what the heck man, aren’t situations and scenarios basically the same thing?” – You, probably.

If you’re not a word nerd like me and you don’t enjoy studying the English language, please allow me to preface this section with a set of definitions.

Situation

a set of circumstances in which one finds oneself; a state of affairs.

Scenario

a postulated sequence or development of events.

Okay so now that we have the OED ⁶ defintions out of the way, I’ll try to explain in my own words. A situation tends to focus on some real set of events and/or circumstances that are either happening to you now, or have happened to you in the past. A scenario tends to focus on some hypothetical set of events and/or circumstances that have not happened to you, but could. Does that make the distinction more clear?

Benefits of a Scenario-Driven Interview

In a scenario-driven interview, the candidate is asked to assume some set of beliefs for the duration of the interview, and are asked to behave as if they are actually in that scenario. Because you’re creating a series of circumstances and events for the candidate to experience with you during that interview, you are giving them an opportunity to show how they might actually handle it in a day to day scenario. It is, in my experience anyways, significantly harder to provide idealized answers to a hypothetical scenario currently being explored than it is to provide idealized answers to specific, direct questions.

A domain-specific scenario will also help get a better understanding of the candidates functional handle on the domain, rather than just their hypothetical handle on it. You are exploring the domain directly, together. If the candidate doesn’t have a conversational level of understanding, perhaps the job they think they are applying for and the job you think you’re hiring for are misaligned.

Drawbacks of a Scenario-Driven Interview

Well, I think the biggest drawback of the scenario-driven interview is that it requires the interviewer to possess the necessary analytical capacity to note the “how” and the “why” of the interviewees actions during the interview, rather than simply the “what” that you’d get by asking direct questions. The outcomes of the scenario-driven interview are indirect for the interviewer - the end state of the scenario is significantly less important than the journey taken to get there.

Another potential draw back, depending on how your organization currently approaches interviewing, is that it’s not likely to produce standardized results. Sure, the interviewer can be looking for the same set of traits across interviewees, but because the focus is a little more meta than just jotting down exactly what the interviewee says, there is margin for error. That margin is even greater if the same scenario is explored by two different interviewers with two different candidates, because different people will take away different subtleties from the same scenario.

Traits To Look For

It depends.

Perhaps even more so than any other interview format, it depends.

We’ll get into some example scenarios momentarily where we can explore the different types of things that might be unique to each scenario, but first we can talk about a few traits that might be more broadly useful to look for.

Common Traits

This is a non-exhaustive list of traits that you might want to consider looking for when conducting any scenario-driven interview, regardless of the scenario domain.

• Is the candidate asking questions throughout the interview to better understand the state of the scenario?
• Is the candidate making assumptions about the scenario that are inaccurate?
• How does the candidate respond when the interviewer challenges the validity of their decision?
• Does the candidate ask for help when they get stuck?
• How does the candidate respond when the interviewer introduces new obstacles to the scenario?
• How does the candidate’s response to new obstacles change as more and more new obstacles keep coming up?
• Does the candidate acknowledge the bounds of their understanding and suggest bringing in assistance from someone more suited for the scenario at hand?
• Is the candidate exploring well established options or are they taking creative new approaches? Is one of these more suitable for the role that you’re hiring for?
• Does the candidate consider customer asks (whether internal customer or external customer) throughout the scenario? Are you hiring them for a role where you’d consider this valuable?
• Is the candidate considering the long-term ramifications of their decisions or are they purely focused on the short-term immediate solution? Are you looking for a long-term visionary, a balance, or someone who can get a problem and quickly turn around a solution?
• Is the candidate too narrowly focused on a specific component of the scenario, such that they miss other objectives of the scenario partially or completely?
• Does the candidate explore down one route for some length of time and then go back to a previous decision and explore another option? If your scenario was a top-to-bottom decision graph, do they ever go back up the graph?

Domain-Specific Traits

Domain-specific traits are going to be harder to capture here, because every domain is going to have a different set of traits that will be valuable to them. Because I’ve worked in Quality & Reliability and Security roles, I’m going to give a few examples for those in particular.

• (QA) Is the candidate able to comfortably take the requirements of the scenario and discuss testing concepts such as feature tests, integration tests, regression tests, load tests, etc?
• (QA) Is the candidate exploring creative test cases and can they reason about situations where those test cases would be important to have the answer to?
• (QA) Can the candidate take the tenets of quality engineering and transfer them to a product or problem space that they are unfamiliar with? E.g. can someone who has done a lot of computer hardware QA transfer the QA skills into doing QA for a fancy new conveyor belt system?
• (Sec) Is the candidate considering the appropriate level of risk or the types of threats involved when discussing the scenario?
• (Sec) Is the candidate considering the severity and/or impact of a vulnerability in the scenario when considering solutions?
• (Sec) Does the candidate consider compensating controls or otherwise view security as a series of controls, or do they narrowly focus on identifying the underlying vulnerability? Does the role you’re hiring for need one of these views more than the other?

Designing a Scenario

Okay so we’ve got a surprisingly long list of possible traits to be on the lookout for, now we need to design a scenario that we can use to answer some of these questions.

The first thing you need to do when designing a scenario-driven interview is throw out all notion of there being a “right” and “wrong” answer. Candidates will be put into scenarios with the specific intent of seeing how they handle the scenario as it evolves, not seeing how much you need to coach them to some pre-defined destination. Some of your scenario-driven interviews might end in similar places with different candidates, however more likely than not you should be ending up in different outcomes.

Think of designing this interview like you’re designing a game. Not super hard platformer with high scores and it costs an hour to play. Rather, an open-world video game like Skyrim, or a tabletop game like Pathfinder, where the player is a character that can make many varying decisions to affect themselves and the world around them. What is the purpose of your game? What objectives is the player trying to reach? What sorts of complications might arise along their journey?

Two paragraphs ago I said that you don’t want to coach the candidate to some pre-defined destination, and then I just said you should ask yourself what objectives the player is trying to reach. This might be a little confusing, but there is a subtle distinction that is worth calling out. You want the candidate to have an objective. What that objective is can vary wildly, but you want them to have something to work towards. You want to give them a purpose in the scenario, and then give them the freedom to approach their purpose however they see fit.

Maybe that objective is “figure out what’s wrong with this broken system,” or maybe it’s “You’re an attacker and your goal is to steal customer credit card data.” Either way, you’ve not defined how they succeed. You’ve merely defined what their success should look like. The candidate is entirely responsible for figuring out how to get there.

A good scenario should have:

• A theme. Your theme will sort of dictate the rest of the components. Maybe your theme is “Outside the box thinking”, maybe it’s “Customer service”, or maybe it’s “Leadership skills.” This will change based on the role that you’re hiring for, but will understanding the theme of the scenario will allow the interviewer to more quickly respond to changes in the scenario during an interview. You do not need to convey the theme to the candidate directly, but should ensure that the interviewer is keeping the scenario in-theme.
• A main character. The main character is played by the candidate and you don’t need to do too much to define them. If they are being asked to assume a role, simply tell them the role. You can be generic if you’d like them to inquire more about their role. E.g. “You are a hacker”
• An objective. What is the character supposed to be doing in this story? E.g. “you want to monetize your hacking”
• Conflict. You should plan for many small conflicts as the plot unfolds. Don’t pre-plan specific conflicts and try to shoe-horn them into the scenario while you’re in it. Instead, be prepared to respond to the candidate’s actions with small complications. E.g. The candidate says “I’ll look for cloud keys in their github” and the interviewer goes along with it, but then when the candidate says “I use the cloud keys to access AWS”, the interviewer says “Invalid credentials.”
• Resolution. Depending on the type of scenario and how it unfolds, you may want to plan to allow the candidate to achieve their objective by the end of the scenario. Other roles, you may be perfectly fine with the scenario never reaching a conclusion.

Interacting with the Candidate

A scenario-driven interview is more like a multiplayer game or a play than it is a formal interview. The interviewer and the candidate should be interacting back and forth. Because the emphasis is on the candidate’s decisions, behavior, and approach, the candidate should be doing most of the talking, however the interviewer still needs to be able to answer questions, ask questions, and inject complications.

When kicking off one of these scenario-driven interviews, explain to the candidate what is about to happen. Explain that there is no right or wrong answer and that it’s a collaborative exploration of a hypothetical scenario. Encourage them to ask questions, raise concerns, and talk through how they are thinking. If it’s a domain-specific scenario, then preface it with “We want to explore a scenario that might occur in the course of your job.” If it’s not domain-specific, or you know that it’s something they aren’t likely to have experienced, then preface it as such - “We want to explore a scenario that you probably haven’t been in, and it’s okay to not have knowledge specific to the scenario.”

Don’t warn them that you’ll inject complications or where, just do the first complication relatively early on so that you leave room for additional complications later in the scenario. Also don’t tell them the specific traits that you’re looking for - this ensures that you’re getting a more authentic representation of them, versus the tailored answers where specific traits are kept in mind. If you tell them the traits you’re looking for, you’ve created a “right” versus “wrong” answer, which you don’t want in a scenario-driven interview.

Example Interviews

Having run these sorts of interviews many times, I thought it would be good to include a few examples for you. I’m not going to dive too deep into what specific traits I looked for in each scenario, but will speak to it briefly.

The Red Team Job

Alright so you’re an attacker, you’re looking for to dump a customer credit card database in this target company. You know that it’s somewhere in this network over here *draws circle representing “production” network*. How do you approach getting this data?

This is a very generic red team scenario. It’s extremely open ended, allowing the candidate to exhibit their preferences, biases, and strengths. It contains just enough information that an inexperienced red team candidate will jump right into trying to break into the production network, maybe phishing the employees or taking a physical approach to getting network access. The focus will largely stay on the objective using the details that were provided,

A more experienced red team candidate might first stop to inquire about the premise of the scenario. “Okay but what kind of attacker am I? Am I part of a government-backed organization with long term strategic goals? Am I part of an organized crime group simply trying to dump the credit card data to sell in bulk online? Am I a bored teenager who thought it would be fun?” They are similarly more likely to consider alternate paths to get the data, such as targeting backups, staging systems, CI/CD pipelines, and other places that aren’t necessarily in the target network but would position themselves to better meet the objective.

In red team scenarios, I tend to care about the types of questions that the candidate asks, the places where the candidate makes assumptions without clearly calling them out as assumptions (e.g. saying they’ll just get domain admin without first making their assumption of an AD environment explicit), how comfortable they are with hitting dead ends, and how comfortable they are with voluntarily taking a step back to look for other paths in the so-called “attack graph”.

The Quality Engineer

Alright so you’ve just been selected to lead the quality assurance efforts for our new product, a WiFi enabled smart toaster. What do you think we should consider for our quality strategy and how do you propose we begin?

When running QA based scenarios, I really like to focus on a product that the candidate has probably not already done quality engineering work on. The reason for this is that I find it encourages and explores more creative answers, while still leaving plenty of room for practical and typical answers. If you ask someone to design a QA plan for a product they’ve already worked on, or a product that they are very familiar with, you’ll end up with mostly a clone of some other QA plan. While this might be okay for some roles, I find the ability to explore new test scenarios to be an extremely valuable skill. Similarly, I also find the ability to apply domain knowledge to an area outside their domain to be exceedingly valuable as it shows a firm grasp of the expertise.

For QA based scenarios, traits I care about will include things like how many questions they ask about the product and it’s potential use cases, how creative their test ideas get, how practical their test ideas get, and perhaps most importantly, how comprehensive their test plan is. Even if some of the tests they come up with are too creative and wouldn’t get implemented, I want to see them considering edge cases, unintended uses, and their rationales for why they think that test is a good idea.

The Troubleshooter

Alright, I’m going to put you into a scenario that you’ve hopefully never been in before and we’re going to troubleshoot an issue together. Have you ever worked in food service? Oh you have? What about hotel management? No? Great, you’re an employee working the front desk at a hotel and a customer approaches you and says “My card doesn’t work.” What do you do?

This scenario may be a good non-specific scenario that can fit many roles. It involves interacting with a customer, solving a technical problem, and being in a relatively unfamiliar environment. There are many areas to insert complications, many opportunities for the candidate to reason about how the system(s) might work, and many opportunities to play a difficult customer or a difficult manager.

In this troubleshooting oriented style of scenario, I like to consider traits like whether or not they remember to take care of the problem for the customer when they realize it’s not a simple fix, how they reason about the systems that might be broken and how those systems interact with one another, how they respond to their attempts to solve the problem not being immediately fruitful, and how well they are able to adapt to the new, unfamiliar environment.

Conclusion

I know this was pretty long, so thank you for sticking with me. I hope that you’ve been able to find some new ideas on how to approach interviewing and ways to find value in the interview process beyond simply asking the candidate to tell you about their value. If you have ideas for ways to improve upon this style of interview, you have ideas for why I’m a terrible person and this will never work, or you have cool ideas for traits or scenarios, please tweet me. If you’re an interview candidate and you absolutely hate these types of interviews, also please tweet me.

(This image originally found at The Oatmeal)