This is part of a series of blog posts describing our thinking around the various types of optimization model testing. Tell us what you think in our community forum and stay tuned for more and make sure to watch our techtalk on this topic!
Scenario tests allow you to play out different scenarios under realistic conditions before committing to a decision. These decisions might center around how many workers you need to hire, if you should expand into a new market region, or what price you select for customer incentives.
These tests involve varying one (or more) model parameter values and comparing the planned output for a range of values. So if you want to determine how many new drivers to hire, you might add a number of drivers parameter to your model, specify the range of drivers you’d like to explore hiring (e.g., from 10 to 20 drivers), and then compare 11 different outputs (from running the model with 10, 11, 12, etc. drivers).
Scenario tests are sometimes known as pressure tests or stress tests. They also sometimes get conflated with simulation. Simulation is another way of studying a system that is typically focused on individual-based interactions and playing out a series of events over time. For example, I’ve spent time doing simulation in the infectious disease world where individuals have a given state — infected, susceptible, recovered (or vaccinated) — and disease spread is based on 1-1 interactions. Scenario testing is focused on population-level interactions. In our infectious disease example, this means the population has pools of infected, susceptible, and recovered/immune individuals, with rates of moving between states (e.g., based on vaccine coverage). Disease spread is based on changes to those groups rather than changes to the individuals. You can think of scenario tests as a simpler and sometimes more practical version of simulation, which are often more time and resource intensive.
Scenario tests are a great way to get started before graduating to other forms of decision model testing and even simulation. Let’s explore this further.
What’s an example of scenario testing?
Imagine you work at a farm share delivery company. You’re in the business of picking up produce boxes from local farmers and delivering them to customers’ homes using a fleet of 5 vans. The vans have limited capacity and shift availability. You have a decision model that automatically optimizes the van delivery routes because it’s more efficient than manually creating the routes.
Business is booming and you’re regularly hitting the limits of your current van fleet. Leadership is willing to invest in additional vans, but is looking to you to figure out how many drivers (for the vans) to recruit. You start by adding a number of vans parameter to your model (and assume all vans have the same capacity) and then run the model for different values of this parameter: 6 vans, 8 vans, 10 vans, 12 vans, and so on. Then you compare the results for metrics such as unassigned deliveries or balanced stops across vans.
In the end, you determine that adding 3 vans to your fleet scales to handle demand while not underutilizing drivers and you share this information with the team. From there, you can put your new finding to the proverbial test and evaluate how a model with the additional vans performs against predefined acceptance metrics using an acceptance test or even move on to real-time testing with live production data.
Why do scenario testing?
The simple answer for why you should do scenario testing is it saves money and time by generating plans before you commit them. It also allows you to test different decisions in the case of what we call “one-way doors”. For many high stakes decisions (e.g., disaster relief distribution, grant budget allocation, natural reserve planning, and infectious disease control), you get one shot at a decision. So scenario testing is especially important as it allows you to test many different potential futures before expending resources. These tests help teams make more informed decisions about changes that can have real impact to the business and its customers.
When you look at the farm share example above, scenario tests help not only inform fleet size, but also what your deliveries look like under poor weather conditions, a big sale or promotion day, or if you introduce electric vehicles that have different range limits. When you’re free to play out “what if” scenarios you know could happen before they happen, you’re primed to plan and adjust effectively while minimizing cost to the business and impact to customers.
When do you need scenario testing?
Scenario testing is appropriate when you’re looking across a sliding scale of variability. This includes increasing or decreasing something (e.g., on-call staffers or incentive prices), diversifying or homogenizing something (e.g., adding bikes to a van fleet or converting a hybrid vehicle fleet to all electric), and designing something (e.g., where to put a new warehouse or how to divide up sales territories for reps).
Scenario testing is also helpful when you’re faced with multiple decision axes. For example, when you want to understand the interaction between varying unassigned stop penalties and vehicle fleet size. This is hard to do manually and keep track of in your head. Having a systematic way to perform these scenario tests makes the process easier and sets you up for other types of testing such as acceptance tests or even shadow tests.
How is scenario testing usually done?
Scenario tests commonly require a series of manual or brute-force steps. First you need a way of reliably varying parameters and generating input files. In an ideal world, the input files you need for scenario tests are automatically generated for you. But scripts for doing so are often tedious to build and maintain. They have to be smart about data randomization and also tend to be specific to the type of data you’re working with. For example, a system focused on geodata for a farm share company needs to select from viable locations and therefore must be aware of (and avoid) geographic features like lakes, oceans, or national forest land. Similarly, shift scheduling data has to line up with the times and days that make sense for your business hours and holidays.
In addition to needing a way to ideally automate the variation of parameters and generation of relevant input data for your specific use case, you will also need a script to summarize and compare the results of the scenario tests. This process can get unwieldy the more values you want to test — and even more unwieldy when you want to account for interactions between multiple parameters.
What’s next?
We’re designing a robust scenario testing experience within the Nextmv platform. Today, it’s possible to run multiple input files that you create through our batch testing capabilities to perform scenario tests. You can get hands-on with model testing by creating a free Nextmv account to get started. You can also learn more about how we’re thinking about our testing framework in this technical talk.
.png)
