When troubleshooting an operational issue or a poorly performing plan, the first step is often reproducing the issue. This process can be challenging: finding the input used, identifying the model version, and replicating the configuration in order to replay the run. Once you’ve found the root cause, the next step is identifying a path forward to fix the issue. For example, do you modify an aspect of model configuration? If yes, by how much?
Earlier, we discussed cloning runs (using the same input and model settings) to recreate the initial run. In this post, we’ll discuss how to use scenario testing, a built-in feature of Nextmv, to explore your options by varying input sets, model versions, and configuration settings.
Example: Investigating a route planning issue
In our post on replaying run history, we dove into an issue that arose while planning routes for our fictional farm share company that delivers produce from local farms to customers’ homes. After running the model to route the following day’s deliveries, we found a large number of unplanned stops. We were able to replay the run by cloning it in the Nextmv UI. After looking at the input data and configuration settings, we deduced that a clustering constraint had mistakenly been added to the model configuration, which caused a significant increase in unplanned stops. We re-ran the model without the clustering constraint and the number of unplanned stops went to zero.
Example: Exploring multiple routing scenarios
Removing the clustering constraint worked to find an immediate solution, but now we have new business rules that we need to include in order to comply with how we’re paying drivers. We need to prioritize balancing the number of stops per route first and, secondarily, clustering stops while limiting the number of unplanned stops. We’ll take a look at a bigger dataset, as this decision will affect the entire Dallas area of our operation.
First, we ran the model with a stop balancing objective and a cluster constraint. Similar to what we saw in the last post, there are a large number of unplanned stops. With a stop balancing objective (1000 penalty) and a hard cluster constraint, there are 27 unplanned stops.
This isn’t ideal… not only does the plan have 27 unplanned stops, but we can also see that some vehicles have as few as 10 stops while others have up to 27. We need to find a solution that is more effective at balancing stops and clustering routes with fewer unplanned stops. Let’s look into this issue further and replay the run with different settings using scenario testing.
To get started, let’s identify the input used so we can replay the run using the same data. We can easily find this input for this run by navigating to “Input” at the top and downloading the file. No datamart hunting and script writing required to replay the run.
Performing a scenario test to tune configuration
Now that we have our input, we can perform a scenario test in order to analyze the KPIs of multiple plans with different model configurations, using a combination of stop balancing and clustering features. With Nextroute, the VRP solver we’re using, there are two ways to include clustering: a clustering constraint and a clustering objective.
In our initial run, we used the clustering constraint (a hard constraint) which resulted in 27 unplanned stops. For the scenario test, we’ll use the clustering objective which has a configurable penalty value that we can adjust. We will look at the resulting plans using a consistent stop balancing objective penalty of 1000 and varying cluster objective penalties of 0, 25, 50, and 100.
Analyzing the results of a scenario test through statistics
First, we’ll look at unplanned stops from each scenario. Since we used a consistent penalty value of 1000 for the stop balancing objective, we see that value across the second row. For the cluster objective, you’ll see that the values of 0, 25, 50, and 100 resulted in 0, 1, 8, and 24 unplanned stops, respectively.
We can click into each of the corresponding runs to see what the clustering and unplanned stops look like on a map.
With a stop balance objective (1000) and a cluster objective (100), the routes are tightly clustered, but there are 24 unplanned stops.
With a stop balance objective (1000) and a cluster objective (50), routes are still clustered, but there are 8 unplanned stops.
With a stop balance objective (1000) and a cluster objective (25), routes are not as tightly clustered, but there’s only one unplanned stop.
Next, let’s go back to the results of the scenario test to look at how stop balancing changed per scenario. Below, we can see the maximum number of stops per route. Again, since we used a consistent penalty value of 1000 for the stop balancing objective, we see that value across the second row. For the cluster objective, you’ll see that the values of 0, 25, 50, and 100 all resulted in a maximum of 27 stops per route.
Lastly, we’ll look at the minimum number of stops per route. For the cluster objective, you’ll see that the values of 0, 25, 50, and 100 resulted in a minimum of 23, 17, 14, and 14 stops, respectively.
The results from our scenario test indicate that using a stop balancing objective (1000 penalty) paired with a cluster objective (25 penalty) will meet our needs of both balancing and clustering routes with very few unplanned stops. From here, we can manually assign the one unplanned stop or perform further testing to tune the objective penalties.
Get started
Replay runs to fix operational issues and tune model configurations in order to find better plans. Ready to try scenario testing for your use case? Sign up for a Nextmv account and please reach out with questions.
