Seeker is a new optimization solver that does both stochastic and deterministic optimization. It has an approachable expressive Python API and many different use cases. Recently, Nextmv co-founders, Carolyn Mooney and Ryan O’Neil, interviewed Seeker creator and CEO of InsideOpt, Meinolf Sellmann, about the recent focus on stochastic optimization, examples of use cases, collaboration between OR and data science teams, and deployment with Nextmv.
The following captures portions of a longer conversation and has been edited for length and clarity.
Carolyn Mooney: How did you decide to develop Seeker, and where do you think it has a lot of power?
Meinolf Sellmann: I've been a developer for 25 years. I've lived in the MIP world forever and built two MIP solvers during my career actually. That means I'm keenly aware of all the limitations that the traditional software has. While we're talking about stochastic optimization today, the reality is that when you optimize in a business context, you quickly hit the limits of what traditional software can do for you.
Typically, a business doesn't have just one KPI that they want to optimize. They have multiple KPIs and then you have to think of ways to integrate them into one hybrid KPI. Or you have to constrain one and optimize the other, but you don't really have a tool that gives you a good tradeoff between them.
Same thing when you have anything non-linear, non-convex, or non-differentiable, you have real problems there. Parallelism is also always an issue. If the problems get hard, you want to be able to throw compute at the problem, and traditional MIP solvers don't really parallelize all too well. It's very hard to do that with any dual approach. And then of course with stochastic optimization, most real-world problems are coming with estimates where you just don't know the future you're optimizing for. You're often finding that the likely optimal solution for the point estimated future actually lets the business down on the day of operation.
We wanted to build a different kind of solver. We had to jettison the whole duality for it. We built a purely primal solver that provides no bounds whatsoever. It doesn't bother trying to assess how good it is, it just finds the best possible plan it can in the time that you have for it.
Carolyn: We hear a lot from the business side of the house that optimality isn't necessarily the concern, or that sometimes the thing you're solving for isn't necessarily the business KPI. There can be some tension there.
A lot of folks are starting to focus on stochastic optimization. What if you could encapsulate a really solid definition of stochastic optimization?
Meinolf: A stochastic optimization problem is literally every problem in which you are planning for a future that you do not know one hundred percent in advance. This can range from a known input value, where the demand is uncertain, to expected profit optimization, where you might also look for value at risk. You might look at conditional value at risk as well, all of which are relatively simple.
Then there’s robust optimization where you have lots of different scenarios, and you want to have a solution that works across all of them. You can do a more pragmatic approach with stochastic optimization in which you have a distribution for everything.
Lastly, you can have two-stage or multi-stage stochastic optimization all the way to policy learning, where you have to make the same decision over and over again. The end state in which you’re leaving your system is going to be the start state for tomorrow. The question becomes: What is actually a good ongoing policy to learn? You can do all of them with the same API essentially when you view and solve them as a simulated optimization problem.
Carolyn: What are some prime use cases when you think about stochastic optimization?
Meinolf: When you're talking TSP (traveling salesperson problem), who knows how long it actually takes to go from one client today to the next client tomorrow. For example, UPS has to do vehicle routing problems, and they do not know what the traffic is going to be tomorrow. Yet they give time windows to their clients. You have to solve that.
Just recently I did a hydropower optimization. No one knows exactly where it's going to rain, and what I also didn't know is that their basins get emptied and refilled during the day. So it really matters how much rain is actually going to fall there, and it's very hard to forecast.
Lastly, anywhere you're setting prices at the same time, you do not actually know for certain how they affect demand. Those are all good examples.
Ryan O’Neil: Some examples from my experience are that we would try to do things in gig economy or shared economy work, where we make offers to independent contractors through models, but there was a risk of offer rejection. We would try to make these discreet choices on who to give an order or delivery to, but there was a lot of uncertainty as to whether orders would get taken.
Carolyn: Do you think there's more focus in this space as data and science has become more prevalent within businesses? We have seen a rise in the last 5 to 10 years of businesses having data science programs, building more predictive models, and now they need to make decisions given those predictions. For example, we have expected value, but what happens if I'm at my P90, or what happens if I'm at my P10 for demand? Do you think that's impacting some of the push towards stochastic optimization and the ML plus OR landscape?
Meinolf: Yes. The industry where you see it the most is supply chain, which is also the largest client for us. So people are keenly aware of the fact that they can't just ignore risk.
Now the question is, how do you properly manage it? If you've ever tried to deploy a real optimization problem, your stakeholders do not know in the beginning what they want optimized. They come with a problem because they have a decision to make. The moment that you show them the result of such an optimization, and you show them the distribution of how their KPIs look, they suddenly have other wishes. You can combine these things in Seeker. You can measure what the probability is that a KPI hits a certain threshold, and then you can optimize that at the same time you’re optimizing for expected profit, for example, using multi objective optimization. The solver allows you to find a solution where, to the extent possible, you're molding the probability distribution of your KPIs.
Carolyn: I think that's how a lot of business stakeholders think about their systems. They're typically willing to take a cut in one metric in order to preserve another, from a risk perspective, or they think about those aspects given the shift they’re seeing.
How should OR teams and even data science teams think about collaborating and incorporating both ML and optimization in one flow within Seeker? How is that possible with Nextmv? And how can I actually go deploy these systems?
Meinolf: Right. How you make that operational is the business question. You can't just tell all the data scientists, please learn optimization. And you can't tell all the OR people, please also learn machine learning. Maybe the bigger question is, why would you? People may be aware that things don't run so well during the day of operation, but they did their due diligence on their data science team, the machine learners are awesome, they did their cross validation performance, and they've increased accuracy by 3 percent in the past year. They think this is the best that the data will ever give them. And then you shove over, and you lose all uncertainty. You just say, these are my points, and hand it over to the OR team. The OR team comes back with a provably optimal solution. As an exec, you may think this is just not knowing the future and the cost of doing business. But this is not the cost of doing business. The mayhem that happens on the day of operation is the result of over optimizing one predicted future.
The handoff is actually the problem, and this is exactly what I think we've addressed reasonably well. What the machine learning owners do differently now is, they don't provide point forecast. They don’t estimate distribution of the demand, and this is typically very easy. Even if you're not using a Gaussian process, which will give you the distribution for free, you can always use your cross-validation performance in order to make things distributed, thereby also acknowledging that in certain times, you have a much clearer idea of what the demand is going to be than in others.
This can be a heteroscedastic forecast, depending on what the features are. You might have a lot of variants today, while tomorrow you might actually know that this is the way it is. So you just shove those distributions directly into the solver. Then you start having an iteration with the business stakeholders, where they realize that this is what actual distribution looks like in terms of risk.
Carolyn: Yes, you can continue to improve your input into the optimization over time now that input is getting to the point where it's a distribution versus a single value. And you can still have your data science teams or your predictive teams working on those predictions over time and making them better for a given feature set. You can continue to leverage that stochastic optimization framework with Seeker, and you can really improve your decisions that way. It’s an interesting parallel path to separate concerns in the decision making system and the input driven from those distributions.
Ryan: One of the concepts that comes to mind for me is the knife’s edge where without uncertainty, the model optimization is just very greedy. It will optimize itself all the way onto this very thin ridge where it will fall off once there's data. That's not what it was expecting. And now that you know that, now that we're getting technologies where we can add that uncertainty to the optimization, it's not that the optimization is necessarily less greedy, it's just that it has more information to work with.
Carolyn: A final question: What is the best way to get started with Seeker and stochastic optimization, and are there any useful things to know when you're beginning that journey?
Meinolf: We have free collaboration on Google, so no installs required. You just go ahead and do it. Of course, the moment that you're actually getting closer to deployment, you should definitely talk to Nextmv and get it done that way. This is actually what we say as a default because we are a software shop. When people come to us and ask if we can run it, we can’t, but we have the right folks to do it for you. It is very refreshing because the modeling is so different. You really have to get out of the mindset of thinking about the dual all the time and think like a human again. It's greatly liberating if you have only lived in the MIP world so far.
