An efficient modeling language in Python suitable for large scale optimization models

Modelling
, , , , , , ,
1

Hello, community of open energy modelers!

As the scale of energy system models increases rapidly for multi-period, multi-region, multi-sector and multi-scenario cases, the performance of modeling software is vital for efficient construction and optimization of energy models.

According to the wiki page, the major open source modeling software widely used by open energy models are Pyomo in Python and JuMP.jl in Julia. The performance of JuMP.jl has been proved to be top-notch and significantly faster than Pyomo. There have been multiple attempts to accelerate modeling in Python, including the excellent linopy project by PyPSA team and python-mip. However, existing Python solutions are still slower than JuMP.jl on large scale models as shown by their benchmarks: linopy, python-mip.

Therefore, we recently release an efficient modeling language in Python named PyOptInterface:

The key features of PyOptInterface that might be useful for open model energy models includes:

  • Fast speed to construct large scale models:

    It uses the C API of the optimizer directly instead of relying on official Python bindings like gurobipy, so the speed to construct optimization model is extremely fast. The benchmark comparing PyOptInterface with some other modeling interfaces can be found here.

    PyOptInterface is among the fastest modeling interfaces in terms of model construction time: 10–20x faster than Pyomo, comparable with JuMP.jl and faster than some official Python bindings provided by vendors of optimizer.

  • Native support for multidimensional structure:

    In energy models, we often need to handle complex variables and constraints nested in multiple dimensions, such as time, region and scenario.

    In PyOptInterface, we provide a convenient tupledict to store and select N-D variables and constraints easily. Compared with xarray or other packages, tupledict is just a lightweight extended version of built-in dict in Python and has excellent indexing performance.

    Additionally, the variables and constraints in PyOptInterface are just lightweight handles that can be stored in Numpy ndarrays directly to form a N-D array. The link shows how we use a 2-D ndarray of variables to solve N-queens problem.

  • Fine-grained manipulation of optimization model:

    PyOptInterface maintains a persistent model that can be incrementally modified and re-solved without rebuilding it from scratch.

    It supports adding/removing variables or constraints, changing objective function, changing the right-hand side of constraint, change the linear coefficient of a variable in constraint, etc.

    This feature is especially useful for multi-scenario analysis where we need to solve a lot of models with small modifications sequentially.

    This is a simple example where we change the load curve in economic dispatch and re-solve it instantly.

Currently, PyOptInterface supports the following problem types:

  • Linear Programming (LP)
  • Mixed-Integer Linear Programming (MILP)
  • Quadratic Programming (QP)
  • Mixed-Integer Quadratic Programming (MIQP)
  • Quadratically Constrained Quadratic Programming (QCQP)
  • Mixed-Integer Quadratically Constrained Quadratic Programming (MIQCQP)
  • Second-Order Cone Programming (SOCP)
  • Mixed-Integer Second-Order Cone Programming (MISOCP)

And supports the following optimizers:

The coverage of supported problem types and optimizers can satisfy a large portion of needs for open energy modeling.

We have released PyOptInterface on PyPI for Windows, Linux and macOS. It can be installed via pip:

pip install pyoptinterface

PyOptInterface has no dependencies other than Python itself. However, to use it with a specific optimizer, you need to install the corresponding optimizer manually. The details can be found on the configurations of optimizers.

It can also be installed with pre-built HiGHS:

pip install pyoptinterface[highs]

This package is still new and any forms of feedback are welcome. We sincerely hope this package to be useful for researchers in open energy modeling field and accelerate your computational work flow.

If you find this package useful or interesting, please also give us a star on Github to support the following development.

Best regards.

9 Likes
2

Hello metab0t,

Thanks for sharing! Are you going to add linopy performances to the benchmark?
Benchmark - PyOptInterface documentation (metab0t.github.io)

Thanks in advance!

3

Thank you for your quesion. I am aware of linopy and its application in PyPSA.

The reason I haven’t included linopy in benchmark is that the facility location problem contains quadratic constraints, and it seems that linopy currently does not support it.

For lqcp problem, implementation with linopy is possible.

1 Like
4

0.2.0 version of PyOptInterface has been released.

It supports callback functions for Gurobi and COPT. We provide a brief introduction and 1:1 translation table for callback API at Callback - PyOptInterface documentation

It also supports model.write() to write the LP or MPS format of model to disk file.

1 Like
5

Just noting that the CPLEX LP and MPS text‑based formats are discussed here (noting that this information came up recently in another topic):

6

@thibaut.knibiehly
In this comment, I prepare a micro-benchmark between PyOptInterface and linopy, and the one-shot time is nearlly 1/4 - 1/3. PyOptInterface especially wins when you need to to modify and solve the model for many times.

7

Hello metab0t,
Thanks a lot for sharing. Your work is really interesting!