Do-a-thon: bringing CBC to the masses

Workshop
,
1

Proposal for: Do-a-thon
by Bryn Pickering and Stefan Pfenninger (ETHZ)

Session Title
Bringing CBC to the masses

Session Description
As will be made apparent in a connected lightning talk, the only viable open-source linear solver is CBC; GLPK is both too slow and inaccurate.

Installing CBC on OSX or Linux is straightforward, there are built packages for both on conda forge. However, many modellers still use Windows, and have to go through a relatively confusing process to get a CBC executable on their device.

We think CBC should be as easily accessible to Windows users as it is to everyone else (even though we actually only work on MacOS!). In this session, we’ll get down and dirty with the CoinCBC conda-forge recipe, with the aim of getting a Windows executable made available.

The intended outcome is simple: a pull request on the CoinCBC conda-forge feedstock respository containing the necessary code to make CBC easily available for Windows devices.

Would you like to be responsible for this Session?
Yes

Do you need any special infrastructure for this Session?
Lots of plug sockets, for all the laptops being hooked up.

Do you have any recommendations who could be part of this Session?
Those with software development experience, particularly with regards to creating conda-forge recipes in which an executable is compiled from c++ code.

2 Likes
2

I’d find this really useful!

3

Just to note the outcome of this do-a-thon: it’s a real pain. We weren’t successful, but here are the notes we took:

There is a small amount of documentation [1][2] for building on windows, but they depend on a cmake file being available.

Currently, the windows binaries are built [3] using ‘coinbrew’ [4].

Existing conda-forge c++ compilers include vs20XX [5], so one could theoretically run coinbrew in conda-forge to generate the executable.

Additionally, windows binaries depend on access to, which would need adding to the package manually:

  • libblas
  • liblapack
  • libgfortran-5
  • libgcc_s_seh-1
  • libbz2-1
  • libwinpthread-1
  • libstdc++-6
  • libquadmath-0

We don’t know if the dependency on coinbrew and access to these libraries would be possible through conda-forge. Having discussed the problem after the do-a-thon with Benjamin Fuchs at DLR, it seems like the easiest would be to point the windows build directly to the already built binaries [6]. We could then just unpack these and repackage it within conda-forge. Not sure if this fit with conda-forge ‘best practice’, however. It also raises the issue of how to build each of the compiler-specific binaries on conda-forge (win32-msvc9, x86_64-w64-mingw32, win32-msvc15, i686-w64-mingw32, win32-msvc14), or whether to just compile one for python2.7 conda environments (win32-msvc9) and one for python3 conda environments (win32-msvc14), in accordance with recommendations in [7].

We’d still welcome anyone with more knowledge on this subject (conda-forge recipes to compile c++ code on a Windows VM) to come forward. In the meantime I will look to implement the workaround, and see if the feedstock maintainers accept the method.

[1] https://conda-forge.readthedocs.io/en/latest/buildwin.html
[2] https://github.com/conda/conda-build/wiki/Windows-recipe-patterns
[3] https://github.com/coin-or/Cbc/blob/master/appveyor.yml
[4] https://raw.githubusercontent.com/coin-or/coinbrew/master/coinbrew
[5] https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/master/recipe/conda_build_config.yaml
[6] https://bintray.com/coin-or/download/Cbc/2.10.3
[7] section 6.1 @ https://readthedocs.org/projects/conda-forge/downloads/pdf/latest/

4

@brynpickering I guess you want a DLL with python bindings, right? Have you considered building on Linux but cross‑compiling for Windows. I’ve only done that for static linking using the GNU GCC compiler. Not sure how that works with shared objects and/or other compilers? Also GLPK used to be built for Windows and with multiple language bindings using SWIG — I don’t know the details but they will be documented or discussed somewhere, if only on the mailing list archive. Just thoughts. R.

5

Are you talking about compiling CBC under windows?

There is a visual studio project.
See this discussion : https://github.com/coin-or/Cbc/issues/252

6

Thanks for drawing my attention to it @SanPen. I’m not sure how automatable the VS project compilation would be, i.e. as part of a conda-forge recipe - do you have any ideas?

Also, belated thanks for your thoughts @robbie.morrison. I don’t doubt that cross compiling could be possible, I just wouldn’t have the first idea of how to go about doing it!

7

What is the goal here? distribute CBC as a python package?

PuLP has that figured out already. The solution is to ship the compiled DLL with the package. It has a version for x86 and x86-64 for windows, linux and OSX. I do the same in GridCal and it works.

Bear in mind that even if you manage the titanic task to compile CBC through a setup.py, the destination needs to be on windows path for other solvers to access it. Plus, you need all the dependencies, including BLAS and LAPACK (which are compiled with fortran) around.

8

Probably best to read the initial thread message!

In short, we’re looking to add a Windows recipe for CBC on conda-forge so that you don’t need to worry about compiling anything yourself; any project can just grab CBC on the fly using conda install -c conda-forge coincbc. This is currently possible with OSX and Linux, but not Windows.

9

I read the original message, but I still do not understand the end goal.

Do you want to make the .dll available as a package, how would then the users consume it?

Unless you want to make a C++ wrapper for CBC, for which you’ll need to provide a way to pass the matrices in CSC format and all the problem in a <= fashion, the default interface is using files. .LP files usually.

With windows, the issue is the lack of unified compiler for C++ and Fortran, hence all that depends on fortran compiled libraries is really hard to get working unless you have the intel compiler installed. For which you have to either spend 500€ or certify that you are going to use it for an open source project. But deep down it gets better, because if you want a statically compiled library (and you want that with CBC) the fortran and the C++ libraries need to be in consonance.

Therefore the PuLP solution is the easiest “for the masses”

10

To return to the earlier suggestion of cross‑compilation from Linux to some suitable generic target like i686-elf and ensuring the target dependencies (headers and libraries) are nil. More background here:

I sometimes cross‑compiled the modeling framework deeco from System V UNIX to Win32, but that was a long time back.

11

GCC is always a good option, however for statically compiled libraries python needs the visual studio compiler, because python itself is compiled with visual studio under windows. Hence my question about the end goal.

You may get way with this if you compile the library as a .dll (dynamic as opposed to .lib static) and you make sure that the usage of the library is through commands or by using a python-only wrapper with ctypes which is a experience I do not recommend.

12

In the context of https://open-plan-tool.org/ open_plan project we have to provide a user-friendly tool (with a GUI and not requiring programming skills) for modelling (small scale) energy systems. The solution I envisioned to “bring CBC to the masses” and ease setting up is to use docker (based on stakeholders meetings where the following needs for the tool were identified: cross platform, does not require extensive programming skills, able to use the tool with a browser (deployable on a local network), simple installation steps)

I want to create another post about a draft GUI for open source frameworks developed within the openmod community, I hope I will find time to do so before the end of the week :slight_smile: