Hi;
I’m trying to understand how to implement a checkpoint in snakemake to handle infeasible solutions in a workflow. This could be relevant to many of us using snakemake and workflows in our work, and perhaps this problem is solved already?
When solving an optimisation problem using a solver, there are three main outcomes:
- a successful solve producing a .sol file
- an infeasible solve (or time out), no output produced
- an infeasible solve, but an analysis file is produced e.g. a .ILP from Gurobi or ISS from CPLEX
These three alternatives create a problem in a workflow, as outcomes 2 and 3 cause the entire workflow to fail. But a desirable outcome would be for the workflow to continue for successful solves (creating plots etc.) and another branch of tasks could set to work on providing useful debugging information on the failed solves.
One solution seems to be to use a snakemake checkpoint, which recomputes the workflow upon the completion of the solve step. The documentation (and concept) is not straightforward. Could anyone help me understand how to go about implementing this?
2 Likes
Hi @willu47 ,
I haven’t used checkpoints with Snakemake, but to address your underlying intention:
a.) Finished and final workflows/models should not fail. My spontaneous reaction on what you’re trying to do is: That’s a bad idea. If I understood your intention correctly, you’re desigining your workflow to be robust against failing. I have been at the same point with my models before and decided against coding a “failure” workflow into my workflow for this reason.
What I am doing instead and what I deem the better approach:
b.) Use the -k / --keep-going option. Snakemake will continue to run all rules which are not connected to the failing rule. I think that’s your desired behaviour. Have your rule which solves the network inform you of a failed run, print additional information on why the solver is failing to a log file. Using -kis an intentional decision by the user calling snakemake, accepting the case that the workflow may fail (immature workflow). After finishing development when you deem the model(s)/workflow mature and stable, you leave out the -k option and run the workflow without any additional changes required.
HTH.
Best,
Johannes
2 Likes
Hi Johannes;
Thanks, this is exactly what I was looking for - I was unaware of the -k “keep going” flag.
For a) not quite - I don’t want the workflow to hide failures, quite the contrary, I would like better debugging information to be passed through and for the workflow to keep going.
One use case is:
run a thousand model runs to conduct a global sensitivity analysis. A number of those runs will definitely be infeasible, and I want some rules to run after the infeasible solutions to highlight which parameter values causes infeasible solutions.
In this case infeasible solves are a valid outcome from the workflow.
Another use case is:
I add a new scenario to my workflow of multiple scenarios. The new scenario is infeasible and I want to debug it.
In this case an infeasible solution is not a valid outcome.
Hi Will,
Nice to hear that -k already helps you.
What I’d to for the sensitivity case (you want your workflow to complete without errors?):
Store the infeasible model anyways under the snakemake.output (the rule will not fail, the feasible/infeasible information is either stored with the model, noted in the log file or there is an additional indicator file indicating feasibiility/infeasibility of the model.
My opinion on this is: A rule should succeed if the results in the rule are within the expected results. Rule failure is reserved for unexpected or unhandled behaviour.