I am looking for projects related to quantitative or qualitative evaluation of models / frameworks (evaluation results in form of publications, presentation of methodologies etc.). These do not necessarily need deal with open source models. Do you know of such projects and/or publications. I did some research on this which was not very fruitful (expect of categorizing reviews). Now I am wondering if there is not much out there or if I am just bad in finding it.
The Wikipedia page on energy modeling contains a number of references that might be of interest.
Finally, Lunz et al (2016) could be worth a look, but focuses more on defining scenarios than comparing models (contact me if you want a copy).
References
Lunz, Benedikt, Philipp Stöcker, Sascha Eckstein, Arjuna Nebel, Sascha Samadi, Berit Erlach, Manfred Fischedick, Peter Elsner, and Dirk Uwe Sauer. (1 June 2016). “Scenario-based comparative assessment of potential future electricity systems: a new methodological approach using Germany in 2050 as an example”. Applied Energy. 171: 555–580. ISSN 0306-2619. doi:10.1016/j.apenergy.2016.03.087.
I am also interested in model evaluation. From the naive perspective of somebody who does not come from a strong background on ESM, I am usually confused with some of the arguments used in ESM papers, particularly with respect to prediction capabilities. Many studies refrain from saying their models are predictive, but if they are not -or at least aspire to be- (and not normative either), I am not entirely sure how results should be analyzed “in the search for insights”.
One can distinguish between forecasts, scenarios, and predictions. When the first capacity expansion energy system models were developed for mainframes in the 1970s using linear programming techniques (the EMF was founded in 1976), they tended to be described as providing forecasts — ofttimes under less‑than-transparent assumptions. Later, scenario analysis was adopted using scenarios to define each model run, with one scenario selected to represent public policy inaction and labeled business-as‑usual or BAU. The remaining scenarios were then compared against BAU by difference. The IPCC assessment reports from about 2000 onward used this technique, also fleshing out each scenario with so‑called storylines on how each scenario might come to pass (technological optimism, sufficient consumption, high growth, and so on). This switch in context was controversial at the time, but can be seen as the transition between forecasting and scenario analysis. Nowadays scenario analysis is standard. Notwithstanding, the over‑interpretation of results remains an issue. Most energy models are deterministic, meaning they will produce the exact same output with the same input — and in that very localized sense, each model run creates a future prediction. How one decides to interpret that prediction are where things get interesting! More background on energy system modeling on wikipedia.
Thanks Robbie for the reply. For an outsider “scenario” (even under internally consistent assumptions in a coherent storyline) is excessively vague. There are infinite futures that do not break laws of thermodynamics. Why study ones and not others if its not for normative reasons or because we think we have an idea of how system is likely to behave?
Settling upon a definition for the concept of “scenario” is difficult. The Open Energy Ontology is going to tackle that question shortly when it begins community engagement in the coming months. At least one member of the steering committee believes the various scenario concepts should be given arbitrary labels and the term “scenario” should be avoided completely. Ditto for the term “model”.
I agree any particular preferred future view (trying here to avoid the word scenario) naturally bundles values, attitudes to risk, opinions regarding future circumstances, and many other influences. That is why IÂ regularly argue that diverse stakeholders should be involved in articulating storylines and scenarios and that task should not be left to modelers alone.
More on the theme of diversity in the context of modeling:
DeCarolis, Joseph F, Paulina Jaramillo, Jeremiah X Johnson, David L McCollum, Evelina Trutnevyte, David C Daniels, Gökçe Akın-Olçum, Joule Bergerson, Soolyeon Cho, Joon-Ho Choi, Michael T Craig, Anderson R de Queiroz, Hadi Eshraghi, Christopher S Galik, Timothy G Gutowski, Karl R Haapala, Bri-Mathias Hodge, Simi Hoque, Jesse D Jenkins, Alan Jenn, Daniel JA Johansson, Noah Kaufman, Juha Kiviluoma, Zhenhong Lin, Heather L MacLean, Eric Masanet, Mohammad S Masnadi, Colin A McMillan, Destenie S Nock, Neha Patankar, Dalia Patino-Echeverri, Greg Schivley, Sauleh Siddiqui, Amanda D Smith, Aranya Venkatesh, Gernot Wagner, Sonia Yeh, and Yuyu Zhou (16 December 2020). “Leveraging open-source tools for collaborative macro-energy system modeling efforts”. Joule. 4 (12): 2523–2526. ISSN 2542-4785. doi:10.1016/j.joule.2020.11.002.