I’m wondering how close (or far) we are to a truly standard, common data format (CDF) for describing power systems? The IEEE CDF is outdated and it looks like the raw PSS/E format is unofficially recognized as the “standard” data exchange format between commercial softwares.
I’ve found a lot of information on CDF’s for energy models (i.e.: Do-a-thon: Draft for 'energy' datapackage standard (?)), some for transient data (IEEE C37.111), but not much for the power system detailed description.
Is there already an open standard being discussed? Is there already an open CDF modders should aim to integrate to their applications to facilitate data exchange between modelling softwares? If so please let it not be XML…
For example I’m currently developping a software that extracts a network’s information from an excel file, builds a SQLite3 database (my custom format, tailored for my specific needs), and converts it to PyPSA, GridCal and PSS/E to either run simulations using these tools or simply exchange in what appears to be the current CDF in use (PSS/E). A truly standard open CDF would greatly simplify development and hopefully enforce sane practices (ex.: no storing of calculated data in the CDF). It would allow me to very easily add other open modelling softwares to the mix, without having to spend hours converting from one data format to the other.
Ideally the standard should cover both the data structure (ex.: buses, branches, transformers, generators) and the file format (ex.: JSON, SQLite3). That way each modelling software could develop a simply conversion utility between the CDF and their internal structure, required for their needs (ex.: python object for each element).
Hi @Miek I am not very familiar with data standards for power systems, but where does the common information model (CIM) for electric power systems fit? And did you look at the ontologies page on this site for clues? Also Medjroubi et al (2017)? HTH, R.
Thank you Robbie, I didn’t know about the IEC CIM. It does appear to be what I was looking for, except for not being open (and using XML for data exchange). I found this API for Python but it’s unfortunately no longer maintained and the link to the doc is dead: PyCIM
I rather liked this presentation from Hawaiian Electric and Open Grid Systems. They clearly demonstrate how a common data format can reduce conversion headache.
I’ll try to have a look at the IEC CIM standard if I can get my hands on it, but I think its price is going to be prohivitive for me.
Hi @Miek The Linux Foundation have been talking to ENTSO‑E about the use and development of open source software for grid operations. As that initiative proceeds, it will almost certainly be supplemented by the development of open standards for data interchange. This will be a multi‑year project though. Whether the accompanying data will also be open is another matter. I spoke to Mike Dolan, Linux Foundation some months back and data is not part of their current discussions.
To formalize this, the Linux Foundation launched LF ENERGY in July 2018 with French transmission system provider RTE. ENTSO-E hosted an introductory event in August 2018.
A promotional video on the LF ENERGY site talks about a “common grid model”. You could email Shuli Goodman, Executive Director, LF Energy Foundation for an update.
LF ENERGY follows similar flagship exercises by the Linux Foundation in the automotive and movie production areas.
To return to CIM, another other interesting thing is that it is recorded using UML. Perhaps that is a useful practice to continue. R.
Hi all- data is definitely on our radar. So central to interoperability and also such a burden on cost. If this is of interest please join the mailing list. We are moving towards convening the governing board and the technical advisory council. Glad to hear there is interest and visibility in LF Energy!
Hello @Shuli The Open Energy Modelling Initiative, who runs various venues including this forum, has no legal standing and no process for endorsing representation. But I would encourage those in our community who work on grid modeling (not me) to find someone to serve on the LF Energy technical advisory council. It would be rather useful if the necessarily detailed grid description that LF Energy develops for its purposes is also roughly compatible with the broader‑brush information that long‑haul energy modelers need. R.
Hi @robbie.morrison. Thank you for your encouragement. ALL open source projects are open, regardless of membership. Anyone can work on a project. Non-members may participate fully in the technical community and with contributions, earn committer, or TSC roles on merit.
As to “It would be rather useful if the necessarily detailed grid description that LF Energy develops for its purposes is also roughly compatible with the broader‑brush information that long‑haul energy modelers need.” If there is a data model project, it would be the contributors that would ensure compatibility and would hopefully build on current efforts. This is precisely why we will need your participation. Please signup here for the mailing list: https://lists.lfenergy.org/g/main/join
As far as I can tell, the powsybl project from the LF Energy has its own grid model format (IIDM at https://powsybl.github.io/docs/iidm/model/), but it’s an internal model, not a model standard that’s meant to be used for data exchange between modelling softwares. For grid model exchange, they mention other closed formats (CIM, CGMES, UCTE…).
It looks like CIM is the best standard we have for grid model exchange at the moment, even if it’s not open. A lot of open-source modelling softwares appear to support it, as well as a few of the commercial softwares I use. I’ll keep looking and report back if I find an open CIM equivalent.
Hi @Miek That all looks very sensible. I forwarded your concerns over pricing to the FSFE coordinator on open standards. A quick look at the IEC bookstore suggests that the CIM suite is circa USD $5000. For a broad take on standards, FOSS, and innovation, try Husovec (2018). HTH, R.
Also worth noting that OpenStreetMap has a tagging model for power system infrastructure. The OSM requirements though are more modest than that needed by energy system modelers, but it would be useful nonetheless if the two were somewhat compatible.
Last year I had the task of implementing CIM for the DMS (Distribution Management System) of my former company (Indra), So I can provide first hand info here.
CIM claims to be a standard. In my opinion it is an over engineered high level design. I am sure that the IT architects love it because it takes the object’s polymorphism to ridiculous extremes. But when I had to implement it,I faced a lot of possible variants that make the word standard fade away.
So, yes, CIM aims to be an standard but it will not be until someone imposes his “view” on the implementation of the whole mess. Because if there are 25 ways of telling the rating of a cable, there you have 25 standards. Also 25 ways of defining a rating mean that the 25 ways can coexist and collide in a CIM file, and that is definitely a design flaw. In the end it is up to the programmer to define what the standard actually is.
My company’s take on this is that if they have some flavor of CIM, it would be easier to adapt to the flavor of the customer. Is this a standard?
My experience from the industry is that PSS/e’s .raw format is the actual standard, as horrific and limited as it may be.
While PSS/E might provide a limited set of standards for specific transmission use-cases, the CIM model is much more open and accessible than the proprietary Siemens PSS/E model format. For example, can you provide a link to the model description, or better yet a machine readable, downloadable model. The PSS/E homepage has no reference to any open model - and hence would be rejected out of hand by any researcher worth his salt.
It doesn’t matter if PSS/e is closed source, it is the de-facto standard. That is why every single SCADA and commercial power system software has .raw import. In fact, open source models like Matpower and PSAT have .raw import as well. To support the file format that dominates the industry is useful if you work with customers or expect a TSO/DSO to provide data to you (even as a researcher).
I develop a program called GridCal and it supports .RAW, matpower and CIM (my interpretation of CIM anyway) among other file formats because while working and researching I find those files and I need to read them.
The situation with CIM is funny because every one claims it is the standard but no one has made an implementation for their program. To support a general version of CIM is probably as time consuming as to support the open source program itself. Some time ago I proposed some open source modellers to come up with a file format to exchange information among our programs. The overall reply was that CIM was there for that, yet I have not seen them implement CIM, which I did.
CIM is provided as a file for Enterprise Architect which is a closed source program that you can use in a 30-days demo or purchase the license. That closes the loop beautifully.
So, to sum up, I am open to discuss a common interpretation of the CIM file format. I actually welcome everyone willing to copy my code into theirs if it makes CIM a used standard.
Well, there is the PSS/e manual which provides a guide on how to read .raw files. To get it you may simply download the PSS/e 50-node demo and see the manual there.
Don’t get me wrong, .raw files are horrible, but once you have a parser to import raw version 29, you have a parser for all the .raw files version 29 that you are going to find out there. That is not the case with CIM.
As nice as XML would be, the CIM “standard” itself is not nearly as clear on how to interprete the data as the .raw file. This is because the polymorphism in CIM makes it incredibly hard to determine unique ways of defining devices. In .raw there are a couple of ways of defining a line, and you know all of them from the manual. In CIM, there are plenty of ways and all maybe concurrent. Observe this:
I have no direct experience with CIM but it is specified using UML. Please note that inheritance relationships cannot be retained when translating a data model from a UML class diagram to an XML schema (but the opposite should work). For specific instances of the data model, this restriction should not be a problem as the inheritance graph should no longer be material?
In this particular case, you see that the inheritance properties define a line in more than one way.
For example, nothing prevents to find in the same XML instances of ACLineSegment (defining the positive and zero sequence line parameters of the pi model), and instances of PerLengthImpedance and the same with all the parents.
The solution in the past has been to choose a single way of defining a line of all the parents and children that can define a line and have the exports write that one object only.
It is alright to have a hierarchical structure, but in practice one should choose a couple of practical implementations of what a line is. The one that resembles the most to what utilities have in their DB is ACLineSegment.
Of course it would be possible to support all the combinations, but I tell you that the C++ program (with the headers generated by Enterprise Architect) for this goes well over the 2 GB. And to maintain that is hard core. On the other hand a reasonable implementation where only a reasonable subset of objects is allowed is no more than a hundred kBytes. I know this by experience.
The ACLineSegment instances in the XML can have the pi model parameters and/or have a referenced PerLengthImpedance (0…1) - as you say. If there are both, the ACLineSegment parameters should be the per length impedance multiplied by the length. Redundant, but allowed.
There should be no (super class) Conductor instances - it has only a length attribute.
Similarly for ConductingEquipment, etc.
As a general rule only leaf nodes of the inheritance tree are instantiated.
