Do-a-thon: Draft for 'energy' datapackage standard (?)

Workshop
, , ,
9

Please use the wiki page:

https://wiki.openmod-initiative.org/wiki/Common_Data_Format_Energy

to upload files for now. Hopefully we will have a better structure in the next days…

10

for IAMC format:
IAMC.ods (14.6 KB)

11

The “criteria group” has found that the design philosophy of the frictionless data specifications already matches our “generic” data-related criteria and could be a good/mandatory starter. Moreover, we agreed that there is already a lot of literature about scenario factsheets, etc. and we should gather all information in a new wiki page or add it to an existing one. This might help in the process of finding a common standard.

12

List of information, papers and standards of criteria :

13

datapackage_in_OEP_metadata_v1.3.txt (1.5 KB)

14

Should we have an other time slot on Thursday on this topic? What time slot would be best?

15

see zip file from datapackage group below…

16

I think we should have another slot, also for looking at the next steps (Wiki-page, etc) …

17

I added the datapackage from yesterday:

datapackage.zip (1.3 KB)

18

More information on the IAMC timeseries data standard can be found here: data.ene.iiasa.ac.at/databases

This standard has been used, amongst others, for the integrated-assessment scenario database compiled by IPCC Fifth Assessment Report (AR5), Working Group 3, see tntcat.iiasa.ac.at/AR5DB, and will also be used for the next IPCC reports.

19

We now have a slot from 14:00 - 15:30 to continue on this one. We meet at Room D44 as listed in the schedule for today.

1 Like
20

Here is the netcdf4 version (jupyter notebook and datapackage as netcdf file) from yesterday.

netcdf4_datapackage.zip (2.8 KB)

21

I would propose to use the follow up meeting at 2 pm to form a kind of “Openmod Working Group” on the topic of this common data format. If the proposal is accepted we should find a way to organize the group in terms of infrastructure, governance etc…

See this post for a general discussion on “Openmod Working Groups”:

22

I uploaded the netCDF example that uses xarray to bitbucket.

23

There is an openmod GitHub repo:

24

for accepting the github invitation:
https://github.com/openmod-initiative/common-energy-data-format/invitations

25

Hello openmod,

we were working on the metadata topic and after including tons of feedback, wishes and requirements we have a multi-purpose metadata datapackage pre-release-version (v1.4-pre) as JSON.

The next step is to create a DCAT-AP conform version that can be used with the ontology and a translator that converts between the easy JSON and the not-so-easy RDF-TTL.
Is anybody familiar with DCAT-AP and is able to support?
Feel free to comment and test the pre-version and I hope this is a next step to create and use an universal ‘energy datapackage’.

26

@ludwig.huelk Great work. I particularly agree with the separation of contributors and copyright notices. A couple of overarching points though:

  • A set of definitions might be useful? As an example: a contributor is the person who originally collected, measured, or observed the data and/or subsequently curated the data.

  • Are the individual feedback comments and responses available somewhere? It would seem sensible to record these so that those not directly involved can gain insight into the design decisions being made.

  • How does this align with similar work by the OPSD team? Are they in agreement with this v1.4 release?

I cannot help with the DCAT stuff though. But good luck anyway.

1 Like
27

Thanks for the feedback!

  • There will be metadata web interface (GUI) with detailed descriptions and examples.
  • The contributions were discussed. From my perspective, it is not possible to differentiate between creation and modification of a data set. At what point do you draw the line? The contributions section is more a changelog of the data and metadata. So we added the object (data/metadata) to indicate if a contribution was made in the data itself (e.g., add a new column) or a major change in the metadata (e.g., grant an open license).
  • The development was done in the OEP-GitHub.
  • The DCAT mapping is in the Wiki.
  • I’m in contact with the OPSD/OSE team. We used their metadata recommendations.

This is the changelog from v1.3 to v1.4:

  • add datapackage name dp
  • add unique id dp
  • specify language bcp47
  • add publication_date OSE discussion
  • add section context discussion
    ** add homepage
    ** add documentation
    ** add source_code
    ** add contact (person)
    This section is new and should improve the link to a the code/model/project that generated the data. In the last version (v1.3) this was stored as different kind of sources. I would like your feedback and ideas on this!
  • add object (data/metadata) in contributors section
  • add path and profile (tabular-data-resource) in section resources
  • add a _metadata_license (CC0)
  • Include _comments like formats
  • Minor changes in order to be valid datapackage. There is a Jupyter Notebook to test stings.
28

After some more feedback and minor changes we are happy to release the metadata version v1.4.