Remedy 1: Design and implementation of unified metadata format under a common data model
Metadata is an increasingly essential tool enabling large-scale, distributed management of resources. Recent years have seen a growth in interaction between previously relatively isolated communities across observing domains and techniques, driven by a need for interdisciplinary research and understanding. However, metadata standards have not been able to meet the needs of interoperability between these to date largely independent communities and networks. Observations without metadata are of very limited use: it is only when accompanied by adequate metadata (data describing the data) that the full potential of the observations can be realized. Format conversions always bring with them the danger of destroying information in the process, in particular in the accompanying metadata, which usually receives less attention.
Several efforts have been undertaken to improve the harmonization of metadata across numerous networks and international programs, but this is still not sufficient. Harmonization effort in the atmospheric science community is starting to be addressed by the emerging WIGOS standards, currently under development and subsequent implementation at the WMO, and by the ESA Climate Change Initiative (CCI), amongst others. Copernicus Climate Change Service Data Store activities are also highly relevant to this gap. There are also challenges that arise due to interoperability across observational domains (surface, atmospheric, oceanic, terrestrial etc.).
GAIA-CLIM metadata standards and format harmonization have been carried out with aim to provide a model for facilitating the users’ access and the usability in-situ data. This exercise included the establishment and documentation of common metadata and data formats for a selected subset of networks that will contribute to the Virtual Observatory. The Virtual Observatory facility shall also support the remedy of this gap by providing data format conversion for various input data and a data extraction function that makes the outputs available in user friendly formats.
GAIA-CLIM activities will be followed up by the Copernicus Climate Change Service, where, for a selected number of networks reviewed within GAIA-CLIM, the harmonization of the data and metadata format and structure is ongoing. According to the requirements provided by the Copernicus end-users through the C3S Sectoral Information System (SIS) projects, this effort involves the implementation of a common data model compliant with the ECWMF Observational DataBase (ODB) and a data-management facility, which shall become part of the operational C3S services at the end of the above-mentioned contract.
G1.03 and G1.04 should be addressed after G1.06
The resolution of G1.06 would bring invaluable benefits to support the resolution of G1.03 and G1.04 by facilitating the review of existing capabilities, starting from rich standardized information and enabling classification of measurement maturity in a more accurate way.
G5.01 should be addressed with this gap
Metadata harmonization across multiple data provides will also positively impact on the interoperability among different data repositories with clear benefits for addressing gap G5.01
The need for extensive and accurate metadata is ever increasing in both research and operations, enabling large-scale, distributed management of resources. Recent years have seen a growth in interaction between previously relatively isolated communities, driven by a need for cross-domain collaboration and exchange of data and products. However, metadata standards have generally not been able to meet the needs of interoperability between independent standardization communities. Observations without useable metadata are of very limited use as the metadata provides key context such as the time, location, and modality of the measurements. Several efforts have been undertaken to improve the harmonization of metadata across the networks and international programs, but currently this is still insufficient.