This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 640276.
G5.07 Incomplete development and/or application and/or documentation of an unbroken traceability chain of data manipulations for atmospheric ECV validation systems
Submitted by Anna Mikalsen on 12 June, 2017 - 15:21
Gap abstract:
Recently established quality assurance and validation guidelines and systems are not sufficiently well recognised or understood in the global community, where validation purposes, methodologies, and results can differ significantly from one report to another. Harmonised practices should now be advertised and applied more universally across the community to avoid (1) missing quality indicators, (2) incoherent results between different validation exercises, and (3) unreliable results or additional methodological uncertainties due to sub-optimal data manipulations. Moreover, there is room for further improvement in validation methodologies, taking advantage of the ever-increasing breadth of measurement, modelling, and data analysis techniques.
Part I Gap description
Primary gap type:
- Technical (missing tools, formats etc.)
Secondary gap type:
- Uncertainty in relation to comparator measures
- Governance (missing documentation, cooperation etc.)
ECVs impacted:
- Temperature,Water vapour, Ozone, Aerosols, Carbon Dioxide, Methane
User category/Application area impacted:
- Operational services and service development (meteorological services, environmental services, Copernicus Climate Change Service (C3S) and Atmospheric Monitoring Service (CAMS), operational data assimilation development, etc.)
- International (collaborative) frameworks and bodies (space agencies, EU institutions, WMO programmes/frameworks etc.)
Non-satellite instrument techniques involved:
- Independent of instrument technique
The tools to be developed to address G5.06 in the context of validation work should be based on the traceability principles and Cal/Val best practices referred to in G5.07. In this sense, G5.06 should be addressed first, as it represents a contribution to the remedy for G5.07 (see G5.07 gap remedy #1).
Detailed description:
In the context of sustainable Earth Observation data services, such as those in development for the Copernicus Climate Change Service (C3S) and Atmospheric Monitoring Service (CAMS), Quality Assurance (QA) and geophysical validation play a key role in enabling users to assess the fitness of available data sets for their purpose. User requirements, e.g., those formulated for the Global Climate Observing System (GCOS), have to be identified and translated into QA and validation requirements. In turn, QA and validation results must be formulated in the form of appropriate Quality Indicators (QI) to check and document the compliance of the data with the user requirements. Metrology practices recommend the development and implementation of traceable end-to-end QA chains, based on Système International d’Unités (SI) and community-agreed standards (as identified for instance in the GEO-CEOS QA4EO framework).
Generic guidelines for such QA systems applicable virtually to all atmospheric and land ECVs are being developed within the EU FP7 QA4ECV project (2014-2018), while more specific guidelines developed in projects like ESA’s CCI and dedicated to atmospheric ECVs are being published (e.g., Keppens et al., 2015a). Generic and specific QA systems and guidelines established in those recent projects are not sufficiently well recognized or understood in the global community, where validation purposes, methodologies, and results can differ significantly from one report to another. Harmonized practices should now be advertised and applied more universally across the community.
The impacts of not adopting a traceable end-to-end validation approach are diverse. Firstly, important quality indicators may be missing in the analysis, e.g. information on spatio-temporal coverage, resolution, dependences of the data quality on particular physical parameters (e.g. solar zenith angle, cloud cover, thermal contrast) etc. Secondly, results may be incoherent between several validation exercises on the same data set and the origin of the discrepancies unclear due to insufficient traceability. Thirdly, methodological uncertainties in, e.g., geographical mapping, in the use of vertically averaging kernels, or in unit conversions using auxiliary data, may lead to unreliable results. Finally, all this may imply sub-optimal use of the true validation capabilities of the ground-based reference network
Operational space missions or space instruments impacted:
- Independent of specific space mission or space instruments
Validation aspects addressed:
- Radiance (Level 1 product)
- Geophysical product (Level 2 product)
- Gridded product (Level 3)
- Assimilated product (Level 4)
- Time series and trends
- Representativity (spatial, temporal)
- Calibration (relative, absolute)
- Spectroscopy
- Auxiliary parameters (clouds, lightpath, surface albedo, emissivity)
Gap status after GAIA-CLIM:
- GAIA-CLIM explored and demonstrated potential solutions to close this gap in the future
The GAIA-CLIM project adds to other EU projects with respect to more ECVs and disseminates results via the "Virtual Observatory" facility but does not close the gap.
Part II Benefits to resolution and risks to non-resolution
| Identified benefit | User category/Application area benefitted | Probability of benefit being realised | Impacts |
|---|---|---|---|
Completeness of the QA and validation reports, addressing all Quality Indicators relevant for the envisaged use. |
|
| Users will have access to more and better information on which to judge the fitness-for-purpose of a particular product for their application |
Homogeneity in adopted Quality Indicators and processing chains allows intercomparison of different validation studies and their results. |
|
| Users can easily compare different products based on their performance in validation exercises that were performed along the same principles and with comparable metrics. |
Improved reliability and minimal methodological uncertainties related to the Cal/Val processing chain. |
|
| Optimal use of the reference data to gauge the quality of the satellite data sets, without unnecessary additional methodological uncertainties; Improved feedback on satellite data production, with greater detail and differentiation. |
| Identified risk | User category/Application area at risk | Probability of risk being realised | Impacts |
|---|---|---|---|
Difficulty to judge the fitness-for-purpose of satellite data products because of missing or poorly-defined Quality Indicators. |
|
| Users of satellite data products may refrain from using these products when they are not sufficiently characterised. This constitutes sub-optimal use of the EO system and may lead to non-realised performance of the services. |
Difficulty to compare different validation exercises, e.g. of different products for a particular ECV. |
|
| Users are often faced with the question “which is the best data set for my application?”. Without comparable validation methods and Quality Indicators applied to all candidate data sets, no reliable, informed choice can be made. This leads to sub-optimal use of the EO system and impacts negatively the application(s) envisaged by the user. |
Part III Gap remedies
Gap remedies:
Remedy 1: Propagation and adoption of metrological best practices in sustained validation activities
Primary gap remedy type:
Governance
Secondary gap remedy type:
Technical
Research
Education/Training
Proposed remedy description:
The remedy proposed here consists in the composition of expert consortia under the umbrella of (and potentially with funding by) overarching bodies and initiatives (WMO, EC, space agencies). These consortia should look into the following highly related aspects of the gap:
- The development of (new) best-practice validation protocols and the corresponding documentation framework;
- The application of these protocols and guidelines in (operational) validation platforms;
- The advertising (including peer-reviewed papers, handbooks, training and courses) to validation teams and service providers.
Some efforts are already ongoing in this direction, for instance in the EC FP7 project QA4ECV (definition of a traceable validation chain and application in the “Atmosphere Validation Server” for a few ECVs), in ESA’s CCI, and in ad-hoc initiatives such as the recent ISSI team “EO validation across scales” (which included GAIA-CLIM and CEOS representatives). Still, these only partially address the gap, and a much wider effort (in terms of ECVs, methods, platforms, and outreach) is required to extend, implement, and operationalise these QA4EO-compliant practices.
Relevance:
The integrated concept of the proposed remedy (including research, technical developments, education, and governance) ensures that the gap is broadly addressed. For optimal acceptance by the scientific community and the major stakeholders, the composition of the expert teams is key.
Measurable outcome of success:
Published protocols and guidelines, endorsed by the large stakeholders, and referred to in the scientific literature. Implementation of these protocols in the validation platforms supported by the space agencies, the Copernicus programme, etc.
Expected viability for the outcome of success:
- Medium
- High
Scale of work:
- Programmatic multi-year, multi-institution activity
Time bound to remedy:
- Less than 5 years
Indicative cost estimate (investment):
- Medium cost (< 5 million)
Potential actors:
- EU H2020 funding
- Copernicus funding
- WMO
- ESA, EUMETSAT or other space agency