This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 640276.
G3.06 Missing comparison (validation) uncertainty budget decomposition including uncertainty due to sampling and smoothing differences
A data validation study is meant to check the consistency of a given dataset with respect to a reference dataset within their reported uncertainties. As such, the uncertainty budget of the data comparison is crucial. Besides the measurement uncertainties on both data sets, the discrepancy between the two datasets will be increased by uncertainties associated with data harmonization manipulations (e.g. unit conversions requiring auxiliary data, interpolations for altitude regridding) and with co-location mismatch, i.e. differences in sampling and smoothing of the structured and variable atmospheric field. In particular, the latter term is hard to quantify and often missing in validation studies, resulting in incomplete uncertainty budgets and improper consistency checks.
Part I Gap description
- Uncertainty in relation to comparator
- Knowledge of uncertainty budget and calibration
- Governance (missing documentation, cooperation etc.)
- Temperature,Water vapour, Ozone, Aerosols, Carbon Dioxide, Methane
- Operational services and service development (meteorological services, environmental services, Copernicus Climate Change Service (C3S) and Atmospheric Monitoring Service (CAMS), operational data assimilation development, etc.)
- International (collaborative) frameworks and bodies (space agencies, EU institutions, WMO programmes/frameworks etc.)
- Climate research (research groups working on development, validation and improvement of ECV Climate Data Records)
- Radiosonde
- Ozonesonde
- Lidar
- FPH/CFH
- Microwave Radiometer
- FTIR
- Brewer/Dobson
- UV/VIS zenith DOAS
- UV/VIS MAXDOAS
- Pandora
- GNSS-PW
G3.01. To be addressed before G3.06
Argument: To quantify the additional errors and uncertainties in a comparison due to co-location mismatch, it is advantageous to have external information on the atmospheric variability on the scale of the co-location mismatch.
G3.04. To be addressed before G3.06
Argument: To quantify the additional errors and uncertainties in a comparison due to co-location mismatch, it is important to know the smoothing and sampling properties of the individual instruments
Ideally, every validation study based on comparisons with ground-based reference data should investigate whether the comparison statistics (bias or mean difference, spread on the differences, drift, etc.) are compatible with the reported random and systematic measurement uncertainties, while taking into account the additional uncertainties due to spatiotemporal sampling and smoothing differences, i.e. non-perfect co-location of the airmasses sensed by both instruments. Indeed, it is only in a few particular cases possible to adopt co-location criteria that result in a sufficiently large number of co-located pairs, while at the same time keeping the impact of atmospheric variability on the comparisons (due to spatiotemporal mismatches) well below the measurement uncertainties. In all other cases, the discrepancy between two data sets will contain non-negligible terms arising from sampling and smoothing differences, which need to be taken into account. In fact, such an analysis is essential to fully assess the data quality and its fitness-for-purpose, but in practice, it is rarely performed, as this co-location mismatch is hard to quantify reliably. Some pioneering work was published by Cortesi et al. (2007) on uncertainty budget closure for MIPAS/ENVISAT ozone profile validation, by Ridolfi et al. (2007) for the case of MIPAS/ENVISAT temperature profiles validation, by Fasso et al. (2013) in the context of radiosonde intercomparisons, by Lambert et al. (2012) on water vapour comparisons, and by Verhoelst et al. (2015) for GOME-2/MetOp-A total ozone column validation. However, no such studies have hitherto been performed for most other ECVs and/or instruments. This gap therefore concerns the need for (1) further research dealing with methods to quantify co-location mismatch, and (2) governance initiatives to include in the common practices among validation teams dedicated efforts to construct full uncertainty budgets, and use these in the consistency checks.
- Independent of specific space mission or space instruments
- Radiance (Level 1 product)
- Geophysical product (Level 2 product)
- Gridded product (Level 3)
- Assimilated product (Level 4)
- Time series and trends
- Representativity (spatial, temporal)
- Calibration (relative, absolute)
- Spectroscopy
- Auxiliary parameters (clouds, lightpath, surface albedo, emissivity)
- GAIA-CLIM explored and demonstrated potential solutions to close this gap in the future
Dedicated studies within GAIA-CLIM aimed for full error (or uncertainty) budget decomposition for representative comparison exercises, involving all non-satellite measurement types targeted by GAIA-CLIM and several current satellite sounders. Moreover, some of these results were transferred into the Virtual Observatory to allow end users to also decompose the uncertainty budget of their comparisons. Nevertheless, further work is required to quantify comparison error budgets in many cases, to operationalise comparison error budget calculations in operational satellite validation and production of higher level services, and to increase awareness in the community of the need for comparison error budget closure.
Part II Benefits to resolution and risks to non-resolution
Identified benefit | User category/Application area benefitted | Probability of benefit being realised | Impacts |
---|---|---|---|
Improved feedback on data quality from the validation work, including on the reported uncertainties. |
|
| Optimized use of the data, avoiding over-interpretation but potentially also allowing greater detail to be extracted. |
Tighter constraints from validation work support product development |
|
| Shortcomings in products are more easily identified, driving further development and ultimately ensuring better, more reliable data products. |
Identified risk | User category/Application area at risk | Probability of risk being realised | Impacts |
---|---|---|---|
Incomplete –or even incorrect- feedback from a validation exercise on the data quality. |
|
| Poorly quantified data quality, affecting all use types. Sub-optimal feedback to data providers slows product development. The potential of the EO system is not fully realized. |
Part III Gap remedies
Remedy 1: Use of Observing System Simulation Experiments (OSSEs)
This remedy concerns Observing System Simulation Experiments (OSSEs), such as those performed with the OSSSMOSE system by Verhoelst et al. (2015) on total ozone column comparisons. These are based on a quantification of the atmospheric field and its variability (c.f. gap G3.01), e.g. in the shape of reanalysis fields, and on a detailed description of the sampling and smoothing properties of the instruments that are being compared (c.f. gap G3.04). The aim is to calculate the error due to spatiotemporal mismatch for each comparison pair, and to derive the mismatch uncertainties from these, so that they can be added to the measurement uncertainties to derive the full uncertainty budget.
The technological and organizational challenges are mostly related to the underlying gaps G3.01 and G3.04. When these are properly addressed, the calculation of the full uncertainty budget of a comparison exercise requires only a low investment in time and resources. Integrating this into an operational validation context does constitute an additional challenge requiring dedicated effort and funding.
This remedy addresses directly the gap.
At a high level, success is achieved when validation (and other comparison) results are published including a full uncertainty budget decomposition, taking into account spatiotemporal mismatch uncertainties. Or when they include a convincing demonstration that mismatch uncertainties are well below the measurement uncertainties and are negligible.
At a lower level, success is achieved if the OSSE allows one to close the uncertainty budget, i.e. the measured differences (or their statistics) are compatible with the sum of all uncertainty sources. Note that this requires reliable measurement uncertainties as well.
- High
- Single institution
- Consortium
- Less than 3 years
- Medium cost (< 5 million)
- No
- EU H2020 funding
- Copernicus funding
- National funding agencies
- National Meteorological Services
- ESA, EUMETSAT or other space agency
- Academia, individual research institutes
Remedy 2: Statistical estimation of typical co-location mismatch effects
An alternative to estimating co-location mismatch (the main missing term in the uncertainty budget decomposition of a comparison) from model simulations, is to employ statistical modelling on the differences, for instance with a heteroskedastic functional regression approach, (as implemented for instance in the STAT4COLL software package). In certain applications, this approach also allows one to disentangle measurement uncertainties from co-location mismatch, at least for the random components. GAIA-CLIM will have employed such an approach for a subset of specific cases (spatial domains and ECVs / measurement techniques). Further efforts are required to generalise the approach and tools to enable broader exploitation, including integration into an operational validation context.
Employ statistical modelling on the differences, for instance with a heteroskedastic functional regression approach. Efforts are required to generalise the GAIA-CLIM approach and tools to enable broader exploitation.
At a high level, success is achieved when validation (and other comparison) results are published including a full uncertainty budget decomposition, taking into account spatiotemporal mismatch uncertainties. Or when they include a convincing demonstration that mismatch uncertainties are well below the measurement uncertainties and are therefore negligible.
At a lower level, success is achieved if the statistical modelling allows one to close the uncertainty budget, i.e. the measured differences (or their statistics) are compatible with the sum of all uncertainty sources. Note that this requires reliable measurement uncertainties as well.
- High
- Single institution
- Consortium
- Less than 3 years
- Medium cost (< 5 million)
- No
- EU H2020 funding
- Copernicus funding
- National funding agencies
- National Meteorological Services
- ESA, EUMETSAT or other space agency
- Academia, individual research institutes
-
Cortesi et al., “Geophysical validation of MIPAS-ENVISAT operational ozone data”, ACP v7, 2007
-
Fassò et al., “Statistical modelling of collocation uncertainty in atmospheric thermodynamic profiles”, AMT v7, 2014
-
Lambert, J.-C., et al., “Comparing and merging water vapour observations: A multi-dimensional perspective on smoothing and sampling issues”, in “Monitoring Atmospheric Water Vapour: Ground-Based Remote Sensing and In-situ Methods”, N. Kämpfer (Ed.), ISSI Scientific Report Series, Vol. 10, Edition 1, 326 pp., ISBN: 978-1-4614-3908-0, DOI 10.1007/978-1-4614-3909-7_2, © Springer New York 2012
-
Ridolfi et al., “Geophysical validation of temperature retrieved by the ESA processor from MIPAS/ENVISAT atmospheric limb-emission measurements”, ACP v7, 2007
-
Verhoelst et al., “Metrology of ground-based satellite validation: Co-location mismatch and smoothing issues of total ozone comparisons”, AMT v8, 2015