G3.06 Missing comparison (validation) uncertainty budget decomposition including uncertainty due to sampling and smoothing differences

Submitted by Anna Mikalsen on 12 June, 2017 - 12:13
Gap abstract: 

A data validation study is meant to check the consistency of a given dataset with respect to a reference dataset within their reported uncertainties. As such, the uncertainty budget of the data comparison is crucial. Besides the measurement uncertainties on both data sets, the discrepancy between the two datasets will be increased by uncertainties associated with data harmonization manipulations (e.g. unit conversions requiring auxiliary data, interpolations for altitude regridding) and with co-location mismatch, i.e. differences in sampling and smoothing of the structured and variable atmospheric field. In particular, the latter term is hard to quantify and often missing in validation studies, resulting in incomplete uncertainty budgets and improper consistency checks.

Part I Gap description

Primary gap type: 
  • Uncertainty in relation to comparator
Secondary gap type: 
  • Knowledge of uncertainty budget and calibration
  • Governance (missing documentation, cooperation etc.)
ECVs impacted: 
  • Temperature,Water vapour, Ozone, Aerosols, Carbon Dioxide, Methane
User category/Application area impacted: 
  • Operational services and service development (meteorological services, environmental services, Copernicus Climate Change Service (C3S) and Atmospheric Monitoring Service (CAMS), operational data assimilation development, etc.)
  • International (collaborative) frameworks and bodies (space agencies, EU institutions, WMO programmes/frameworks etc.)
  • Climate research (research groups working on development, validation and improvement of ECV Climate Data Records)
Non-satellite instrument techniques involved: 
  • Radiosonde
  • Ozonesonde
  • Lidar
  • FPH/CFH
  • Microwave Radiometer
  • FTIR
  • Brewer/Dobson
  • UV/VIS zenith DOAS
  • UV/VIS MAXDOAS
  • Pandora
  • GNSS-PW
Related gaps: 
G3.01 Incomplete knowledge of spatiotemporal atmospheric variability at the scale of the measurements and of their co-location
G3.04 Limited characterization of the multi-dimensional (spatiotemporal) smoothing and sampling properties of atmospheric remote sensing systems, and of the resulting uncertainties

  • G3.01. To be addressed before G3.06

    Argument: To quantify the additional errors and uncertainties in a comparison due to co-location mismatch, it is advantageous to have external information on the atmospheric variability on the scale of the co-location mismatch.

    G3.04. To be addressed before G3.06

    Argument: To quantify the additional errors and uncertainties in a comparison due to co-location mismatch, it is important to know the smoothing and sampling properties of the individual instruments 

Detailed description: 

Ideally, every validation study based on comparisons with ground-based reference data should investigate whether the comparison statistics (bias or mean difference, spread on the differences, drift, etc.) are compatible with the reported random and systematic measurement uncertainties, while taking into account the additional uncertainties due to spatiotemporal sampling and smoothing differences, i.e. non-perfect co-location of the airmasses sensed by both instruments. Indeed, it is only in a few particular cases possible to adopt co-location criteria that result in a sufficiently large number of co-located pairs, while at the same time keeping the impact of atmospheric variability on the comparisons (due to spatiotemporal mismatches) well below the measurement uncertainties. In all other cases, the discrepancy between two data sets will contain non-negligible terms arising from sampling and smoothing differences, which need to be taken into account.  In fact, such an analysis is essential to fully assess the data quality and its fitness-for-purpose, but in practice, it is rarely performed, as this co-location mismatch is hard to quantify reliably. Some pioneering work was published by Cortesi et al. (2007) on uncertainty budget closure for MIPAS/ENVISAT ozone profile validation, by Ridolfi et al. (2007) for the case of MIPAS/ENVISAT temperature profiles validation, by Fasso et al. (2013) in the context of radiosonde intercomparisons, by Lambert et al. (2012) on water vapour comparisons, and by Verhoelst et al. (2015) for GOME-2/MetOp-A total ozone column validation. However, no such studies have hitherto been performed for most other ECVs and/or instruments. This gap therefore concerns the need for (1) further research dealing with methods to quantify co-location mismatch, and (2) governance initiatives to include in the common practices among validation teams dedicated efforts to construct full uncertainty budgets, and use these in the consistency checks. 

 

Operational space missions or space instruments impacted: 
  • Independent of specific space mission or space instruments
Validation aspects addressed: 
  • Radiance (Level 1 product)
  • Geophysical product (Level 2 product)
  • Gridded product (Level 3)
  • Assimilated product (Level 4)
  • Time series and trends
  • Representativity (spatial, temporal)
  • Calibration (relative, absolute)
  • Spectroscopy
  • Auxiliary parameters (clouds, lightpath, surface albedo, emissivity)
Gap status after GAIA-CLIM: 
  • GAIA-CLIM explored and demonstrated potential solutions to close this gap in the future

Dedicated studies within GAIA-CLIM aimed for full error (or uncertainty) budget decomposition for representative comparison exercises, involving all non-satellite measurement types targeted by GAIA-CLIM and several current satellite sounders. Moreover, some of these results were transferred into the Virtual Observatory to allow end users to also decompose the uncertainty budget of their comparisons. Nevertheless, further work is required to quantify comparison error budgets in many cases, to operationalise comparison error budget calculations in operational satellite validation and production of higher level services, and to increase awareness in the community of the need for comparison error budget closure.

 

Part II Benefits to resolution and risks to non-resolution

Identified benefitUser category/Application area benefittedProbability of benefit being realisedImpacts
Improved feedback on data quality from the validation work, including on the reported uncertainties.
  • All users and application areas will benefit from it
  • High
Optimized use of the data, avoiding over-interpretation but potentially also allowing greater detail to be extracted.
Tighter constraints from validation work support product development
  • All users and application areas will benefit from it
  • High
Shortcomings in products are more easily identified, driving further development and ultimately ensuring better, more reliable data products.
Identified riskUser category/Application area at riskProbability of risk being realisedImpacts
Incomplete –or even incorrect- feedback from a validation exercise on the data quality.
  • All users and application areas will suffer from it.
  • High
Poorly quantified data quality, affecting all use types. Sub-optimal feedback to data providers slows product development. The potential of the EO system is not fully realized.

Part III Gap remedies

Gap remedies: 

Remedy 1: Use of Observing System Simulation Experiments (OSSEs)

Primary gap remedy type: 
Research
Proposed remedy description: 

This remedy concerns Observing System Simulation Experiments (OSSEs), such as those performed with the OSSSMOSE system by Verhoelst et al. (2015) on total ozone column comparisons. These are based on a quantification of the atmospheric field and its variability (c.f. gap G3.01), e.g. in the shape of reanalysis fields, and on a detailed description of the sampling and smoothing properties of the instruments that are being compared (c.f. gap G3.04). The aim is to calculate the error due to spatiotemporal mismatch for each comparison pair, and to derive the mismatch uncertainties from these, so that they can be added to the measurement uncertainties to derive the full uncertainty budget.

The technological and organizational challenges are mostly related to the underlying gaps G3.01 and G3.04.  When these are properly addressed, the calculation of the full uncertainty budget of a comparison exercise requires only a low investment in time and resources. Integrating this into an operational validation context does constitute an additional challenge requiring dedicated effort and funding.

Relevance: 

This remedy addresses directly the gap. 

Measurable outcome of success: 

At a high level, success is achieved when validation (and other comparison) results are published including a full uncertainty budget decomposition, taking into account spatiotemporal mismatch uncertainties. Or when they include a convincing demonstration that mismatch uncertainties are well below the measurement uncertainties and are negligible. 

At a lower level, success is achieved if the OSSE allows one to close the uncertainty budget, i.e. the measured differences (or their statistics) are compatible with the sum of all uncertainty sources. Note that this requires reliable measurement uncertainties as well. 

Expected viability for the outcome of success: 
  • High
Scale of work: 
  • Single institution
  • Consortium
Time bound to remedy: 
  • Less than 3 years
Indicative cost estimate (investment): 
  • Medium cost (< 5 million)
Indicative cost estimate (exploitation): 
  • No
Potential actors: 
  • EU H2020 funding
  • Copernicus funding
  • National funding agencies
  • National Meteorological Services
  • ESA, EUMETSAT or other space agency
  • Academia, individual research institutes

Remedy 2: Statistical estimation of typical co-location mismatch effects

Primary gap remedy type: 
Research
Proposed remedy description: 

An alternative to estimating co-location mismatch (the main missing term in the uncertainty budget decomposition of a comparison) from model simulations, is to employ statistical modelling on the differences, for instance with a heteroskedastic functional regression approach, (as implemented for instance in the STAT4COLL software package).  In certain applications, this approach also allows one to disentangle measurement uncertainties from co-location mismatch, at least for the random components. GAIA-CLIM will have employed such an approach for a subset of specific cases (spatial domains and ECVs / measurement techniques). Further efforts are required to generalise the approach and tools to enable broader exploitation, including integration into an operational validation context.

Relevance: 

Employ statistical modelling on the differences, for instance with a heteroskedastic functional regression approach. Efforts are required to generalise the GAIA-CLIM approach and tools to enable broader exploitation. 

Measurable outcome of success: 

At a high level, success is achieved when validation (and other comparison) results are published including a full uncertainty budget decomposition, taking into account spatiotemporal mismatch uncertainties. Or when they include a convincing demonstration that mismatch uncertainties are well below the measurement uncertainties and are therefore negligible. 

At a lower level, success is achieved if the statistical modelling allows one to close the uncertainty budget, i.e. the measured differences (or their statistics) are compatible with the sum of all uncertainty sources. Note that this requires reliable measurement uncertainties as well. 

Expected viability for the outcome of success: 
  • High
Scale of work: 
  • Single institution
  • Consortium
Time bound to remedy: 
  • Less than 3 years
Indicative cost estimate (investment): 
  • Medium cost (< 5 million)
Indicative cost estimate (exploitation): 
  • No
Potential actors: 
  • EU H2020 funding
  • Copernicus funding
  • National funding agencies
  • National Meteorological Services
  • ESA, EUMETSAT or other space agency
  • Academia, individual research institutes
References: 

  •  Cortesi et al., “Geophysical validation of MIPAS-ENVISAT operational ozone data”, ACP v7, 2007

  •  Fassò et al., “Statistical modelling of collocation uncertainty in atmospheric thermodynamic profiles”, AMT v7, 2014

  • Lambert, J.-C., et al., “Comparing and merging water vapour observations: A multi-dimensional perspective on smoothing and sampling issues”, in “Monitoring Atmospheric Water Vapour: Ground-Based Remote Sensing and In-situ Methods”, N. Kämpfer (Ed.), ISSI Scientific Report Series, Vol. 10, Edition 1, 326 pp., ISBN: 978-1-4614-3908-0, DOI 10.1007/978-1-4614-3909-7_2, © Springer New York 2012

  • Ridolfi et al., “Geophysical validation of temperature retrieved by the ESA processor from MIPAS/ENVISAT atmospheric limb-emission measurements”,  ACP v7, 2007

  • Verhoelst et al., “Metrology of ground-based satellite validation: Co-location mismatch and smoothing issues of total ozone comparisons”, AMT v8, 2015