G3.06

G3.06    Missing comparison error/uncertainty budget decomposition including errors/uncertainties due to sampling and smoothing differences

Gap detailed description

Ideally, every validation exercise based on comparisons with ground-based reference data should investigate whether the comparison statistics (bias or mean difference, spread on the differences, drift, etc.) are compatible with the reported random and systematic measurement uncertainties, while taking into account the additional uncertainties due to spatiotemporal sampling and smoothing differences, i.e. non-perfect co-location of the airmasses sensed by both instruments.  Indeed, only in a few particular cases is it possible to adopt co-location criteria that result in a sufficiently large number of co-located pairs, while at the same time keeping the impact of atmospheric variability on the comparisons (due to spatiotemporal mismatches) well below the measurement uncertainties. In all other cases, the discrepancy between two data sets will contain non-negligible terms arising from sampling and smoothing differences, which need to be taken into account.  In fact, such an analysis is essential to fully assess the data quality and its fitness-for-purpose, but in practice, it is rarely performed. Some pioneering work was published by Cortesi et al. (2007) on uncertainty budget closure for MIPAS/ENVISAT ozone profile validation, by Ridolfi et al. (2007) for the case of MIPAS/ENVISAT temperature profiles validation, by Fasso et al. (2013) in the context of radiosonde intercomparisons, and by Verhoelst et al. (2015) for GOME-2/MetOp-A total ozone column validation. However, no such studies have hitherto been performed for most other ECVs and/or instruments.

Activities within GAIA-CLIM related to this gap

This gap is a key focal point for tasks T3.1 and T3.2 in WP3.  Dedicated studies will aim for full error (or uncertainty) budget decomposition for representative comparison exercises, involving all non-satellite measurement types targeted by GAIA-CLIM and several current satellite sounders. Moreover, some of these results will be transferred into the Virtual Observatory to allow end users to also decompose the uncertainty budget of their comparisons.

Gap remedy(s)

Remedy #1

Specific remedy proposed

Observing System Simulation Experiments (OSSEs), such as those performed with the OSSSMOSE system by Verhoelst et al. (2015) on total ozone column comparisons. These are based on a quantification of the atmospheric field and its variability (cfr. gap G3.01), e.g. in the shape of reanalysis fields, and of the sampling and smoothing properties of the instruments that are being compared (cfr. gap G3.04). The aim is to calculate the error due to spatiotemporal mismatch for each comparison pair, and to derive the mismatch uncertainties from these, so that they can be added to the measurement uncertainties to derive the full uncertainty budget. 

Measurable outcome of success

At a high level, success is achieved when validation (and other comparison) results are published including a full uncertainty budget decomposition, taking into account spatiotemporal mismatch uncertainties. Or when they include a convincing demonstration that mismatch uncertainties are well below the measurement uncertainties and are therefore negligible.

At a lower level, success is achieved if the OSSE allows one to close the uncertainty budget, i.e. the measured differences (or their statistics) are compatible with the sum of all uncertainty sources. Note that this requires reliable measurement uncertainties as well.

Achievable outcomes

The technological and organizational challenges are mostly related to the underlying gaps G3.01 and G3.04.  When these are properly addressed, the calculation of the full uncertainty budget of a comparison exercise requires only a low investment in time (less than a year) and resources (less than 1 FTE).

Relevance

This remedy comprehensively addresses the gap.

Timebound

Performing the OSSE for a specific validation/comparison exercise requires only a small amount of time (order of months), but the development and scientific assessment of the building stones (high-resolution atmospheric fields and observation operators quantifying the true 4-D measurement sensitivity) can take years of effort (cfr. gaps G3.01 and G3.04).

Remedy #2

Specific remedy proposed

To compute an uncertainty budget based on the heteroskedastic functional regression approach, which is named STAT4COLL, for each ECV/instrument comparison for the selected/available datasets. As a result mismatch uncertainties are obtained from statistical modelling, so that they can be added to the measurement uncertainties to provide the full uncertainty budget. 

Measurable outcome of success

As for remedy #1

Achievable outcomes

The technological/organizational viability is considered medium and the cost is estimated low.  Nonetheless, challenges are still related to the insufficient number of reference measurement available to precisely address the proposed gap for some ECV/instrument comparisons.

Relevance

GAIA-CLIM will approach this gap using advanced statistical approaches whose application may be generalized and extended to other ECVs. As a post GAIA-CLIM development, integration between remedy #1 and #2 could have a big potential in instrument comparisons.

Timebound

Performing STAT4COLL for a specific validation/comparison exercise requires only a limited amount of time (approximately one year).

Gap risks to non-resolution