G3.02 Missing standards for, and evaluation of, co-location criteria

Submitted by Anna Mikalsen on 12 June, 2017 - 12:07
Gap abstract: 

The impact of a particular choice of co-location criterion is only rarely studied in the scientific literature reporting on satellite validation results. However, without some quantification of the impact of the co-location criterion that was adopted, it is virtually impossible to assess the contribution of natural variability to the total error budget of the data comparisons. As such, this gap impacts significantly the potential interpretation of the data comparison result in terms of data quality. Some in-depth studies do exist, but testing multiple criteria, or using criteria based on the latest results of such exploratory work, is far from common (indeed, often arbitrary) practice(s). This gap thus concerns the need for more awareness among validation teams, for more detailed studies comparing the (dis-)advantages of various co-location criteria, and for community-agreed standards on co-location criteria that are broadly adopted in the context of operational services.

Part I Gap description

Primary gap type: 
  • Uncertainty in relation to comparator
Secondary gap type: 
  • Governance (missing documentation, cooperation etc.)
ECVs impacted: 
  • Temperature,Water vapour, Ozone, Aerosols, Carbon Dioxide, Methane
User category/Application area impacted: 
  • Operational services and service development (meteorological services, environmental services, Copernicus Climate Change Service (C3S) and Atmospheric Monitoring Service (CAMS), operational data assimilation development, etc.)
  • International (collaborative) frameworks and bodies (space agencies, EU institutions, WMO programmes/frameworks etc.)
  • Climate research (research groups working on development, validation and improvement of ECV Climate Data Records)
Non-satellite instrument techniques involved: 
  • Independent of instrument technique
Related gaps: 
G3.04 Limited characterization of the multi-dimensional (spatiotemporal) smoothing and sampling properties of atmospheric remote sensing systems, and of the resulting uncertainties
G3.06 Missing comparison (validation) uncertainty budget decomposition including uncertainty due to sampling and smoothing differences
G6.03 Lack of sustained dedicated periodic observations to coincide with satellite overpasses to minimise co-location effects

  • G3.04. To be addressed before G3.02

    Argument: Ideally, co-location criteria take into account the smoothing and sampling properties of the measurements. Consequently, studies on co-location criteria can benefit from a proper characterization of these smoothing and sampling properties.

    G3.06. To be addressed before G3.02

    Argument: The merit of certain co-location criteria can best be assessed when the uncertainty budget of the comparisons is decomposed in measurement and co-location mismatch uncertainties.

    G6.03. To be addressed after G3.02

    Argument: Deciding on the best time and location for targeted reference observations should be informed by information on the optimal co-location criteria

Detailed description: 

Co-location criteria should represent an optimal compromise between the obtained number of co-located measurements (as large as possible to have robust statistical results) and the impact of natural variability on the comparisons (as low as possible to allow a meaningful comparison between measured differences and reported measurement uncertainties). Hitherto, only a few ground-based satellite validation studies explored the impact of the adopted co-location criteria on the comparison results (e.g. Wunch et al., 2011, and Dils et al., 2014, for CO2 , Verhoelst et al., 2015, for O3, Pappalardo et al., 2010, for aerosols, Lambert et al. 2012, for water vapour, Van Malderen et al. 2014, for integrated water vapour). Still, atmospheric variability is often assumed –or even known- to impact the comparisons, but without detailed testing of several co-location criteria (or by extensive model-based simulations), this impact is hard to quantify. Besides the need for dedicated studies, this gap also concerns the “community practices” regarding co-location approaches, which are neither consistent across different studies, nor optimal as they often rely on historical co-location criteria, which are not necessarily fit-for-purpose for the accuracy and spatiotemporal sampling properties of current measurement systems. Consequently, to ensure reliable and traceable validation results, as required in an operational context, community-agreed standards for co-location criteria should be developed, published, and adopted. 

Operational space missions or space instruments impacted: 
  • Independent of specific space mission or space instruments
Validation aspects addressed: 
  • Radiance (Level 1 product)
  • Geophysical product (Level 2 product)
  • Gridded product (Level 3)
  • Assimilated product (Level 4)
  • Time series and trends
  • Representativity (spatial, temporal)
  • Calibration (relative, absolute)
  • Spectroscopy
  • Auxiliary parameters (clouds, lightpath, surface albedo, emissivity)
Gap status after GAIA-CLIM: 
  • GAIA-CLIM explored and demonstrated potential solutions to close this gap in the future

Two activities within GAIA-CLIM targeted this gap to some extent:

Within GAIA-CLIM, a dedicated task (T3.2 in WP3) dealt with data intercomparison studies, focusing on a closure of the comparison uncertainty budget and including an exploration of different co-location criteria, see for instance the results on total ozone columns published by Verhoelst et al. (2015, their Fig. 11).

The Virtual Observatory developed within GAIA-CLIM offers the user the possibility to adjust co-location criteria and to visualize the resulting impact on the comparison results.

However, no attempt has been made within GAIA-CLIM to produce an authoritative analysis and resulting documentation on this matter. 

Part II Benefits to resolution and risks to non-resolution

Identified benefitUser category/Application area benefittedProbability of benefit being realisedImpacts
Greater awareness of the impact of natural variability on the comparison results;
  • All users and application areas will benefit from it
  • High
More reliable feedback on data quality, in particular on the reported uncertainties. This in turn increases confidence in the data for the end user and allows more meaningful use in a variety of applications.
Improved definition of appropriate co-location criteria for validation work, minimizing errors due to co-location mismatch.
  • All users and application areas will benefit from it
  • High
Lower uncertainty due to co-location mismatch will result in tighter constraints on the products from validation work, supporting further instrument and algorithm development.
Facilitates intercomparison of different validation studies
  • All users and application areas will benefit from it
  • High
More reliable comparisons between different products (each having its own validation report) to better assess their fitness-for-purpose for a specific user application.
Identified riskUser category/Application area at riskProbability of risk being realisedImpacts
Poor feedback on data quality (in particular on the reported uncertainties) from validation studies due to unknown/unquantified influence of atmospheric variability.
  • All users and application areas will suffer from it.
  • High
Poor confidence in data and services; potential over-interpretation; difficult/unreliable assimilation; Potential of both EO and ground segments not fully realized.
Difficulty to compare validation results on similar products performed by different teams
  • All users and application areas will suffer from it.
  • High
Sub-optimal choice of data product for a given application.

Part III Gap remedies

Gap remedies: 

Remedy 1: Systematic quantification of the impacts of different co-location criteria

Primary gap remedy type: 
Research
Secondary gap remedy type: 
Governance
Proposed remedy description: 

Dedicated studies are required which explore in detail the advantages and disadvantages of several co-location methods and criteria.  Dedicated working groups or activities could/should be set up within the framework of the ground-based observing networks, as already initiated for meteorological variables at a GRUAN-GSICS-GNSSRO WIGOS workshop on Upper-Air Observing System Integration and Application, hosted by WMO in  Geneva  in May 2014. Dissemination among, and acceptance by, the key stakeholders may be challenging and can probably best be achieved in the context of overarching frameworks such as the CEOS Working Group on Calibration & Validation (WGCV). The financial cost should be very low. Also, the space agencies and service providers could/should insist on sufficient attention for (and analysis of) the adopted co-location criteria in the validation protocols followed by their validation teams. 

Relevance: 

These studies and the proposed associated governance support target this gap directly. They will provide stakeholders with a traceable, authoritative reference on which to base their validation requirements and protocols regarding co-location criteria. It will also facilitate meta-analysis of different validation studies without the need to take into account differences in results due to differences in the impact of co-location mismatch on the results. 

Measurable outcome of success: 

Peer-reviewed publications or widely distributed technical notes on the subject, from an authorative body; Explicit inclusion of requirements on the co-location methodology and criteria in validation protocols. 

Expected viability for the outcome of success: 
  • High
Scale of work: 
  • Single institution
  • Consortium
Time bound to remedy: 
  • Less than 3 years
Indicative cost estimate (investment): 
  • Low cost (< 1 million)
Indicative cost estimate (exploitation): 
  • No
Potential actors: 
  • EU H2020 funding
  • Copernicus funding
  • National funding agencies
  • National Meteorological Services
  • WMO
  • ESA, EUMETSAT or other space agency
  • Academia, individual research institutes
References: 
  •      Dils et al., “The Greenhouse Gas Climate Change Initiative (GHG-CCI): comparative validation of GHG-CCI SCIAMACHY/ENVISAT and TANSO-FTS/GOSAT CO2 and CH4 retrieval algorithm products with measurements from the TCCON”, AMT v7, 2014
  • Lambert, J.-C., et al., “Comparing and merging water vapour observations: A multi-dimensional perspective on smoothing and sampling issues”, in “Monitoring Atmospheric Water Vapour: Ground-Based Remote Sensing and In-situ Methods”, N. Kämpfer (Ed.), ISSI Scientific Report Series, Vol. 10, Edition 1, 326 pp., ISBN: 978-1-4614-3908-0, DOI 10.1007/978-1-4614-3909-7_2, © Springer New York 2012

  • Pappalardo et al., “EARLINET correlative measurements for CALIPSO: First intercomparison results”, J.G.R.: Atmospheres v115, 2010

  • Van Malderen, R. et al., “A multi-site intercomparison of integrated water vapour observations for climate change analysis”, AMT v7, 2014

  • Verhoelst et al., “Metrology of ground-based satellite validation: Co-location mismatch and smoothing issues of total ozone comparisons”, AMT v8, 2015

  •      Wunch et al., “A method for evaluating bias in global measurements of CO2 total columns from space”, ACP v11, 2011