This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 640276.
G1.03 Lack of internationally recognised and adopted framework for assessment of fundamental observing capabilities
Submitted by Anna Mikalsen on 8 June, 2017 - 14:18
Gap abstract:
There currently exists no universally recognised approach for assessing quantifiable aspects of the measurement system maturity of existing observing networks. Although absolute measurement quality cannot be assured, fundamental properties of the measurement system that build confidence in its appropriateness and metrological verity can be assessed. The lack of an agreed international framework for such an assessment leads to heterogeneity in the approaches used to select the most suitable measurement series for any given application. This frequently has deleterious effects for downstream applications in that often the measurements are used in a manner that is not optimal or even not appropriate.
Part I Gap description
Primary gap type:
- Parameter (missing auxiliary data etc.)
Secondary gap type:
- Governance (missing documentation, cooperation etc.)
ECVs impacted:
- Temperature,Water vapour, Ozone, Aerosols, Carbon Dioxide, Methane
User category/Application area impacted:
- Operational services and service development (meteorological services, environmental services, Copernicus Climate Change Service (C3S) and Atmospheric Monitoring Service (CAMS), operational data assimilation development, etc.)
- International (collaborative) frameworks and bodies (space agencies, EU institutions, WMO programmes/frameworks etc.)
- Climate research (research groups working on development, validation and improvement of ECV Climate Data Records)
Non-satellite instrument techniques involved:
- Independent of instrument technique
-
The resolution of the current gap will aid resolution of G1.04 by providing an assessable basis with broad buy-in to classify individual contributing measurement systems
G1.03, as well as G1.04, must be addressed after G1.06, which will provide all the required information to proceed towards an effective approach to resolving both G1.03 and G1.04.
Detailed description:
No systematic effort has been made to define and broadly agree amongst global stakeholders on the measurement and network characteristics underlying a systemic approach to Earth Observation. Nor is there any recognised approach in place to ensure a consistent way of assessing where any given observation sits within such a framework.
Different observational domain areas (atmospheric, composition, marine, terrestrial, cryospheric, etc.) use domain-specific, but overlapping naming conventions. These often use the same label such as ‘reference’ or ‘baseline’ to mean very different things. The unwary user is faced with an unenviable task as a result, and this yields sub-optimal and / or incorrect usage of available observational records in many cases and confusion for funders, users, and stakeholders.
This gap potentially inhibits realisation of the full benefits of an explicitly system-of-systems architecture (trickle down calibration, characterisation, etc.) across the global networks. It also places the burden of appropriate use of data squarely on the user, which is an unrealistic expectation in the majority of cases as the user is not, at least ordinarily, sufficiently expert in the nuances of observational programs (and nor should they be expected to be so).
The gap has been recognised in the most recent (2016) GCOS Implementation Plan and an action (G13 Review of ECV observational networks) associated, which speaks to elements of this gap.
Operational space missions or space instruments impacted:
- Independent of specific space mission or space instruments
Validation aspects addressed:
- Representativity (spatial, temporal)
- Calibration (relative, absolute)
Gap status after GAIA-CLIM:
- GAIA-CLIM has partly closed this gap
The GAIA-CLIM-related activities are described in peer-reviewed literature (Thorne et al., 2017). This clearly articulates the method that GAIA-CLIM used, but does not close the gap as it is, at this stage, only an approach used by a single project. Therefore, while it shows a potential approach to solving the gap, it lacks the broad community and institutional buy-in aspects necessary to close the gap. Remaining aspects to be addressed include a broader assessment of applicability to other observational capabilities and discussion and agreement by appropriate international entities.
Part II Benefits to resolution and risks to non-resolution
| Identified benefit | User category/Application area benefitted | Probability of benefit being realised | Impacts |
|---|---|---|---|
Homogeneous basis for choice of appropriate observations for particular applications |
|
| Consistent use of observations across diverse applications on a verifiable basis. Increased confidence for users. Increased provenance behind data selection decisions |
Identified pathways to improving quality of observational programs |
|
| Targeted basis for improving quality of assessable aspects of measurement programs. Enhanced informed funding support decisions programmatically and internationally |
Realising synergies between observational programs |
|
| Assessment process would highlight potential synergies achievable between national, regional, and global observational capabilities. See gaps related to governance ( G06.XX) to which this may contribute as a result |
| Identified risk | User category/Application area at risk | Probability of risk being realised | Impacts |
|---|---|---|---|
Continued ad hoc decision-making process for selection of observations for given uses |
|
| Incomparability of analyses owing to differences in choices of observations to use. Inappropriate observations being used and risk of making false inferences as a result (conflating observational error with real phenomena) |
Support decisions targeting the wrong observation programs |
|
| Good observational programs put under pressure / discontinued. Not realising the full benefit of past financial investments for science and society. Reduction of cost-effectiveness in the use of resources. Synergies between observing capabilities not realised leading to degraded assessments of observational change |
Full value of programs such as WIGOS not realised |
|
| Reduced utility of global observational capabilities and coordination of programs. Lack of buy-in at national and regional level to integrated observing system concepts |
Continued within and across domain confusion in naming conventions and data-quality assessments |
|
| Confusion to end-users on what different data streams constitute |
Part III Gap remedies
Gap remedies:
Remedy 1: Further deployments and refinements of the GAIA-CLIM approach
Primary gap remedy type:
Education/Training
Secondary gap remedy type:
Deployment
Research
Governance
Proposed remedy description:
To develop, refine, and deploy a system-of-systems measurement maturity assessment as developed by GAIA-CLIM across a range of use cases to determine the degree to which it is potentially applicable across non-satellite observing platforms and problems. Already under way for the Arctic domain under the H2020 INTAROS project in the context of the Copernicus Climate Change Service Evaluation and Quality Control program, its use and further development could be undertaken across a broader range of cases and with a range of international programmatic cases. This would constitute further refinement and proof-of-concept testing of the applicability, utility, and value of a measurement system maturity assessment approach to enable subsequent adoption. This testing should include a consideration of applicability across a diverse range of observational networks and across the full range of observational domains (surface, atmospheric, oceanic, terrestrial, hydrological, cryospheric). This will permit an evaluation of the value of the measurement maturity assessment, as well as its fitness-for-purpose for applications such as the Copernicus Climate Change Service and the WMO Integrated Observing System (WIGOS).
Relevance:
The application of GAIA-CLIM approach to other cases shall lead to improvements in the guidance and approach and enable greater buy-in from a more diverse range of stakeholders.
Measurable outcome of success:
One or more reports or peer-reviewed papers describing the application and developments.
Expected viability for the outcome of success:
- High
Scale of work:
- Single institution
- Consortium
Time bound to remedy:
- Less than 3 years
Indicative cost estimate (investment):
- Low cost (< 1 million)
Indicative cost estimate (exploitation):
- No
Potential actors:
- EU H2020 funding
- Copernicus funding
- WMO
Remedy 2: Adoption of measurement systems approach and assessment by international bodies
Primary gap remedy type:
Deployment
Secondary gap remedy type:
Education/Training
Governance
Proposed remedy description:
Adoption of the GAIA-CLIM approach or of a similar approach to measurement maturity assessment established by globally responsible entities, such as the Global Climate Observing System (GCOS) or WIGOS and / or in subsequent relevant scientific projects. A single approach needs to be formulated, adopted, and rolled out across a broad range of non-satellite observing capabilities to assess their maturity and appropriately categorise their role in the global observing system. Periodic re-review of observing capabilities should then be instigated to ensure that assessments reflect up-to-date snapshots of measurement capabilities. A mechanism of feedback to the contributing measurement networks should be codified and enacted. The results of the assessments should be made available in a way that provides actionable information to end-users and to ensure they use the most appropriate data for their applications.
Relevance:
The adoption of an international programmatic effort to assess measurement capabilities would directly address the gap and ensure broad buy-in.
Measurable outcome of success:
Documentation of adopted mechanism, results of assessment available to users.
Expected viability for the outcome of success:
- High
Scale of work:
- Programmatic multi-year, multi-institution activity
Time bound to remedy:
- Less than 3 years
Indicative cost estimate (investment):
- Low cost (< 1 million)
Indicative cost estimate (exploitation):
- Yes
Low recurring costs for evaluations and process management
Potential actors:
- Copernicus funding
- WMO
- ESA, EUMETSAT or other space agency
References:
- Global Climate Observing System GCOS 2016 Implementation Plan. https://ane4bf-datap1.s3-eu-west-1.amazonaws.com/wmocms/s3fs-public/programme/brochure/GCOS-200_OnlineVersion.pdf?PlowENiCc1RGh9ReoeAoGBT0QhnJYm6_
- Thorne, P. W., Madonna, F., Schulz, J., Oakley, T., Ingleby, B., Rosoldi, M., Tramutola, E., Arola, A., Buschmann, M., Mikalsen, A. C., Davy, R., Voces, C., Kreher, K., De Maziere, M., and Pappalardo, G. (2017) : "Making better sense of the mosaic of environmental measurement networks: a system-of-systems approach and quantitative assessment", Geosci. Instrum. Method. Data Syst., 6, 453-472, https://doi.org/10.5194/gi-6-453-2017, 2017.