Detailed description

The task of characterizing satellite measurements by means of comparison to reference measurements needs consistent and reliable access to data and documentation of various fiducial reference measurements for the analysis of the quality of satellite measurements and/or derived geophysical data products. This task can be massively complicated and time-consuming arising from the need to collect data from multiple locations also often offering the data on various types of user interfaces with which a user needs to become familiar. In many cases, data downloads do not follow specific data exchange standards, which makes it difficult to automate access to them. In addition, the available bandwidth at the provider side might be too small to serve many customers, which can result in extended waiting times for the data. This applies even more when co-located ground based and satellite data are to be offered to the user. The range of data policies that a user needs to adhere to further complicates the issue. These include timeliness of the data exchange.

A common source that integrates several reference-data networks with satellite data considering traceable uncertainty does not exist but is needed according to the GAIA-CLIM user survey. A key first step to this is consistent access to reference quality measurement systems in a harmonised data format that contains requisite discovery metadata and for which the data usage policy and restrictions are clearly articulated. Many of the existing data policies can be very different, e.g.,

  • Completely open access for all users including commercial users;

  • Open access for research purposes only;
  • Open access after a set time delay;
  • Access only upon request to PI.

Several sources for co-located data sets exist, but most of them are specialized to very particular use cases. Most are not fully utilizing the potentially available information on uncertainty or including uncertainty arising from spatiotemporal mismatch of the compared data streams. Some of the existing datasets are publically available via the internet, while others are run internally to organizations like space agencies to monitor data quality in real time. While many validation activities are performed, they do not use the available uncertainty information in an optimal way, which has resulting impacts on the quality of the research and the robustness of any conclusions drawn from such validation exercises.

In summary, the issues over data discovery and access are pervasive and inhibit their effective usage in a broad range of application areas, including satellite Cal/Val activities. The recently instigated Copernicus Climate Change Service contract C3S311a Lot3 which is concerned with access to data from baseline and reference networks may go a considerable length towards addressing this gap for non-satellite reference measurements and is discussed under remedy G5.01(R1). 

Operational space missions or space instruments impacted
Independent of specific space mission or space instruments
Gap status after GAIA-CLIM
GAIA-CLIM explored and demonstrated potential solutions to close this gap in the future

Some of the work within GAIA-CLIM WP1 and WP5 will provide unified access to a range of reference quality data products via the VO facility. However, this access shall not be operational and substantive further work would be required. It also will not permit universal access for other applications to integrated holdings.

Dependencies

Gap 1.06 pertains to unifying metadata format and discovery metadata, which would naturally form a component of resolving the current gap. This critical dependent gap should be addressed with this gap. 

Presently, access to high-quality reference network data and satellite data is obtained through a variety of portals, using a broad range of access protocols, and the data files are available in an array of native data formats that lack interoperability (see Gap 1.06). There also exists a broad range of data policies from open access through delayed mode restricted access. To make effective usage of the full range of reference-quality measurements, e.g., for the characterisation of satellite data, therefore presently requires substantial investment of time and resources to instigate and maintain a large number of data-access protocols and data read/write routines, as well as to fully understand and adhere to a broad range of data policies and timeliness. This is a substantial impediment to the effective usage of data for applications, such as the GAIA-CLIM Virtual Observatory or similar application areas.