Copernicus, the European Earth observation and modelling initiative, offers its data to a multitude of individuals, organisations and institutions openly and free of charge through its thematic information services. In particular, the Copernicus Climate Change service has more than 130 000 users, and has provided more than 88 000 TBs of data through its Climate Data Store. And all this in just 4 years, as the service has been operational since 2018. To continue to provide an innovative service, and to remain at the international forefront, Copernicus keeps moving forward in a second phase. And for this, it will count on our contribution. All thanks to a project that will update the European portals Climate Data Store, of the Climate Change service, and the Atmosphere Data Store, of the Atmospheric Monitoring service, which provide climate change and air quality data, respectively. The project consists of 6 partners spread across Europe and is led by B-Open, an Italian company. The project will run for up to 4 years and has a budget of 4 million euros.
The aim of this project is to further improve free and open access to Copernicus climate and environmental data in order to continue to support decisions related to climate change or air quality. These are two portals with thousands of daily users, serving thousands of gigabytes of data every day and need to adapt to new demands and technologies. The new versions of the portals will adopt mostly open source technologies and will be oriented to be compatible with the demands of users, including Machine Learning technologies, Data Cubes and interactive Notebooks.
Predictia will be in charge of three specific aspects of the project: system monitoring, design and implementation of an observation repository; as well as the development of a common content and process manager for the portals. Monitoring is essential in this type of operational services, since it allows to monitor the system in real time and to have historical metrics on its performance and operation. At Predictia we implement an automated system to guarantee the correct functioning of the service and to obtain information about which parts of the infrastructure need different types of improvements: download speeds, user satisfaction, waiting time to access the data...
Part of the data handled by the Climate Data Store is related to observation data from weather stations. The complexity of these data lies in their great heterogeneity: data from very different sources must be integrated, all in a single repository and organised in a homogeneous way. In addition, we have to guarantee the quality of the data, for which a complex validation process must be carried out. To do all this, we will rely on the University of Vienna, another European partner of the consortium, a specialist in this type of data. Predictia will be responsible for the design and technical implementation of the observation repository.
To guarantee the quality control of the data, Predictia will implement a system to document and manage all the quality control processes and make this information available to users. Before the data in the repositories are made public, there is a long process: there are specific contracts to manage, to go through an exhaustive quality review process... The aim is to centralise the management of these processes, saving Copernicus time and resources. This will make it possible to monitor the status of each project, integrate the data generated in the catalogues, manage internal reports...
Finally, the project is supported by the Institute of Physics of Cantabria, which will support OpenStack, the infrastructure management software in the cloud where the project will be used.
The Copernicus Climate Change and Atmospheric Monitoring services are managed by the European Centre for Medium-Range Weather Forecasts (ECMWF), on behalf of the European Commission.