8. Monitoring data use

This step can be implemented as follows

The person promoting data sharing in the organisation

  • considers which impact criteria are to be measured in their own organisation,
  • monitors and measures the use of published data and the benefits and impacts of its use, and
  • reports on the use, benefits and effectiveness of the opened data.

Use monitoring

This section describes why and how the organisation can monitor the use of its opened datasets.

Why should the use of opened datasets be monitored?

Investment decisions in open data publications often need to be justified by value creation, i.e. the benefits derived from the investment. The organisation may also have set quantitative targets for such benefits. It is important to measure the benefits created by open data, making it possible to report them and to use them to justify future investments.

Organisations that have already opened their data are often interested in finding out the extent to which their datasets have been viewed or downloaded in the data portal. It is important to note that while the viewing and downloading data concerning datasets published on the portal or other online service do not indicate if the dataset has actually been used and how, they do give an indication of the interest attracted by the opened dataset.

How should the use of opened datasets be measured?

Measuring open data is difficult, but the scale of its use can be measured and estimated. You can choose from a number of indicators, such as view and download statistics, or API use metrics. In addition, the utilisation of data can be measured with other factors, such as media visibility and the number of organisations and applications making use of your data. Information on open data use can, for example, be collected by means of media monitoring (keywords) and surveys or by offering various incentives to report on data use. The table below contains examples of different use-related factors and indicators.

Examples of measurement methods

Measured factorExample indicator
DownloadsNumber of files downloaded
ViewsNumber of files browsed
API useAPI usage statistics
Data utilisationMedia monitoring and press articles
Number of organisations using the data
Number of applications using the data
Surveys and interviews
Use of data in hackathons
Contacts by higher education institutions
Data qualityFeedback: contacts, number, content, background organisations

Monitoring and measuring the use of opened data also depends on the format it was published in. Different statistics can be obtained on the use of data published in a dynamic form, often through an API, and its use is reasonably easy to monitor. On the other hand, monitoring the use of static data (files) is more challenging.

Measuring data published as files

Static data is usually downloaded only once, after which the user has unlimited access to it. Static data downloads do not necessarily convey any information about the extent of data use or the impact of its publication. In addition, open data licences permit the re-sharing of data, which means that other parties can also share the file in their services. For example, an individual media outlet may download the dataset once and publish a news item based on it, which spreads to several other media outlets. Consequently, even one download can have a high impact.

Measuring data published through an API

It is possible to collect analytics on API use, which can then be used to develop the API and possibly also to improve data quality. Statistics can be compiled using various applications, such as Matomo. For example, analytics can be collected on the general volume and methods of data use, API requests and user locations.

If the use of the API requires an API key, this key can be used to distinguish between data users and to monitor the API’s utilisation rate. API keys can also be distributed anonymously, allowing data users to retain their anonymity, and the use of the API key does not undermine the basic principles of openness. If someone misuses an open data API that requires an API key, the key can be used to differentiate between users and, if necessary, limit server loads.
 

Monitoring methods used in Suomi.fi Open Data service

The view and download rates of data published on Open Data service are displayed openly on the page for each dataset. In addition, the service’s Statistics page provides additional information on e.g. the most popular datasets. The service also has an application gallery, which maintains an overview of applications that use open data, with emphasis on the central government’s open datasets.

Monitoring methods used by the Finnish Meteorological Institute

The Finnish Meteorological Institute uses Spatineo Monitor for usage monitoring (downloads, user groups). In addition, the development of a continuous situational picture and dashboard is under way with Spatineo. The Finnish Meteorological Institute reports on the use of its open data in its data balance sheet.

Monitoring benefits and impacts

This section describes how the organisation can monitor the use-related benefits and impacts of its opened datasets.

After opening your data, it is a good idea to monitor and measure the benefits and impacts that this process has had. The benefits and impacts of opening data are impossible to estimate accurately, as the opening of data has many indirect benefits that are difficult to measure. For more information on the benefits of opening data, see step 3.

No established set of indicators exists for evaluating the impact of open data, as the development of research and measurement tools is still in its infancy. However, the measurement of the impact of open data has been carried out both at the organisational level and internationally. Examples of impact studies can be found at the end of this section.

The following methods can be used to obtain information on the impact of the opened data:

  • Collecting statistics on data downloads and, if possible, use cases.
  • Monitoring data use in new services, innovations and applications. 
  • Collecting information on the impacts of data directly from users with various surveys.
  • Conducting cost-benefit analyses on the economic benefits of opening data. The monetary value of many benefits is difficult to measure, however.
  • Commissioning reports or studies.

When assessing the impacts, it is advisable to take into consideration the entire life cycle of the data and the impacts of opening it from a wider societal perspective. Assessing the impacts can help the organisation to improve its effectiveness, fulfil its public mission better and increase the transparency of its work.

Impact studies

The European Commission's Open Data Maturity in Europe study

The data.europa.eu service, administrated by the European Commission, conducts an annual Open Data Maturity in Europe survey. One of the survey’s sections compares European countries on the basis of their open data impact.

The impact of open data can be assessed from e.g. political, societal, economic, and environmental perspectives:

Political impact
For example, open data has made it possible to 

  • increase administrative efficiency and the effectiveness of public services,
  • increase the transparency and accountability of public administration, and
  • foster better policy-making and decision-making.

Social impact
For example, open data has made it possible to 

  • pay more attention to marginalised groups and reduce inequalities, and
  • raise awareness of the factors related to housing, health, well-being, and education.

Economic impact
For example, open data has helped increase 

  • employment,
  • the creation of technologies and innovations, and
  • entrepreneurship and the creation of new businesses.

Environmental impact
For example, open data has made it possible to

  • raise awareness of biodiversity,
  • increase the environmental friendliness of cities,
  • raise awareness of climate change and related disasters, and
  • encourage the use of renewable energy sources while reducing the use of fossil fuels. 

Learn more about measuring impact in this course provided by data.europa.eu.

Impact monitoring at the Finnish Environment Institute

The Finnish Environment Institute offers open data to assist in the building of a sustainable environment and society. The Institute's open data service provides datasets on surface waters and groundwater, the Baltic Sea, environmental pollution and disturbances, valuable natural environments, soil cover and the built environment, among other things. This data has mainly been produced and collected by the agencies of the central government's environmental administration, in particular the Finnish Environment Institute and the Centres for Economic Development, Transport and the Environment.

The Finnish Environment Institute's goal is to be a pioneer in the development of the environmental data value chain. Open data serves this strategic goal. To support the assessment of impact, open data use is monitored by annually collecting information on API usage and file downloads.

In addition, the Finnish Environment Institute has conducted surveys and individual interviews with open data users. Surveys can give a general idea of such aspects as the users of open environmental data, the environmental themes that attract users’ interest, and general information about the uses made of the data. In surveys, as uses of open environmental data have been mentioned various tasks related to decision-making and planning, research and education, investigating the quality of the environment, influencing matters related to user's immediate surroundings and maintaining or developing different products and services.

Impact monitoring at the Finnish Meteorological Institute

The Finnish Meteorological Institute conducted an open data impact study in 2019. This analysis was found useful, and the Finnish Meteorological Institute is planning to conduct similar studies at regular intervals in the future. The results of the study were published in the Finnish Meteorological Institute's Data balance sheet 2020.

Studies and reports on the use of open data

Turunen, S. & Ruokonen, T. 2018. Using open data in municipalities.(In Finnish, pdf)

Herrala, Antti. 2018. Benefits from Open Data: Barriers of Supply and Demand of Open Data in Private Organizations.

Muurinen, R. & Open Knowledge Finland. 2017. Business from open data. Current review of 100 cases. (In Finnish, pdf)

Koski, H., Honkanen, M., Luukkonen, J., Pajarinen, M. & Ropponen, T. 2017. Use and impacts of open data . Publication of Government’s analysis, assessment and research activities 40/2017.(In Finnish, pdf)

Koski, H. 2015. The Impact of open data – a preliminary study.Publications of the Ministry of Finance 15a/2015.

Kukkamäki, J., Mikkonen, A., Stormi, I. & Mäntyneva, M. 2020. Using open data in education and researchIn J. Kukkamäki & M. Tarkkala (eds.), Avoin Häme. HAMK Unlimited Journal 25 Aug 2020. (In Finnish)

Poikola, A., Kola, P. & Hintikka, K. A. 2010. Public data - guide to opening resources. Ministry of Transport and Communications. (In Finnish, pdf)

Verhulst, S. G. & Young, A. 2018. Open Data Demand. Toward an open data demand assessment and segmentation methodology

Ahonen-Rainio, P., Mäkelä J. & Virrantaus K. 2014. A Method to assessing impact of open spatial data. (In Finnish, pdf)

Muukkonen, M., Leppälä, S., Kauhanen-Simanainen, A. & Myllyniemi, N. 2022. Public sector data - the impact of fees. Publication of Government’s analysis, assessment and research activities 2022/50. (In Finnish, pdf)

Support materials on the topic

This section contains support material related to the topics discussed in this step.

Training courses on the data.europa.eu website: