Open Data – one researcher’s experience
Sarah Callaghan 
sarah.callaghan@stfc.ac.uk
@sorcha_ni
Italsat F1: Owned 
and operated by 
Italian Space 
Agency (ASI). 
Launched January 
1991, ended 
operational life 
January 2001. 
The problem: rain and cloud 
mess up your satellite radio 
signal. How can we fix this?
Creating data: a 
radio 
propagation 
dataset 
Inside the receive cabin – the 
instruments my data came 
from
The receive cabin at Sparsholt 
in Hampshire
One day’s worth of raw data from one of 
the receivers
My job was to take this...
Creating/processing data
...turn it into this....
...with the final result being 
this.
Analysing data
…a process which 
involved 4 major steps, 4 
different computer 
programmes, and 16 
intermediate files for each 
day of measurements. 
Each month of 
preproccessed data 
represented somewhere 
between a couple of days 
and a week's worth of 
effort. 
It was a job where 
attention to detail was 
important, and you really 
had to know what you 
were looking at from a 
scientific perspective. 
Part of the Italsat data archive – on CDs in a shelf in 
my office
Preserving data (the wrong way!)
What the processed 
data set looks like on 
disk
What the raw data files 
looked like.
(I do have some Word 
documents somewhere 
which describe what all 
this is…)
Example documentation
Note the 
software 
filenames in 
the 
documentation
.
I still have the 
IDL files on 
disk 
somewhere, 
but I’d be very 
surprised if 
they’re still 
compatible 
with the 
current version 
of IDL 
What it all came down to:
Composite image from Flickr user bnilsen and Matt Stempeck (NOI), shared under 
Creative Commons license
And I wasn’t even preserving my data 
properly! 
As for sharing the data…
I did share, but there was a lot of non-disclosure agreements (I am not a 
lawyer!) 
And I didn’t feel like I got the credit for it.(The first publication based on the 
data wasn’t written by me, and I didn’t even get my name in the 
acknowledgements.)
Publications – grey literature
Publications – journal paper
Where’s the data?
Good news: the data is all on the BADC now
The UK’s Natural Environment Research Council 
(NERC) funds six data centres which between them 
have responsibility for the long-term management of 
NERC's environmental data holdings.
We deal with a variety of environmental 
measurements, along with the results of model 
simulations in:
• Atmospheric science
• Earth sciences
• Earth observation
• Marine Science
• Polar Science
• Terrestrial & freshwater science, Hydrology and 
Bioinformatics
Who are we and why do we care about data?
Journals have always published 
data…
Suber cells and mimosa 
leaves. Robert Hooke, Micrographia, 
1665
The Scientific Papers of William Parsons, 
Third Earl of Rosse 1800-1867
…but datasets have gotten so big, it’s not 
useful to publish them in hard copy 
anymore
Hard copy of the Human Genome at the 
Wellcome Collection, London
Creating a dataset is hard work!
"Piled Higher and Deeper" by Jorge Cham
www.phdcomics.com
Managing and archiving data so that it’s understandable by other 
researchers is difficult and time consuming too.
We want to reward researchers for putting that effort in!

Why make data open?
• Pressure from government to make 
data from publicly funded research 
available for free. 
● Scientists want attribution and credit 
for their work
● Public want to know what the 
scientists are doing
● Good for the economy if new 
industries can be built on scientific 
data/research
• Research funders want reassurance 
that they’re getting value for money 
●  Relies on peer-review of science 
publications (well established) and 
data (starting to be done!)
• Allows the wider research community 
and industry to find and use datasets, 
and understand the quality of the data
• Need reward structures and 
incentives for researchers to 
encourage them to make their 
data open – data citation and 
publication
Why bother linking the data to the publication? Surely 
the important stuff is in the journal paper?
If you can’t see/use the data, then you can’t test the 
conclusions or reproduce the results! It’s not science!
Most people have an idea of what a publication 
is
Some examples of data (just from the Earth 
Sciences)
1. Time series, some still being updated 
e.g. meteorological measurements
2. Large 4D synthesised datasets, e.g. 
Climate, Oceanographic, Hydrological 
and Numerical Weather Prediction 
model data generated on a 
supercomputer
3. 2D scans e.g. satellite data, weather 
radar data
4. 2D snapshots, e.g. cloud camera
5. Traces through a changing medium, e.g. 
radiosonde launches, aircraft flights, 
ocean salinity and temperature 
6. Datasets consisting of data from 
multiple instruments as part of the 
same measurement campaign
7. Physical samples, e.g. fossils
Should ALL data be open?
• Most data produced through 
publically funded research should 
be open.
But!
• Confidentiality issues (e.g. named 
persons’ health records)
• Conservation issues (e.g. maps of 
locations of rare animals at risk 
from poachers)
• Security issues (e.g. data and 
methodologies for building 
biological weapons) There should be a very 
good reason for publically 
funded data to not be 
open.
Open is not enough!
“When required to make the data available by 
my program manager, my collaborators, and 
ultimately by law, I will grudgingly do so by 
placing the raw data on an FTP site, named 
with UUIDs like 
4e283d36-61c4-11df-9a26-edddf420622d. I 
will under no circumstances make any attempt 
to provide analysis source code, 
documentation for formats, or any metadata 
with the raw data. When requested (and ONLY 
when requested), I will provide an Excel 
spreadsheet linking the names to data sets 
with published results. This spreadsheet will 
likely be wrong -- but since no one will be able 
to analyze the data, that won't matter.” 
- 
http://ivory.idyll.org/blog/data-management.htm
l
https://flic.kr/p/awnCQu
Summary and maybe conclusions?
• Data is important, and 
becoming more so for a wider 
range of the population
• Conclusions and knowledge are 
only as good as the data they’re 
based on
• Science is supposed to be 
reproducible and verifiable 
• It’s up to us as scientists to care 
for the data we’ve got and 
ensure that the story of what we 
did to the data is transparent
– So we and others can use 
the data again
– And so people will trust 
our results
http://scienceblogs.com/clock/2007/04/framing_politics
_based_on_scie_1.php
Thanks!
Any questions?
sarah.callaghan@stfc.ac.uk 
@sorcha_ni
http://citingbytes.blogspot.co.u
k/ Image credit: Borepatch 
http://borepatch.blogspot.com/2010/06/its-not-what-you-dont-know-
that-hurts.html
“Publishing research without data is simply 
advertising, not science” - Graham Steel
http://blog.okfn.org/2013/09/03/publishing-research-without-data-is-simply-advertising-not-science/