ZooCon 2015 (Part 2/2)

Next up was a stellar talk Alissa Bans, from Disk Detectives, which is investigating the formation of planets and planetary systems. Currently we understand that planetary systems form when gravity pulls particles together into a collapsing cloud of dust and gases until a hot core develops to form a star. The remaining particles start to rotate until eventually they form a flattened disk. Particles clump together to form planetesimals, which in turn eventually form planets. Details of this process, however, are sketchy, so astronomers have been hunting for these disks, birthplaces and homes of planets, for decades.

Disk Detectives identify two categories of disks, Young Stellar Objects (YSOs) and debris disks, that represent different stages of planetary formation. YSOs are young stars, generally less than five million years old, found in clusters, which remain surrounded by large amounts of gas, out of which giant planets like Jupiter form; in contrast, debris disks are found around stars older than five million years old, contain little or no gas and are littered with rocky and icy debris that look like asteroid belts. Both contain large amounts of dust created when debris collides, so they both shine brightly in infrared wavelengths. The NASA WISE (Wide-Field Infrared Survey Explorer) satellite mission has just finished mapping the entire sky in mid-infrared wavelengths, which provides the most powerful resource to date for finding dusty disks. Although the WISE images on Disk Detective don’t have detail (rather they will contain disks in a single point of light), radiation from the dust in the disk enhances radiation coming from the star.

Unfortunately, galaxies, asteroids, active galactic nuclei, and interstellar clouds of dust all emit and glow brightly at these wavelengths, which can easily confuse computer algorithms designed to automatically search for disks, so all disk candidates must be examined by eye to make sure they really are stars with disks, and not impostors. Computer programs can only detect what we tell them to measure and preselected, from 745 million objects captured by WISE, only the brightest 539,000 infrared objects, at wavelengths of 22 microns. Each of these will ultimately be examined by at least ten users, who respond to simple questions asking, for example, whether the view contains multiple objects, or whether objects go outside of the circle targeting them. For each image, the project provides a Spectral Emission Distribution (SED), in addition to example SEDs (provided by volunteers themselves!) to help classification of the more difficult images (Figure 1). Astronomers then examine the SED of each disk candidate and do follow-up observations to confirm which candidates are disks.

Figure 1: Example Spectral Energy Distributions of potential candidates provided by a volunteer.

Figure 1: Example Spectral Energy Distributions of potential candidates provided by a volunteer.

The Disk Detective team predict volunteers will find about 375 previously undiscovered disks, a list of which will be made public and allow astronomers to study the statistics of the disk population. In particular, the WISE data will help us learn about disks around stars that have evolved in areas of the sky not previously known to be forming stars. Astronomers will also use the data to inform targets for observation of some nearby disks at high resolution (and maybe even spot planets!) with NASA’s Hubble or the James Webb Space Telescope, which will replace it in 2018, to explore the origins of planets and life, amongst other science topics, and take more spectacular images.

What I really love and admire about this project is its focus on its volunteers and how clear it is about the nature and quality of the data it desires. They appear to have an amazing relationship with their volunteers, even building up a group of super users they consult for opinions and favourite images, following up classifications with ground-based telescopes and cataloguing example SEDs to aid classification (Figure 1). This has paid off, with impressive results that seem to have even discovered a new type of binary debris disk. This project also provides a precedent for using computer algorithms to preselect images for classification, which we intend to investigate with iMars imagery.

Tom Hart then gave us an overview of Penguin Watch, a project in collaboration with the charity Penguin Lifelines, that launched in September 2014 and has had around 2.5 million hits already from volunteers who love to look at pictures of penguins. (Personally I find it hard to believe that there are people who don’t!) Particularly over the last 50-60 years, the world’s sea bird population has declined. Although some threats in some areas of the world (such as fishing, climate change, harvesting and direct disturbance) are well known, the picture is much hazier than one might assume; the nature and magnitude of threats vary considerably between different places and impact different species to a greater or lesser extent. For example, the number of Gentoo Penguins is actually increasing, albeit at a slower rate than others are in decline. Some parts of this picture are unfortunately even hazier than others because of lack of data; some areas are simply more difficult to reach than others to capture the data for a whole host of reasons that I’m not going to even begin to go into here! Suffice to say that Antarctica is one such place. It’s 50 times the size of the UK, twice the size of Europe and twice the size of the US, and much of the continent is a logistical nightmare to reach. Scientists have traditionally been confined to bases in the most easily accessible places to carry out their research in Antarctica; even in these locations, the infrastructure surrounding bases can itself threaten sea birds, and scientific missions require a huge amount of planning a long time in advance. Result: our knowledge of the impact of climate change on the continent (in terms of air and sea temperature) is not as detailed as we now understand that it needs to be for such a large area. Bear in mind that it’s not just the climate that has changed; whilst whaling was a massive problem in the past, it is now krill fisheries in the Scotia Sea that pose the biggest threat. Krill is a significant constituent of a sea bird’s diet in this region. The ice releases a huge plume of life into the sea each summer when it melts, which requires few resources for the local populations to capture.

This is another project that has carefully considered why they need people power. The threats to sea birds are too complex to understand and manage with traditional approaches to research in Antarctica, limited to small areas over short periods of time. It requires a new approach to data capture and analysis, especially to have any impact on policy. For example, population counts for colonies are typically limited to those close to bases, and only capture birth/death rates, which miss details of how hard the penguins are having to work to raise chicks, how stressed they are and their productivity. The team’s challenge was to capture data in as many places as possible all of the time and to be able to analyse the data remotely. They set about taking any form of transport going, including chartering yachts, and being strategic in how they acquired equipment to distribute a network of American hunting cameras at, currently, around 60 sites, in addition to 40 run by the Australian Antarctic Division, which take around 20 images a day. They have adapted the cameras by coating them in resin, extending their life to five years, and hope to install fuel cell batteries that could potentially extend this to ten years. The latest set of equipment even records air pressure data of interest to climatologists, but already they record: individual nests; individual colonies; the laying and hatching of eggs to within a couple of hours; the volume of sea ice; and, the presence/absence of chicks/adults/predators etc.

These images are then classified by the volunteers online. And this is where I get my excuse to post a penguin picture (Figure 2).


Figure 2: PENGUINS!

This data has revealed surprising but basic information on how deep into winter the penguins stay in particular areas; krill fishermen have thus far justified their practice on the premise that they are not depriving penguins of an important food source. The timing of the breeding cycle has also been found to vary around the Scotia Sea, which can inform the operation and management of the fisheries. The dates of the arrival and departure of penguins is also noted in addition to huddling behaviour, which can be a proxy for the chicks’ health and level of independence. Computer vision experts have also already used the data to train an algorithm to identify an Adélie penguin! And, uh, they found that the Gentoo Penguin melt ice with their poo so that they can breed.

In the future the team wants to feed the data back to the volunteers, sponsors of the cameras and adopters of colonies through Penguin Lifelines, so that they can interact with the data and do their own analysis (by colony, time lapses etc) and research. This includes policy makers. They are also looking into capturing data with sensors activated by the penguins themselves, and not limited to 20 images a day, for example, which could capture chick survival and the arrival and departure of their parents, how long the parents spend at sea, and how hard they work. Simply put, they want the penguins to power the research. Treaties such as the Antarctic Treaty need data to be enforced and the data from Penguin Watch has just formed the basis of a report recommending trial closure of some fisheries, to see if it aids the recovery of penguin populations in some areas, and points to the potential for fishing practices to respond to the data more quickly in future. The data also played a significant part in the establishment of the world’s largest Marine Protected Area in South Georgia and the South Sandwich Islands.

Brooke Simmons finally gave a presentation of two halves. The first half reflected on the Zooniverse as a whole. She showed a graph demonstrating that the community of volunteers has continued to grow, at the same rate or higher, consistently over its lifetime; with each new project launch, volunteers appear loyal to existing projects whilst embracing new ones. When she saw this graph, Brooke became interested in why does the Zooniverse work? Why do people volunteer? What is their personal motivation and what do they hope to get out of it? What can the Zooniverse do better? These are especially important questions to ask now that researchers can start their own projects, without the help of the team. What experience can the team pass on? First, its community is a major factor in their success, many of whom interact with each other and/or scientists on projects’ discussion pages. Second, survey after survey has found that the main goal of volunteers is to contribute to science. She then narrowed in on the following sentence in Phil Marshall’s recent paper:

“The most productive citizen astronomy projects involve close collaboration between the professionals and the amateurs involved.”

The paper discusses what does and does not work in terms of how this collaboration manifests itself. Each project can and does take different decisions about how this collaboration takes place and for how long it is sustained, be it through discussion boards, blogs, social media. The authors took this a stage further and engaged the community in its publication, listing the names of the volunteers who contributed. But, will the statement above always hold true? What else is needed? How do we measure it? I won’t list them all here because you can read them in the paper she wrote with Joe Cox, which lists, calculates and defines many more metrics of success under the umbrellas of public engagement and contribution to science and combines them to present a success matrix (Figure 3) for Citizen Science, which she is interested in taking further to establish what can we learn from it? Are some methods of engaging volunteers more successful than others?

 Project positioning matrix from [Cox _et al_, 2015](http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7106413). The success matrix appears to show a positive trend relating to the positioning of Zooniverse projects and their size, indicating that projects receiving more classifications tend to be more successful.

Figure 3: Project positioning matrix. The success matrix appears to show a positive trend relating to the positioning of Zooniverse projects and their size, indicating that projects receiving more classifications tend to be more successful. (Cox et al, 2015)

This followed Swanson’s talk nicely and reminded me of a conversation I had the day before ZooCon about whether I am designing a user interface to obtain accurate results or engagement.  I don’t personally see them as being mutually exclusive, even though I do recognise that a retired professor, for example, might require a different user interface to a schoolchild. When you consider, though, that someone engaged with a project can develop proficiency over time, the notion of “expertise” becomes very blurred. This requires another blog post sometime and can probably only be unpicked at the project level. I would love your thoughts below this post.

The second half of her talk focused on the development of project in response to the earthquake in Nepal. It was only the second project after Snapshot Serengeti that was developed from start to finish with the new Panoptes development platform. This project reflects her preference for moving away from the term Citizen Science to talk about People-Powered Research. Planet Labs provided the team with satellite imagery, who consulted with the UK Charity Rescue Global to find out how they could help. Rescue Global were interested to know if there were any towns that were completely cut off and, as a result, had not yet been reached. By combining the established images of the area around Kathmandu with the crisis mapping information that was coming in they identified two towns that had been cut off.  This impressed at the highest levels of NATO whose Director General of Military Staff is quoted as saying:

“I also noted the organisation’s testing of a new approach to the use of people, computers and social media, to identify Search and Rescue targets of interest.”

That someone so high up should recognise the impact of what they did, so explicitly, boggles my mind.

Her take-homes were:

  1. Engage the community for success: if you don’t do this well then, yes, you’ll still get science out, but if you engage the community you’ll get better science out and faster;
  2. Publish papers with the volunteers is a good metric for this, which should happen more;
  3. Talk is powerful and remains a focus of the Zooniverse.

To cap off what was already a super day, Brooke saved me hours of work because she explained to me (in the pub afterwards!) how to implement some functionality in my project that I didn’t think existed yet. THANKS BROOKE. And thanks ZooCon. What I heard humbled me and reassured me that focusing on the science and the user experience is a good thing, and the questions that have been bothering me might not be so stupid after all. I hope we can help each other in our quest for answers.


ZooCon 2015 (Part 1/2)

That’s right. I have finally managed to shackle the first half of my ZooCon thoughts to the metaphorical mast before they fly away. Hold on tight now for the first of two posts. But first I should admit that I attribute a percentage of my excitement about ZooCon to the fact that it took place on Keble Road, Oxford. Yes, Keble Road. The road named after, and adjoining, the college at which I studied as an undergraduate. I don’t need any excuses to return to Keble but, since ZooCon gave me one on a plate, I instantly booked to stay there the night before. I tell you this only because you cannot even begin to imagine my face, a cocktail of disbelief and glee, when the girl on reception presented me with the key to the very same room that I lived in for two years. What a way to start my ZooCon weekend. A little backwards time travel before I was catapulted into the future the next day.

Becky Smethurst kicked us off with a talk of two halves. First she reported a live Citizen Science project from earlier this year on the BBC’s Star Gazing Live (i.e. classification against the clock!) to find supernovae. From what I understand, when a star explodes and forms a supernova, for a very brief period of time it shines with the same brightness before it diminishes; the apparent brightness of a supernova, as a proportion of their full brightness, therefore indicates how far away they are thus how much the universe has expanded, and thus the age of the universe. To capture how bright a supernova is at its birth would require action on a large scale in a limited period of time. For three consecutive nights, Brian Cox asked viewers of Stargazing Live to trawl through 100,000 images taken by the SkyMapper telescope in Australia and classify them before the sun set in Chile, where a telescope would be turned to focus on those found. More than 40,000 volunteers provided almost 2 million classifications for the images over the three days to find four, ultimately confirmed, supernovae, one of which was a Type 1A, which scientists use to age the universe. I was impressed enough that volunteers found one new supernova, but even more impressive is that this one data point resulted in the calculation of the age of the universe to within 200 million years of the agreed answer (14.03 billion years)! That, my friends, is the power of the crowd. Not just the power of Brian Cox. It also raised the question of how we communicate the science that results from classifications and how we can improve this. The iMars team is already concerned about how we will manage and give feedback to volunteers, for example how we are going to administer discussion board posts. Second, she told a story about how she had taken ten photos of Orion’s belt on a windy evening in Hawaii (a word that always grabs the attention of my ears), each ten seconds apart. She then used open source software to stack the photos and produce one image showing the movement of the sky in that time. BBC viewers used the same process and sent in 790 photos which, stacked together, produced the image in Figure 1.

Figure 1: Orion. As captured by BBC Star Gazing Live viewers.

Figure 1: Orion. As captured by BBC Star Gazing Live viewers.


I don’t know about you but this image stole my breath. You can read all the details and download it for yourself here.

Second, Alexandra Swanson talked about Snapshot Serengeti, a personal favourite project for which volunteers identified animals in 1.2 million images taken by 225 camera traps across a 1,125 km squared of the Serengeti National Park in Kenya, to improve our understanding of the migration and interaction of its 1.6 million wildebeest and zebra. The project published its first paper recently, which got an unusually high amount of publicity; this is atypical of Citizen Science projects, which tend to receive more publicity for their launch than their results. Their paper swam against this tide to report what happened to the 10.8 million classifications that over 28,000 registered and around 40,000 unregistered volunteers contributed. It was fascinating to hear how she’d used the project’s data to explore the certainty of individual classifications and, more specifically, how many classifications were needed to be certain that any one classification was correct for different species. Because I’m a nerd and we have similar decisions to make on our project, I was interested to note their criteria for removing images from circulation:

  1. five consecutive “nothing here” classifications;
  2. ten non-consecutive “nothing here” classifications;
  3. ten matching (consecutive or non-consecutive) classifications of species or species combination;
  4. 25 total classifications (whether in agreement or not).

For each image, the team applied a plurality algorithm to produce a “consensus dataset”. Next, for each species, the team calculated the level of agreement between classifications, or “evenness score”; a score of 0 indicated total agreement (i.e. a species that was relatively easy to identify) and a score of 1 indicates total disagreement between classifications (i.e. a species that was relatively easy to identify). Finally, five experts produced 4,428 classifications from a subset of 4,149 images, which could be compared against the combined intelligence of the crowd. Remarkably, 97% of the volunteers’ species classifications, and 90% of their species counts, matched those of the experts. This data further allowed them to delve deeper into the 3% that were incorrect and find that there were two main types of errors: 1) false negatives, when volunteers missed an animal that was there, and 2) false positives, the flip side of the same coin, where volunteers classify something that isn’t there (which is relatively common for rare species such as rhinos that people get excited about). The analysis found that species with a high rate for one type of error, they had a low rate for the other. The interesting implication of this is the potential for dynamic removal of images from projects to help manage their increasing volume; researchers might consider some images to be considered correctly classified after only three matching classifications whereas other species may be more difficult to identify and require more matching classifications i.e. how many pairs of eyes do we need to look at an image? As an extension of this, researchers may also require different levels of classification accuracy, depending on the question they are using the data to answer. It’s worth pointing out that Greg Hines has also explored the weighting of classifications, not only by species, but by individual accuracy and quantified the increase in certainty obtained with each individual classification. Figure 2 shows the graph from this paper that illustrates that rarer species require more eyes for the same level of certainty.

Figure 2:  The number of users required to classify a photo for an accurate result increases according to the level of disagreement between their classifications (Hines et al, 2015).

Figure 1: The number of users required to classify a photo for an accurate result increases according to the level of disagreement between their classifications.

It sparked a conversation in the ensuing break between me and my project mate about how we will measure performance in our project and what we can use to assess the accuracy of classifications. As I said in my last post, the team at UCL’s Mullard Space Science Laboratory is developing an algorithm at the same time that we are developing the Citizen Science platform for iMars. Because both techniques are new we start with the hypothesis that the algorithm will successfully filter for images that show geological change, so that volunteers can do the more fun and interesting job of defining the change, which it is more challenging to code into the algorithm. How we will determine the successful performance of either, however, is still to be determined itself!

After the break Victoria Van Hyning presented the wide range of Humanities projects the Zooniverse team is working on. The main challenge that her work addresses is that computers cannot read handwriting and there aren’t enough experts in the world to create the data to train a computer and garner such rich insights into life/the world before print. She gave us a tour of her current and future plans.

  • Operation War Diary: 1.5 million pages of war diaries, which detail daily events on the front line, the decisions that were made and activities that resulted from them. The key learning of this project was not to make the tasks too complex.
  • Science Gossip: This is a collaboration between ConSciCom (an Arts and Humanities Research Council project investigating the role of naturalists and ‘amateur’ science enthusiasts in the making and communication of science in both the Victorian period and today) and the Missouri Botanical Garden who provide material from the Biodiversity Heritage Library, a digital catalogue of millions of pages of printed text between the 1400s and today related to the investigation of the natural world. Since March 2015, 4,413 volunteers have tagged illustrations and added artist and engraver information to 92,660 individual pages through this website; this will help historians identify why, how often, and who made images depicting a whole range of natural sciences in the Victorian period. This project has the potential for the development of graphical search functionality for these catalogues.
  • Ancient Lives: over 1.5 million transcriptions of the Oxyrhynchus Papyri. The neatest thing on this project is the Greek character keyboard (Figure 3), which users can become proficient without any expertise.
Figure 3: The Greek Character keyboard of the Ancient Lives project.

Figure 3: The Greek Character keyboard of the Ancient Lives project.

Future projects include:

  • Diagnosis London: a future project with the Wellcome Trust and University of Leicester, which will again go back to Citizen Science of the 19th Century and look at public health records.
  • Anno.Tate: a project with Tate UK to transcribe artists’ notebooks, diaries etc.
  • Shakespeare’s World: in collaboration with the Folger Shakespeare Library in Washington DC, seeks volunteers to transcribe records of life when Shakespeare was alive.
  • New Bedford Whaling Logs: a spin off from Old Weather that targets climate data and social history.
She is getting excited that, over the next 12 months, the new web-based Zooniverse platform, Panoptes, is going to be adapted for the Humanities to enable a granular approach to classification, which will facilitate: different levels of participation (so that volunteers can dip in an out of the same document without having to abandon it completely); mitigation of fatigue, and; construction of algorithms. Volunteers to will also be able to create what she called crib sheets of letters and/or words as their own personal reference, which has the potential to improve their consistency and proficiency and combine the crowd’s knowledge to alleviate any common misclassifications. Linguists are interested in taking this data further to examine the evolution of spelling and punctuation, which is often inconsistent. Audio and visual classifications are also something the Humanities arena might explore, with applications within domains such as archaeology.
The second half of the day was no less interesting so I intend to write it up separately in due course. (I haven’t got to the penguins yet!) Until then I would love to hear your thoughts, or recommendations for reading, on any of the issues I have highlighted in bold. They all represent elephants that have been stamping around in my head since I started out on this project, so I was very grateful to hear them aired at ZooCon and intend to remain mindful of them as my project progresses.