Amy Whitehead's Research

the ecological musings of a conservation biologist

1 Comment

Nigel Latta (& me) in Antarctica (on the telly)

Last season, when I was in Antarctica, we had a visit from Nigel Latta, a criminal psychologist turned comedic documentary-maker who makes humorous shows about serious subjects. He was sent to Scott Base, Antarctica to live among the scientists and to discover what life on the Ice is really like and whether it holds the key to our futures. The first episode airs on March 4th at 8:30pm on TV One (in NZ only) and promises to be an entertaining ride.

Filming with Nigel and his team was one of the more entertaining media experiences I’ve had – we defused bombs in a tent, counted (& recounted & recounted & recounted & ….) a constantly moving mass of penguin chicks by trying to name them all, discussed the parenting abilities of penguins and their monogamous (or lack thereof) relationships. I watched (with delight) as Nigel got shat on by a South Polar skua and attacked by angry penguins, we restaged Shackelton’s famous dinner photo and generally had a rollocking good time. I look forward to seeing his thoughts of life On Thin Ice*.

If you miss it (& happen to live in NZ), you should be able to watch it OnDemand.

* Not that I will actually be able to watch it, what with not being in NZ and the restrictions of TV OnDemand….but I’d love to know what you think of it.


Copying files with R

Following on from my recent experience with deleting files using R, I found myself needing to copy a large number of raster files from a folder on my computer to a USB drive so that I could post them to a colleague (yes, snail mail – how old and antiquated!).  While this is not typically a difficult task to do manually, I didn’t want to copy all of the files within the folder and there was no way to sort the folder in a sensible manner that meant I could select out the files that I wanted without individually clicking on all 723 files (out of ~4,300) and copying them over.  Not only would this have been incredibly tedious(!), it’s highly likely that I would have made a mistake and missed something important or copied over files that they didn’t need. So enter my foray into copying files using R.

R has a nice set of file manipulation commands in the base package that make it really easy to find out if a file exists (file.exist), rename files (file.rename), copy them to a different directory (file.copy) or delete them (file.delete).  Basically, you point R at the directory where your files live, identify the files that you want to manipulate and then tell it what you want to do to them.  In my case, I wanted to identify all Geo-tiff formatted rasters whose filenames ended in “SDM” and copy them to a new directory.

# identify the folders
current.folder <- "C:/Where my files currently live"
new.folder <- "H:/Where I want my files to be copied to"

# find the files that you want
list.of.files <- list.files(current.folder, "SDM\\.tif$",full.names=T)

# copy the files to the new folder
file.copy(list.of.files, new.folder)

This will chug away for a bit (time for a coffee, anyone?) and then produce a vector of TRUE/FALSE the same length as your list.of.files that identifies whether it was able to copy them or not.  Pretty simple really.  The only tricky bit can be getting the regex pattern right to pull out the files you want to manipulate. There are many regex guides online – I often head over to Rubular to test a pattern if I’m having trouble (note that it’s actually designed for Ruby and not R but they seem similar enough that it has always worked so far).


A day in the life of a penguin wrangler

By now you’ve probably figured out that I have something to do with penguins and occasionally disappear for months on end into the wilds of Antarctica. I often get asked what we actually do all day when we’re out in the field but my usual response – “oh, you know, count penguins and stuff” – isn’t really that satisfactory. So I’ve tried to document a typical day in the field at Cape Bird…

08:001 Bleep bleep bleep. The alarm goes off and I yank on the piece of string to remove the tightly wedged piece of cardboard in the window and let the daylight stream in. One of the problems of 24 hours of daylight is that it’s hard to block the light out. However, we’ve managed it to overcome that problem so well that we now struggle to wake up because there are no daylight cues to indicate that it’s morning. Hence the string. Eventually dragging myself out of bed, I stumble blindly out to the kitchen and get the coffee pot going. The boys are busy burning the toast while trying to identify a seal drifting past on the sea ice. At first glance, it looked like it might be a leopard seal but, on closer inspection through the binoculars, it turns out to be a weddell seal. They’re pretty common in this neck of the woods, so attention quickly turns back to rescuing breakfast. Then there’s a quick discussion about the plan for the day, which quickly digresses into some random conversation totally off topic!

10:00 We start to layer on the gear in preparation for going outside. It’s not a particularly cold day outside (maybe hovering just below 0°C), so no need to go overboard with the layers. Just a pair of polar fleece trousers, insulated overalls, merino t-shirt, merino longsleeved top, fleece sweater, primaloft jacket and windstopper jacket, topped off with a fleece-lined woollen hat, a neck buff, sunglasses and a pair of possum-merino gloves. Oh, plus a pair of thick woollen socks and insulated boots. It takes a while to get ready! Then it’s outside to start the day’s work.

Peter heads off to start bandsearching – walking the edge of the subcolonies and looking for birds with flipper bands. Once located, he’ll record the band number and the bird is up to. This is usually a straightforward process but there is always someone who flaps their flippers or turns so that you can’t read the band. This usually turns into a frustrating game where you and the penguin dance around each other for five minutes (or more!) until eventually you manage to outwit the bird. Once is tolerable but when you’re doing this for 8 hours or more a day, it can get pretty tedious.  On the other hand, spending this much time walking around the colony means you get to see a lot of interesting things and take a lot of pictures.

Hamish & I head down the hill to the weighbridge colony. This small subcolony of approximately 200 nests is surrounded by a mesh fence, with the only access point into the colony across a bridge. This bridge hides a set of scales that weigh birds as they cross, as well as recording whether they coming or going. Every couple of days we download the data from the weighbridge and record the status of the marked nests – which adults are present and how many eggs or chicks they have – as well as the total number of adults and chicks in the subcolony. This information is used to work out some pretty interesting information about how long adults are out at sea and how much food they bring back for their chicks.  It also lets us compare how the colony is doing from year to year.

11:00 Next we head off to count adults and chicks at two more reference colonies. Unlike the weighbridge colony, these subcolonies are not surrounded by a fence. We monitor 30 marked nests in these colonies from early November when the eggs are laid until the chicks creche in mid January. While we’re walking between colonies, we find a freshly dead chick that has just been killed by a skua. Penguin colonies are filled with death and destruction and it can take a bit of getting used to. But it can also offer some unique opportunities. All dead chicks in reasonable condition (i.e. they are still whole and not super stinky) are weighed, measured and then dissected. Looking at the stomach contents gives us some idea about what’s happening out at sea. This chick has been fed mostly krill but the grey mush suggests that adults are also starting to bring in silverfish. This tends to happen later in the season when the chicks are about three weeks old.


12:00 Chick counting done for the moment, it’s off to do some actual penguin wrangling. Since we arrived in mid December we’ve been catching banded adults with chicks and attaching small devices called accelerometers. These collect information about a bird’s foraging behaviour: how long they spend out at sea on one foraging trip, how often and how deep they dive, and the types of movements they are making while under the water. This morning’s task is to look for birds with accelerometers that that have returned from sea. Foraging trips typically last anywhere from 2 – 8 days, depending on the conditions out to sea. This year they seem to be at the longer end of the scale, suggesting that it’s taking longer for birds to find enough food to feed themselves and their chicks.

We aim to recatch birds when they have returned to their nests and are happily brooding their chicks. This is for three reasons:
1) it hopefully gives the adult time to feed their chick(s) before we turn up to disturb them; 2) it’s by far the easiest way to find them (imagine looking for a penguin with a small black device attached to its black back amongst ~40,000 other black-backed penguins!); and 3) adults are much easier to catch if they are sitting on a nest. Grabbing a penguin off a nest is much easier than you might expect – you simply weave your way through the surrounding nests (getting thoroughly pecked and beaten by the neighbours in the process) and pick them up. A second person collects the chick(s) and leaves a cover over the nest to stop the neighbours stealing all the rocks while you’re away. Then it’s onto the business of taking a blood sample, removing the accelerometer and taking a range of measurements such as weight and flipper length. Once the adult has been processed, we weigh and measure the chicks before marking them with a temporary plastic tag. These individually-numbered tags mean that we can follow the growth and survival of these chicks throughout the season. Once we’re done, we release the adult and the chicks back on the nest. The whole process takes less than 20 minutes for each bird and is relatively stress-free for both penguin and wrangler. This morning we manage to retrieve three of the ten accelerometers we have out, which is a pretty good haul.


14:00 Blood samples and accelerometers in hand, it’s time to head back to the hut and process the samples. Vials of blood are loaded into the centrifuge and sent spinning merrily on their way, slides are fixed in alcohol, feather samples are stored away under the bench and the first accelerometer is plugged into the computer to download. It must be time for lunch! This year we’ve become masters of the scone and today’s lunch includes a healthy dose of the chocolate and date variety, with a side of toasted cheese sandwich and some dried apple slices.

It can take up to 40 mins to download the data from each accelerometer. Given that we have three to do today, we have plenty of time to do our daily mammal survey. This involves staring out the window for an hour every day, scanning the beach and water for mammals. We often see Weddell seals on the beach and some days will be treated to a Antarctic minke whale or a pod of orca swimming past. Alas, today is not one of those days and it’s a very long hour staring out the window with binoculars without seeing a single mammal. At least the view isn’t too shabby.

16:30 Blood processed, accelerometers downloaded, lunch eaten and the lack of mammals surveyed, it’s time to head back outside to finish the rest of the day’s fieldwork. We hope to put the three accelerometers that we retrieved this morning back out on some new birds. We have a list of target nests with banded birds of known ages, so it’s simply a matter of walking around and checking those nests until we find somebody at home. Once we’ve located a victim customer, it’s a matter of grabbing the bird and its chicks off the nest and attaching the accelerometer. We do this using thin strips of tape that are layered under the feathers, a technique that is robust enough to stay on for up to three foraging trips. A nice, non-wriggly bird will take about 5-6 minutes to process and return to the nest. A wriggly bird may take a bit longer and will likely result in some strong words from the handler and the tape sticker! Oh how we hate the wriggly birds!

18:00 Three accelerometers deployed, it’s time to go and count some more penguins. As the season progresses, the number of adults at a subcolony decreases as the chicks are left to creche. This leaves the chicks particularly vulnerable to skua predation. We have some small subcolonies of penguins just below the hut that are quite isolated and surrounded by skua nests. These colonies drop in size quite dramatically as chicks start to disappear into the mouths of skuas. [skua swallowing chick photo] The skua effect can be so severe that the smallest of these colonies rarely manages to fledge any chicks. Every couple of days we count the number of adults and chicks to document the skua-induced declines. Today everyone seems to be well and accounted for but there are six hungry skua stalking the edge of the colony, so I suspect that the next counts will be somewhat lower.

18:30 Heading back up to the hut, we dump our packs and pick up some shovels and a wheelbarrow, and head over to the snowbank. Our hut has no plumbing system, so we have to collect snow to melt for water. A couple of times a week, we shovel snow into the wheelbarrow and dump it in a large container in the hut where it slowly melts. We also have to carry all our waste water (including pee) in buckets down to the sea. As such, washing is an event that is much less regular than would be socially acceptable in the real world! Luckily, we all smell equally of penguin.


20:00 “Scott Base, Scott Base, this is Cape Bird”. Every night we check in with Antarctica NZ at Scott Base on the VHF radio to let them know we’re okay and haven’t been eaten by skua or drifted away in a boat2. It’s our only opportunity to talk to someone outside of our group of three, hear some news from the outside world and get a weather report. Then it’s time for dinner – usually some sort of stirfry/pasta/curry dish. Today it’s a variation on lamb stirfry, followed by a special treat – passionfruit cheesecake! As far as field food goes, we have it pretty good out at Cape Bird. We have a freezer, so we can have frozen meat and vegetables, and there is a large well-stocked pantry with most of the things you need. Like most field huts though, Cape Bird is the place the food goes to die and expiry dates are treated more as a game (“Guess how many years since tonight’s dinner ingredients expired!”3) than a guideline for edibility. And you definitely start to crave fresh food – what we wouldn’t give for a simple salad.

21:00 Fed, watered and dishes washed, we all sit down at our computers and enter the day’s data, download photos, and work out a plan for tomorrow. This year Peter is trialling a new approach to data entry by entering it directly into a tablet in the field. It seems to be working well and saves having to enter up to 14 pages of data at the end of the day but there is still room for improvement. I spend some time in the lab sorting out the bleeding kit; finding more needles, pre-labelling sample bags.

00:00 How did it get to be this late already?! It’s hard to keep track of time when it never gets dark outside and we often find ourselves working much later than we intended. The light at this time of day is often stunning and it’s tempting to head back outside to take pictures. Tonight we set up the timelapse camera to try and capture the moving sea. Then it’s time to wedge the cardboard back in the window and drift off to sleep, counting penguins…


1 This hour of rising may be somewhat optimistic and is purely here for the benefit of my boss (who I’m hoping doesn’t read footnotes!). Even this is much later than the season when he was there but that’s what happens when you leave me in charge!

2 This happened to a group of researchers at Cape Bird in the 1970s. It took five days before they were rescued. Needless to say, we are no longer allowed boats!

3 I think 2004 was the oldest expiry date encountered this year, although there were a couple of items that I think actually pre-dated expiry dates!


Converting shapefiles to rasters in R

I’ve been doing a lot of analyses recently that need rasters representing features in the landscape. In most cases, these data have been supplied as shapefiles, so I needed to quickly extract parts of a shapefile dataset and convert them to a raster in a standardised format. Preferably with as little repetitive coding as possible. So I created a simple and relatively flexible function to do the job for me.

The function requires two main input files: the shapefile (shp) that you want to convert and a raster that represents the background area (mask.raster), with your desired extent and resolution. The value of the background raster should be set to a constant value that will represent the absence of the data in the shapefile (I typically use zero).

The function steps through the following:

  1. Optional: If shp is not in the same projection as the mask.raster, set the current projection (proj.from) and then transform the shapefile to the new projection ( using transform=TRUE.
  2. Convert shp to a raster based on the specifications of mask.raster (i.e. same extent and resolution).
  3. Set the value of the cells of the raster that represent the polygon to the desired value.
  4. Merge the raster with mask.raster, so that the background values are equal to the value of mask.raster.
  5. Export as a tiff file in the working directory with the label specified in the function call.
  6. If desired, plot the new raster using map=TRUE.
  7. Return as an object in the global R environment.

The function is relatively quick, although is somewhat dependant on how complicated your shapefile is. The more individual polygons that need to filtered through and extracted, the longer it will take.

shp2raster <- function(shp, mask.raster, label, value, transform = FALSE, proj.from = NA, = NA, map = TRUE) {
    require(raster, rgdal)

    # use transform==TRUE if the polygon is not in the same coordinate system as
    # the output raster, setting proj.from & to the appropriate
    # projections
    if (transform == TRUE) {
        proj4string(shp) <- proj.from
        shp <- spTransform(shp,

    # convert the shapefile to a raster based on a standardised background
    # raster
    r <- rasterize(shp, mask.raster)
    # set the cells associated with the shapfile to the specified value
    r[!] <- value
    # merge the new raster with the mask raster and export to the working
    # directory as a tif file
    r <- mask(merge(r, mask.raster), mask.raster, filename = label, format = "GTiff",
        overwrite = T)

    # plot map of new raster
    if (map == TRUE) {
        plot(r, main = label, axes = F, box = F)

    names(r) <- label

Below is a trivial example based on some readily available data in the maptools and biomod2 packages. Here I load a raster and a shapefile that represent our background of interest and foreground feature, respectively, and then plot them.


## example: import world raster from package biomod2 and set the background
## values to zero
worldRaster <- raster(system.file("external/bioclim/current/bio3.grd", package = "biomod2"))
worldRaster[!] <- 0
plot(worldRaster, axes = F, box = F, legend = F, main = "The world")

# import world polygon shapefile from package maptools
data(wrld_simpl, package = "maptools")
plot(wrld_simpl, add = T)

The World

One of the nice things about working with shapefiles in R is that you can subset the data based on attribute data the same way that you would any dataframe. This is really useful when combined with the shp2raster function as it means that we only need to convert the parts of the shapefile that we are actually interested in. For example, you may wish to create separate rasters for different landuse types that are contained in one shapefile as polygons with different attributes. You can see this in the trivial example below where I create two rasters from our world polygon data that select specific countries to convert based on the attribute NAME in the wrld_simpl shapefile.

# extract all Australian polygons and convert to a world raster where cells
# associated with Australia have a value of 1 and everything else has a
# value of 0.
australia <- shp2raster(shp = wrld_simpl[grepl(c("Australia";), wrld_simpl$NAME), ],
    mask.raster = worldRaster, label = "Where Amy currently lives", transform = FALSE, value = 1)

## Found 1 region(s) and 97 polygon(s)


# extract Australia, NZ & USA and convert to a world raster where cells
# associated with these countries have a value of 3 and everything
# else has a value of 0. <- shp2raster(shp = wrld_simpl[grepl(c("Australia|New Zealand|United States"),
    wrld_simpl$NAME), ], mask.raster = worldRaster, label = "All countries Amy has lived in",
    transform = FALSE, value = 3)

## Found 5 region(s) and 384 polygon(s)

NZ, USA & Australia

Clearly these are quite trivial examples, where the raster and polygon layers don’t match very well (I mean, what is that blob of four pixels that represents my home country?!). But you hopefully get the general idea. In this case, I haven’t needed to do any transformation of the projections.

Below is an example where I was interested in creating a layer that represented protected areas within the Hunter Valley, Australia, for a conservation planning exercise. The available shapefile represented four different reserve types (National Parks, Nature Reserves, Regional parks and State Conservation Areas). In this case, the initial mask and polygon layers were in different projections, so I converted the shapefile to the correct projection by using transform=TRUE and proj.from and in the shp2raster call. I also wanted to extract a subset of the reserve types contained within the shapefile that related to National Parks (NP) and Nature Reserves (NR). The protected areas of interest needed to have a value of 3, while the background raster has a value of 0. Unfortunately I can’t supply the original layers but a real example of the code in action and the output map are below.

# set relevant projections
GDA94.56 <- CRS("+proj=utm +zone=56 +south +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs")
GDA94 <- CRS("+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs")

LH.mask <- raster("LH.mask.tif")
# set the background cells in the raster to 0
LH.mask[!] <- 0

NPWS.reserves <- readShapePoly("NPWSReserves.shp", proj4 = GDA94)

# convert the NPWS.reserves polygon data for National Parks &amp; Nature
# Reserves to a raster, after changing the projection.
NPWS.raster <- shp2raster(shp = NPWS.reserves[grepl(c("NP|NR"), NPWS.reserves$reservetyp),],
    mask.raster = LH.mask, label = "National Parks & Nature Reserves", value = 3,
    transform = TRUE, proj.from = GDA94, = GDA94.56)

## Found 111 region(s) and 837 polygon(s)

Hunter Valley

The function should work with both polygon and polyline data. You can also use point data if you buffer it by a small amount first to create tiny polygons around the points. I’d pick a buffer size that makes sense relative to the resolution of the mask.raster.

Go forth and convert! Feedback on improvements definitely welcome – please leave a comment.

1 Comment

Wildlife Wednesday: Nesting Adélie penguins

Unlike their cousins the emperor penguin, and despite what most people expect, Adélie penguins do not nest on the ice. In fact, they need ice-free areas of gravel to breed on over the summer months. This is largely because they build nests of small stones in which to lay their eggs. Stones are a valuable resource in a penguin colony and it’s not uncommon to see birds stealing stones from their neighbour’s nest when they think no one is looking. This stealing behaviour means that the nests are often very evenly-spaced, at a distance that is slightly beyond the reach of the neighbour.

Occasionally someone will branch out into alternative building materials. For some reason, this bird decided that the dessicated remains of dead penguin chicks are better than stones. Unfortunately, the ladies didn’t agree and he wasn’t successful in finding a mate.  Sometimes it doesn’t pay to be different!

1 Comment

Wildlife Wednesday: Adélie penguins

I know I’ve featured Adélie penguins on Wildlife Wednesday before.  But it’s been a while and I have seen quite a few of them since last time. So over the next few weeks, I’m going to feature a series of Adélie penguin photographs that highlight different aspects of their natural history.  Please forgive me my penguin indulgences!

Adélie penguins at Cape Bird feed mostly on krill and silverfish, diving down to over 100 m to catch their prey. They have backwards pointing “spines” on the roof of their mouths and their tongues to stop the prey items escaping once they have managed to chase them down.


Remotely deleting files from R

Sometimes programs generate a LOT of files while running scripts. Usually these are important (why else would you be running the script?). However, sometimes scripts generate mountains of temporary files to create summary outputs that aren’t really useful in their own right. Manually deleting such temporary files can be a very time consuming and tedious process, particularly if they are mixed in with the important ones. Not to mention the risk of accidentally deleting things you need because you’ve got bored and your mind has wandered off to more exciting things… watching orca swim past from the hut window

…like watching orca swim past the hut window!

I had exactly this problem a few months ago when I had ~65,000 temp files from a modelling process that were no longer needed, inconveniently mixed in with the things I needed to keep. Clearly deleting these files manually wasn’t really going to be an option. There are a number of ways to tackle this problem but R provided a simple two-line solution to the problem.

The first step is to identify if there are any patterns in the file names that will help you remove only the files that you want to delete (and not the really important ones!). Then construct a regular expression that matches the pattern. A handy reference guide to regular expressions can be found here. In my case, all the file names to delete contained a text string followed by this ".xxxx" pattern, where x is a number (i.e. Iamafiletodelete.1234.csv). Therefore, my regex pattern looked like this: ".[0-9]" (but see note below)*.

Then we can simply point R at the appropriate folder using dir(), identify the list of offending files, and delete them using file.remove(). Note that this has the potential to go horribly wrong if you aren’t careful! Make sure that you check very carefully that the pattern recognition selects only those files that you want to delete before you delete anything! This will result in a permanent delete (i.e. no rescuing things back from the recycle bin) and cannot be undone! <- dir("C:/the folder I want to delete from/",pattern=".[0-9]",recursive=T,full.names=T)

You can also see if the files exist either before or after you delete them as a useful check to make sure it worked.

Go forth and delete things but use at your own peril!

*As Patrick pointed out in the comments, the way that I have written the regex pattern technically isn’t correct. While it worked, it could also have gone horribly wrong! A good example of why checking the selected file strings before you actually delete the files is a very good idea. Patrick’s suggestion for the correct pattern is “.\d{4}.". He also points out that you can test regex code at Rubular, which seems like a very good idea!

Related posts 

Get every new post delivered to your Inbox.

Join 777 other followers