Translating between Matlab and R

I mentioned in a previous post that I’d write a guide about translating Matlab code to work in R, so that others can avoid the same mistakes I made. This should also function as an R users guide to learning Matlab syntax and vice versa. I hope some people find it useful!

Full article below.


Continue reading

Translation Issues

Where on earth did the time go? One minute I was looking at ice sculptures and wading through snow drifts and now the snow is gone (mostly) and various wildlife has emerged from hiding.

Look, beavers!

I’ve also managed to get out on the water myself. And eat bacon while doing it. Canadian bacon is Different.

riverbaconRiver bacon!

Perhaps the main reason I’ve lost track of time in the last few weeks is that whether I’ve been awake or asleep, this keeps on flashing in front of my eyes.


I haven’t been able to escape it. My office desktop has looked like this on and off for the whole month. I hope to later show what this has resulted in, but in the meantime I’m going to moan about the amount of pain it’s caused me.

The main source of trouble was that this code was originally written in Matlab. I decided to save us from having to acquire a Matlab license by translating the many scripts that make up the code into R.

Initially this was tedious. There are enough syntax differences (not to mention different names of functions etc) between R and Matlab that this required me to go through the code line by line. Then, even when I done this a number of errors arose simply due to the differing ways that the two programs handle data. I’ll post a guide based on what I learnt in separate post, featuring less pictures of beavers.

Much hair pulling later I got the code running, fed it my data and got a result. These results were consistent with some previous findings obtained using simpler methods. So far so good.

I then decided that instead of feeding my data to the code all in one go, it would be useful to give it one day at a time and then collate the results. “Fine” I thought. “Just modify my overarching processing code, no trouble”.

I was wrong.

c149182a696e73890de876f3d392e2da(Found via googling evil Matlab)

Once again the way in which the two programs handle data required me to make a lot of modifications to the various scripts. Cue more hair pulling. I should also mention that I’ve written this code to be run in parallel, utilising all of my computers cores to increase speed, which means R’s normal debugging tools don’t work.

Finally I got the code to run again and got a result. However, something had changed. A previously suggested relationship had completely reversed in direction. Was this simply due to the new way of feeding the data in? Or was it due to a bug in my code? Or due to me deleting some faulty data? I ran the code again using the original way of processing the data.

Even using parallel processing, this code can take anything from several hours, to all night to run. This meant that getting results was a slow process. So after waiting several hours for the code to run again using the original data processing, once again I got results.

The relationship had flipped direction in these results too.


This suggested that the changes I’d made to accommodate the new data processing method had resulted in a COMPLETELY DIFFERENT RESULT. On the one hand, this was good. It meant that the biologically unrealistic result was due to my error rather than a fundamental problem with the methods. On the other hand, this is the sort of thing that can wake a scientist up at night screaming. A series of small changes in the way data was analysed leading to completely misleading results. In this case we’d caught it before we went too far, but if we’d approached this naively it might have been very easy to miss.

So, now I needed to work out which of my changes had caused this change. Luckily I save all my working files in Dropbox, which keeps a backup of all previous versions. I found a word document containing graphs I’d made to show my supervisor before I’d made changes and reverted all my code to a date before then. Then one by one I reinstated my changes.

In the end I pinned it down to one file. In that file, one line of code.


One line of code had resulted in huge, significant changes to my final result. As I said, the stuff of nightmares.

In the end I stripped out all the changes I’d made and carefully rewrote the scripts to deal with the new method of data processing. So my tale of woe has a happy ending, the code now works and perhaps I’ll even have some results soon. For everyone who made it this far, here is a view of Gatineu Park:


Chickadees are hard to take photos of



I think I’ve been in Canada almost two months now. It’s hard to say as the initial turmoil of moving to a new country slowly changes into everyday routine. This generally involves getting up, trying to do some science until some time in the evening and drinking far too much tea.

I’ve met my study species properly now. A few weekends ago we went for a walk at a place called Muddy Lake (currently frozen and definitely not muddy). When walking along the paths you will generally be followed by a cloud of chickadees, who live in hope that you will be one of the many people that feed them. They whiz around your head, dancing from branch to branch, waiting for the food to be provided. If you DO have food, they will quite cheerfully eat out of your hand.


Despite their tameness taking a photo of them in a more natural setting is challenging, due to their dislike of sitting still for more than a second.


As well as having met my study species properly, I’ve also been working to get to grips with both social network analysis and the study of personality in animals. I had a passing interest in both of these topics before, but now I’m having to rapidly learn about how these analyses are done.


Chickadees are highly social and tend to move about in small flocks. We have information about which birds were with which other birds at feeders.This is generated by special feeders which can identify individual birds fitted with RFID readers.

Picture1An RFID feeder, photo by Teri Jones

We also have information about how certain birds reacted to personality tests as well as which birds are dominant and which are subordinate. There are quite a few interesting questions we could answer by bringing these datasets together. Wrangling the various files together is and working out how to analyse them is the main thing I’m currently dealing with. This involves spending a lot of time shuffling data about in R. I’m also trying to automate and improve some of the workflow for generating this type of data.

We’ve  had a few quite heavy snowfalls. This was my building the day after a particularly severe one:


and this was the window to our office on the same day as the sun was steadily blotted out:


A few weeks ago I gave a departmental seminar on my ENTIRE PHD. This involved talking for longer than I ever had before, which made it very easy to lose track of time. I was extremely afraid of speaking for too long and boring everyone to death.

Picture2So many slides!

I felt it went quite well. My main mistake was towards the end. I glanced at a clock (the WRONG clock) as it turned out and was convinced I’d gone overtime. I sped up for the last few slides (which is unfortunate as they are the most interesting) and then apologised at the end for speaking for so long. As it turned out, I’d looked at the wrong clock and came in under time. At least it left lots of time for the many questions, which hopefully indicate that people are interested in what I spent about four years of life doing.

I have been finding time to try and do some more CANADIAN things. For example, I’ve now tried Poutine which according to Wikipedia is Canada’s national food. It’s basically chips, cheese and gravy. I think that undersells it, it’s warm, tasty, cheesy, potatoey goo.


I also ambled up to Winterlude (a winter festival thing held in Ottawa), where the international ice sculpture championship:IMG_0759OWL


This included a demonstration of ice carving, which is apparently mostly done with powertools now!


I clearly need to find some more CANADIAN things to do. Someone mentioned something called Nanaimo bars..


Snowy owls, snowmobile trails and beaver tails

So I think I’ve been in Canada for nearly two weeks now. I am gradually learning things.

For example, when someone talks about getting beaver tails, they actually mean some sort of delicious pastry:


I also learned how to write some basic video analysis stuff in python, and that a single chickadee in a controlled environment is an easier thing to track than multiple shags on the ocean:vlcsnap-2016-01-23-23h49m11s193I learnt that they deliberately thicken the ice here:


This is so the canal (which I walk along on the way to the office) can be turned into a massive ice skating rink. It opened for the first time today. I saw many people gliding effortlessly along, as well as some children being dragged along on sleds, which looks more my speed.  Since I got here I have occasionally been asked if I skate, to which I give a rueful laugh. I don’t think me and skates would really mix.

Another thing I learned is that batteries drain very quickly in the cold weather. Today myself and the grad students in my research group went to see if our lab vehicle still worked after sitting in a garage for a month or so. The answer was, it didn’t.


Battery is flat.

However once a man from the Canadian AA turned up and jump started it, the car had to be driven somewhere to charge the battery. So I got to go on jaunt to one of the field sites where chickadees are studied, alongside a snowmobile trail. This was great as I was keen to get out of the city and see some Canadian countryside, and see some chickadees in the wild.


We found many chickadees, but also a bald eagle!



Shortly after seeing this I left the trail to go and look at a heap of rusty farm machinery buried in the snow. When I came back to the path, I knew something was afoot. It was then that I learnt that Canadian ambushes are exceptionally polite, as I was warned I might want to put my binoculars away:

12615379_10208372060120941_7821439240601366873_oArg! (Photo by Teri Jones)

It was then decided that we would head back toward Ottawa and try and find some snowy owls that had been reported in the area. On the way we would stop for “Timbits”. I did not know who Tim was, or the answers to any related questions. I found out:


It seems that Timbits are a big box full of the centres of various doughnuts! These tided us by as we headed toward where we might find snowy owls.

We stopped at the edge of a field in the general area where the owls had been seen. We all climbed out of the car and had a general scan of the trees and hedgerows. Nothing. We got back in the car to try a different spot. This was rather similar to my previous experience of attempting to see specific birds, so I wasn’t overly hopeful about finding a snowy owl in an expanse of snowy fields.

Then Shannon who had been looking out of the moving car with her binoculars (I am fairly certain doing this would make me carsick) suddenly spotted something on an electricity pylon in the middle of a field. We parked as close as we could to have a look. The post was quite a distance from the road, but through our binoculars we could clearly see a snowy owl! I had never even seen an owl in daylight, let alone that clearly.

I tried to get a picture but even at maximum zoom it wasn’t enormously clear.


Still, the view through the binoculars was great. We stood and watched the owl for a bit, before deciding to head back down the road to try and find the female that was also supposed to be about. Once again I was sceptical. I think I’d just said finished muttering something along those lines when I suddenly has to ask:

“What’s that on the post?”


There, on a post right next to the road was another snowy owl. We parked up right next to it, getting a much better view than before. I decided I had to try and take a photo.

It was at this point was once again reminded that batteries drain incredibly fast in the cold. Like the car earlier, my camera refused to start up. I fumbled with the various spare batteries. None of them worked. This was absolutely typical but luckily the owl was fairly accomodating. Eventually through luck and the strategic warming up the batteries, my camera finally fired up:


Tomorrow: statistics course!

Arriving in Canada and what on earth do I call this blog now?

On Saturday the 9th, I finally left Cornwall after 8 years at Exeter’s Penryn campus. I’ll skip all the various Feelings  I had leaving all my friends and regular haunts. I’m fairly certain I’ll be back at some point anyway. Suffice it to say, I had a lovely series of leaving dos and made the most of my last few days in Cornwall.

One last sea swim. Good to acclimatise me to very cold weather

Then I was off to CANADA. After packing of course.

Over Christmas I had raided various outdoor shops in search of cold weather gear. I’d had various warnings about -25° C temperatures and therefore engaged in a last minute scramble to make sure that I’d be warm enough when I arrived. All this and various other aspects of my life would need to fit into a 100 l duffel bag and a 60 l rucksack. Funnily enough, this didn’t actually prove too much of a problem. The downsizing I did have to do was quite cathartic. The internal processing went something like this:

“Do you need <THING>?”


“Are you sure you need <THING>?”


“Are you sure? Are you sure? Are you sure?”

“… no”


and thus I did not take a million and one pairs of socks (which would be too thin for this time of year anyway) or old t-shirts which, while I was very attached to them, were full of holes. I did take my climbing equipment, reasoning that indoor climbing would be a sensible sport to continue. I ultimately decided against taking camping equipment, mainly for space reasons. It’s probably a bit cold right now anyway, I may attempt to get it posted out when the weather was a little more clement.

Hoicking by now 21 kg (still in the weight limit) duffel bag onto my back I then boarded the series of trains, cars, underground trains  and planes which would take me to Canada. The flight passed uneventfully (I watched The Martian!) and after standing in various queues to get through passport control, immigration etc I was finally officially in Canada. I pulled out my new thick coat and mitts and headed outside.


Being British, I had generally come to accept that winters would involve mainly wind and rain, and only rarely snow. Here however, there was more snow than they knew what do with.

No possibility of getting excited though, I had a heavy bag and needed to bus into town before it got even darker and colder.

The bus journey ended up being completely free, which was good as all the buses had “exact change only” labels on them. I definitely did not have exact change having only just arrived. The first buses till was broken. The second bus driver saw that I had nothing smaller than $10 and told me I could travel for free if I told him where my accent was from. Which was extremely kind of him.

I arrived at the place I am staying, starting to feel the effects of jetlag. I think it was about 11:00 UK time when I arrived, and I’d been up from 7. Though I was very nearly napping at one point, I managed to stay awake until about 2:30 UK time (9 PM here) which meant I woke up at a reasonable time the next day.

The next day I visited my new department. It is very much bigger than the Cornwall! My new supervisor/boss Julie led me through a maze of corridors and stairwells showing me various bits of the department, including my new office space (Fourth floor. Not the first, second or third which I discovered by process of elimination the following day). I met other members of my research-group as well.

I then went off and did some dull but necessary things, but afterwards ambled up to Parliament hill


I imagine this could be quite a touristy place but there wasn’t a soul around except for the police. I then headed down to where the Rideau  canal meets the river.


Once again there wasn’t anybody around. It was really quite eerie, as even the traffic noises died away leaving nothing but the sound of the wind and the creaking ice.

Finally I headed home, back along the canal. Here is a photo from the footbridge to the university over the canal, which is my regular walk into the office now.


I hope to try and do some more touristy stuff on the weekend. In the meantime, more boring yet necessary stuff as I set my life up over here. And soon, some science!

Python is Nice.

During my undergraduate degree, we had one statistics module in our second year. While this module gave us a rather nice theoretical introduction into the use of the basic statistical tests such as t tests, the practical side was a little hazier. We had a few scheduled practical sessions in which theoretically we would be introduced to the various options for statistical analysis available. Practically however, we were encouraged very heavily to use SPSS. R was mentioned as an option, but only as a SCARY option. I had a brief look through some lists of commands, but without any real sense of direction and soon reverted to using SPSS like the rest of my class.

(I should add that this has all changed a lot in my University now, students now get introduced to R and other statistical techniques in their first year)

It was only when doing my masters course that I first started getting to grips with using R, my first basic coding language. While initially daunting I soon found that it appealed to me for a variety of reasons, laziness being one of them. In SPSS, deciding to change a parameter on a test would require re-clicking through a whole host of dialogues. In R a simple code change can produce the change immediately. The same was true of graphs. While SPSS had a friendly WYSIWYG interface, it could prove immensely fiddly when you had a particular graph in mind or if you needed to produce a series of identical graphs.

Beyond laziness we started playing with real, messy datasets. The advantage of being able to manipulate large  amounts of data quickly and efficiently soon became apparent, not to mention the various powerful tools R provided to analyse these datasets.

When  began my PhD I also started using matlab to play with mathematical models. While similar to R, it had some annoying differences in syntax that frequently tripped me up when I was starting out. Still, I was very much coming round to the idea that “Code is Good” and decided that being able to do all this stuff would save me an infinite number of headaches. This very much proved to be the case, even for something as daft as wanting to have all individual birds I had GPS coordinates for share a similar naming convention. Suddenly I was playing around with the regular expressions commands to manipulate strings. I can’t pretend I 100% understand these even now, but I can understand how powerful they can be.

I don’t know perl 😦

Throughout my MSc and even into my PhD I knew other for whom code just would not click. I saw this again when demonstrating on undergrad and masters stats modules. Occasionally (especially at undergrad level) I saw students making self fulfilling prophecies that anything that would involve writing code was not for them, and that engaging in it was therefore worthless. I’d always argue that is definitely worth engaging in. The stuff commonly used by bioscientists is infinitely friendlier than “proper” programming languages once you get to know it, and is so ridiculously useful.

This brings me round to the main thrust of this blog post, python. Python is Nice. It’s really really nice. You just won’t begin to imagine how very nice it is. R is powerful for statistics, but starts to fall down when you start getting into the realms of image analysis, file manipulation etc. Matlab is pretty powerful and flexible, but costs MONEY.

Python is free, friendly and powerful. Using python I have been able to chuck files around different directories automatically, process large data files, run bayesian statistics and create some pretty graphs. Here is one that ended up getting cut from what I’m currently working on, so it’s probably ok to show.



The initial setup can seem slightly confusing, and I highly recommend using a distribution like winpython which installs everything required and comes with lots of useful packages (including the essential numpy which lets you do all the sorts of n-dimensional matrix manipulations that you might be used to from R or matlab). Using a prepackaged version largely avoids all the confusion between firing up python in windows commandline vs. using a python interpreter that I certainly puzzled over initially. It also comes with an interpreter/editor spyder which makes the writing and trialling of big scripts extremely straight forward, in that it combines the script editor/console in the same way as Matlab or R commander might.

Will Python be replacing R for me? No, definitely not, R does too much good statistical stuff and has too many clever people writing packages for it. Will I be using it to do some of the things that I used to do that pushed R out of it’s comfort zone? Yes. Will I be using it instead of Matlab? Yes, because it is free!

I suppose the other main point of this rather rambly post is for students just starting out with all this: Don’t be afraid of code. Getting to grips with it is daunting at first, but incredibly rewarding in the long run.



the Final Official (anticlimatic!) End and what happens after

Today I received this e-mail:


So that’s definitely it. More or less four years exactly after starting, the PhD is officially completed. I’d say that’s the end of shags for me except, as I’ve commented before, there are bits of my thesis that I’m keen to write up as papers. That process is still ongoing and I might enthuse about things  (like how pretty the graphs that python can produce are) at some point in the future. There is however, now an extra incentive to get those papers finished.

Back when I wrote about my viva experience I finished on a question:

“…but for the most part I now have to think about some big questions. Namely, what on earth do I do next?”

At the time that was a huge and scary question. With the viva over I felt like I was lacking in purpose. I had corrections and papers to work on, but those were distractions from the sudden looming nothingness that came from the sudden end of what had taken up three and a half years of my life. I’d stop being paid a while back and was relying heavily on the kindness of my friends, subsisting on my savings and whatever demonstrating work I could pick up, anything to avoid having to leave Falmouth.


This was important to me. I’ve lived here for over eight years now, and as such was reluctant to leave. I wanted to try and remain near the university in some capacity, where I could benefit from collaborations on papers and get advice from others’ experience while applying for jobs. I also felt it was important to try to maintain my independence.

However, I knew I was going to have to leave eventually. Eight years is quite a long time to remain at one institution and so any job I applied for would definitely not be down here. In some ways, the further away the better. My main objectives with finding a post-doc was that it would allow me to continue studying something relevant to my research interests (social information use, group behaviours etc.) and that my skills were adequate. Aside from that, I would go anywhere and study anything.

So here I was, malingering, doing teaching work, making pretty figures in python, trying to write papers, applying for some post-docs and getting used to rejection.

Then a friend in the office e-mailed me a job that they had seen advertised that they thought sounded relevant to my interests. Which it was, enormously. I spent a good long while checking my CV and cover letter were as good as they could be. There was a lot of proof reading by various people. To cut a long story short I got a skype interview and then, to my infinite surprise, came home late one night to find I’d been offered the job.

I always said that if I were to move away from Falmouth that I’d rather move a long way so as to make it a clean break. I definitely succeeded with this job. The reason that I was interviewed using skype was because this particular post-doc happened to be at the University of Ottawa, Canada.

I am very excited about this.

Canada was a place that was definitely on my list of places that I’d like to visit, but at the time of application this seemed of secondary importance compared to the project. I deliberately avoided thinking about it, as I didn’t want to get too excited about a job I might not get. I received quite a lot of mockery upon telling the office that I’d got that job, when they realised I hadn’t even looked at where  Ottawa was on a map. I have to admit, I always pictured Canada to be something like this:

2008-05-19-082With apologies to Three Panel Soul


Still, now I have to face the reality of moving there. I moved down to student accommodation in Falmouth eight years ago. I’ve never properly moved to a new city, let alone a new country. There are a million things to do before I go and it’s all somewhat terrifying. Very exciting as well.

I’ll definitely try to keep this blog going. I’m not quite done with studying shags just yet and there will no doubt be amusing trials and tribulations as I move to Canada and start my new job.

I may have to consider a new name though. The chickadee project doesn’t really roll off the tongue.