Translating between Matlab and R

I mentioned in a previous post that I’d write a guide about translating Matlab code to work in R, so that others can avoid the same mistakes I made. This should also function as an R users guide to learning Matlab syntax and vice versa. I hope some people find it useful!

Full article below.

rmatlab

Continue reading

Translation Issues

Where on earth did the time go? One minute I was looking at ice sculptures and wading through snow drifts and now the snow is gone (mostly) and various wildlife has emerged from hiding.

Look, beavers!

I’ve also managed to get out on the water myself. And eat bacon while doing it. Canadian bacon is Different.

riverbaconRiver bacon!

Perhaps the main reason I’ve lost track of time in the last few weeks is that whether I’ve been awake or asleep, this keeps on flashing in front of my eyes.

codegif.gif

I haven’t been able to escape it. My office desktop has looked like this on and off for the whole month. I hope to later show what this has resulted in, but in the meantime I’m going to moan about the amount of pain it’s caused me.

The main source of trouble was that this code was originally written in Matlab. I decided to save us from having to acquire a Matlab license by translating the many scripts that make up the code into R.

Initially this was tedious. There are enough syntax differences (not to mention different names of functions etc) between R and Matlab that this required me to go through the code line by line. Then, even when I done this a number of errors arose simply due to the differing ways that the two programs handle data. I’ll post a guide based on what I learnt in separate post, featuring less pictures of beavers.

Much hair pulling later I got the code running, fed it my data and got a result. These results were consistent with some previous findings obtained using simpler methods. So far so good.

I then decided that instead of feeding my data to the code all in one go, it would be useful to give it one day at a time and then collate the results. “Fine” I thought. “Just modify my overarching processing code, no trouble”.

I was wrong.

c149182a696e73890de876f3d392e2da(Found via googling evil Matlab)

Once again the way in which the two programs handle data required me to make a lot of modifications to the various scripts. Cue more hair pulling. I should also mention that I’ve written this code to be run in parallel, utilising all of my computers cores to increase speed, which means R’s normal debugging tools don’t work.

Finally I got the code to run again and got a result. However, something had changed. A previously suggested relationship had completely reversed in direction. Was this simply due to the new way of feeding the data in? Or was it due to a bug in my code? Or due to me deleting some faulty data? I ran the code again using the original way of processing the data.

Even using parallel processing, this code can take anything from several hours, to all night to run. This meant that getting results was a slow process. So after waiting several hours for the code to run again using the original data processing, once again I got results.

The relationship had flipped direction in these results too.

Moving-animated-clip-art-dont-panic-picture

This suggested that the changes I’d made to accommodate the new data processing method had resulted in a COMPLETELY DIFFERENT RESULT. On the one hand, this was good. It meant that the biologically unrealistic result was due to my error rather than a fundamental problem with the methods. On the other hand, this is the sort of thing that can wake a scientist up at night screaming. A series of small changes in the way data was analysed leading to completely misleading results. In this case we’d caught it before we went too far, but if we’d approached this naively it might have been very easy to miss.

So, now I needed to work out which of my changes had caused this change. Luckily I save all my working files in Dropbox, which keeps a backup of all previous versions. I found a word document containing graphs I’d made to show my supervisor before I’d made changes and reverted all my code to a date before then. Then one by one I reinstated my changes.

In the end I pinned it down to one file. In that file, one line of code.

as.matrix(Y)

One line of code had resulted in huge, significant changes to my final result. As I said, the stuff of nightmares.

In the end I stripped out all the changes I’d made and carefully rewrote the scripts to deal with the new method of data processing. So my tale of woe has a happy ending, the code now works and perhaps I’ll even have some results soon. For everyone who made it this far, here is a view of Gatineu Park:

IMG_3201

the Final Official (anticlimatic!) End and what happens after

Today I received this e-mail:

official

So that’s definitely it. More or less four years exactly after starting, the PhD is officially completed. I’d say that’s the end of shags for me except, as I’ve commented before, there are bits of my thesis that I’m keen to write up as papers. That process is still ongoing and I might enthuse about things  (like how pretty the graphs that python can produce are) at some point in the future. There is however, now an extra incentive to get those papers finished.

Back when I wrote about my viva experience I finished on a question:

“…but for the most part I now have to think about some big questions. Namely, what on earth do I do next?”

At the time that was a huge and scary question. With the viva over I felt like I was lacking in purpose. I had corrections and papers to work on, but those were distractions from the sudden looming nothingness that came from the sudden end of what had taken up three and a half years of my life. I’d stop being paid a while back and was relying heavily on the kindness of my friends, subsisting on my savings and whatever demonstrating work I could pick up, anything to avoid having to leave Falmouth.

11403249_10153593172859769_2081981389068106492_n

This was important to me. I’ve lived here for over eight years now, and as such was reluctant to leave. I wanted to try and remain near the university in some capacity, where I could benefit from collaborations on papers and get advice from others’ experience while applying for jobs. I also felt it was important to try to maintain my independence.

However, I knew I was going to have to leave eventually. Eight years is quite a long time to remain at one institution and so any job I applied for would definitely not be down here. In some ways, the further away the better. My main objectives with finding a post-doc was that it would allow me to continue studying something relevant to my research interests (social information use, group behaviours etc.) and that my skills were adequate. Aside from that, I would go anywhere and study anything.

So here I was, malingering, doing teaching work, making pretty figures in python, trying to write papers, applying for some post-docs and getting used to rejection.

Then a friend in the office e-mailed me a job that they had seen advertised that they thought sounded relevant to my interests. Which it was, enormously. I spent a good long while checking my CV and cover letter were as good as they could be. There was a lot of proof reading by various people. To cut a long story short I got a skype interview and then, to my infinite surprise, came home late one night to find I’d been offered the job.

I always said that if I were to move away from Falmouth that I’d rather move a long way so as to make it a clean break. I definitely succeeded with this job. The reason that I was interviewed using skype was because this particular post-doc happened to be at the University of Ottawa, Canada.

I am very excited about this.

Canada was a place that was definitely on my list of places that I’d like to visit, but at the time of application this seemed of secondary importance compared to the project. I deliberately avoided thinking about it, as I didn’t want to get too excited about a job I might not get. I received quite a lot of mockery upon telling the office that I’d got that job, when they realised I hadn’t even looked at where  Ottawa was on a map. I have to admit, I always pictured Canada to be something like this:

2008-05-19-082With apologies to Three Panel Soul

or

Still, now I have to face the reality of moving there. I moved down to student accommodation in Falmouth eight years ago. I’ve never properly moved to a new city, let alone a new country. There are a million things to do before I go and it’s all somewhat terrifying. Very exciting as well.

I’ll definitely try to keep this blog going. I’m not quite done with studying shags just yet and there will no doubt be amusing trials and tribulations as I move to Canada and start my new job.

I may have to consider a new name though. The chickadee project doesn’t really roll off the tongue.

 

 

 

The viva

CJzTSwVW8AEu52K

This sign has been at the  bottom of my road all week. Constantly reminding me that my viva is imminent (SOON. DON’T FORGET. DON’T STOP THINKING ABOUT IT FOR EVEN A SECOND. DID YOU STOP THINKING ABOUT IT? LOOK AT THIS SIGN! etc). Making it impossible to forget that on Monday I was going to have to defend my thesis.

This translates into me sitting in a room for several hours while experts in my field discuss my thesis in great detail. One of these was Matt Witt from my own university, who I at least knew. The other was Ian Couzin, who I did not. I had seen him give a plenary at Behaviour in Newcastle. This talk basically seemed to consist of one extremely clever experiment after another, all flowing naturally into each other.

Now almost all the people I’d spoken to about their own vivas had told me that I should enjoy it, as it was the only opportunity I’d ever have to discuss my work with experts in my field and the only time anyone would ever care about my work that much. I told them they were filthy liars and went back to worrying about having to talk to these people, both of whom had done a ridiculous amount of fantastic science, about my little corner of shag-related study.

The final week of fretting eventually ended and the big day arrived. People in my office were treated to the strange and comical sight of my head protruding from a suit.

20150713_094454

After about twenty minutes of wearing a hole in our office carpet I finally headed to the room where the viva was due to take place. My thought process at this point basically consisted of white noise and the occasional “This is it!”. I’d been doing this all day yesterday as well: “By this point tomorrow, it will be over”, “One more sleep” and so on.

So as I lurked in the corridor waiting to be asked in, all I could think about this was going to be the culmination of about three and a half years worth of work.

I am not sure how much I can say about the actual viva. It’s all a bit hazy. Facts:

  • I was in there for about three and a half hours (hey, that’s about an hour for each year of PhD!).
  • The examiners tried to put me at my ease immediately (which definitely  helped, but I was probably still extremely tense throughout the whole thing)
  • We opened with a discussion about working with shags. I think I may have chuckled ruefully as some point.
  • We discussed a lot of interesting ways that the field of collective behaviour might go and how technological development might help that.
  • There were biscuits at one point.

Then it was over. That was it, done! I picked up my thesis and proceeded to the traditional cake and bubbly celebration in our common room.

11855442_10155927591285268_882798125_nCake! (By Sheridan)

There were also presents, the main one being a rather nifty knife/multitool with a custom message on it:

And a shag’s skull in amber from my friend Sarah:

IMG_0569

Then, it being a Monday lunchtime, everyone shuffled off back to work. Leaving me wondering what to do next. I went and had some lunch myself, then hurled myself into the sea to try to counteract the effect of the bubbly I’d been drinking. It felt extremely odd to have finished.

In the evening we celebrated by visiting my normal haunt, The Waterfront which actually appeared on the front of my viva card, courtesy of Emma Wood:

IMG_0573The Front!

Followed by a curry, and then more drinks at the Front. I did my best to cope with the many drinks and not fall asleep before midnight.

11868665_10153274384093551_2095771882_n

Falmouth on a Monday night being what it is (not to mention that I was feeling extremely sleepy at this stage) after midnight we returned to my house for some viva whisky. This ended up taking a while.

So it’s all over. I have some minor corrections to make but for all intents and purposes I have finished my PhD. Naturally I’d like to try and get some of the other chapters published, which will mean going back through the data and writing I’ve already collected/written, but for the most part I now have to think about some big questions. Namely, what on earth do I do next?

 

 

 

 

What do you think you’re doing?

Oh look, it’s going to be February this weekend.

Moving-animated-clip-art-dont-panic-picture

This week I have been:

  • Trying to write a chapter, which involves putting words on a page in an order that doesn’t make my supervisors want to murder me.

 

  • Writing a detailed methodology for the chapter, which involves putting ALL the words about what how my simulation works. Which is a surprising amount. However, I also need to avoid writing too many words on the bits of my simulations which have already been written elsewhere, by someone else. This can be frustrating.

 

  • Wrestling with LaTeX, which can turn this:

    \overrightarrow{f}\! \! _{i,int}= \omega _{rep}\overrightarrow{f}\! \! _{i,rep}+ \omega _{al}\overrightarrow{f}\! \! _{i,al}+\omega _{att}\overrightarrow{f}\! \! _{i,att}+\omega _{front}\overrightarrow{f}\! \! _{i,front}

Into this:

eq

  • Producing surprisingly pretty excel spreadsheets:

EXCELMAGIC

  • Trying to work out how best to quantify which model is better than another, and how to convince others about this (“Trust me!” just doesn’t cut it).

results

  • Not Panicking.

More virtual birds

I have returned to my simulated rafts of shags. As well as general code refinements, I have also completely overhauled the way they dive and surface.

The dive rules have been in place for a while. They are as follows:

diverules

1. The virtual bird will dive wherever and whenever it wants, without paying any attention to what other birds are doing. Given that my hypothesis is that birds are using the diving behaviour of others to make their own diving decisions, this is the null model.

2. Birds are more likely to dive if another bird dived or surfaced recently. How likely they are to dive depends on how close the other bird was in space and time.

3. Similar to 2. but birds only have a limited cone of vision. They will not be away of a dive going on behind them for example.

 

Previously I only had one rule for surfacing: birds would surface at a random time after they had dived, within a circle of a random radius centred around where they initially dived . The distance they were allowed to travel was constrained by how long they’d been underwater. They were also more likely to surface in proximity to other birds, so as to maintain the cohesion we see in real birds.

Just before ISBE, my supervisor suggested that actually we probably want to try out a variety of surface rules to try a few different scenarios. This required a complete rewrite of the simulation, but this was a good opportunity for me to merge all my various dive rules into one piece of code too. Though there was quite a bit of hair pulling involved, I eventually ended up with a simple unified function.

The surfacing rules birds can now obey are as follows:

surfrules

1. Surface completely randomly in time and space. Virtual birds can reappear at any time within a circle of random radius centred around their dive, with the maximum radius depending on how long a bird was underwater. They don’t care if they are near another bird on the surface or not.

2. Same as rule 1, but birds are more likely to surface closer to another bird. The time they stay underwater is still random, which still controls the maximum possible distance they can travel underwater.

3. Similar to 2. with an important addition: A bird is more likely to surface in close proximity (in time and space) to a bird it dived in close proximity to (in time and space). This is to attempt to simulate birds following each other more closely underwater.

Of course. rewriting my code so that I can switch between which rules a simulation is using just by changing a number in my simulation uses was somewhat tricky, but eventually I hammered out all the bugs. Unifying all my code like this will make the selection of a best fitting model much simpler.

Virtual birds

A few months ago I gave a talk at the ASAB Easter conference in Sheffield about how I’m analysing the foraging rafts using collective behaviour. Of course, what with one thing and another, the work wasn’t as far along as I would have liked. After I got back from fieldwork and crunched the new data, I decided it was time to look at this work again.

One of the main reasons for pushing this work is that I am giving another talk on this at an international conference. In New York. In a few weeks. Do not let my restrained tones hide how terrified/excited I am about this.

The conference is ISBE, the Internation Society of Behavioural Ecologists. I applied for a talk back at the beginning of the year and several months later, here we are:

program

 

(Last talk of the day, hopefully some people will be awake)

I’d really better get some results together hadn’t I?

To recap what I had already done:

-Extracted positions of individual birds from video.

vlcsnap-00028-Tracked these individuals and then used these positions to create trajectories.

– Ran these trajectories through a correction matrix, to account for any distortions the camera might introduce

– Created code to remove unrealistic tracks.

– Wrote further functions that I can use to manually delete and merge tracks.

– Extracted the dive and surface events

dsplot

– Created various graphing functions, such as taking the corrected trajectories and plotting them back to their original positions to check my working.

vlcsnap-00030

What I have done in the last week (working frantically):

– Created a simulation based on zonal interaction models. Virtual birds that mooooooove.

– Wrote code that will compare the real data with my models, using things like radial density.

relpos– Added the ability for my virtual birds to DIVE.

– Modified simulation so the birds can react to these dive and surface events.

What am I doing now?

So the goal now is to produce results by choosing the simulation that best matches up with the data. In order to do this I need to choose the best “weight” for each of the things that might affect a birds movement (the effect of repulsion from other birds vs. the effect of attraction to other birds for example). I am currently exploring these weights manually, but eventually I need to run some code to optimise each of these weights for each piece of data.

I’d also like to add some more complicated diving rules (At the moment diving is totally random) where they will be influenced by other dives. I then need to come up with ways of checking this method of diving against the data, to determine how realistic this sort of behaviour is..

I fly out to New York in a few weeks. This is why I spend my weekends locked in the office.

Just to finish up here is a video from one of my early simulations with some random parameters. Those birds can’t seem to get away from each other fast enough! I suspect they might also need a bit more autonomous movement to stop them doing the crazy little dance when they can’t find other birds.