# GMMevents

So a while back I reported I was going a tad insane translating code from Matlab to R. I’ve since written a guide that I hope will help others with similar issues. The code I translated is now available as part of the ASNIPE package for R. ASNIPE stands for Animal Social Network Inference and Permutations for Ecologists. The code I translated is gmmevents, which runs a Gaussian mixture model on time series data. A gaussian mixture model is a form of unsupervised machine learning, sorting one dimensional data into clusters, similar to k-means clustering. To illustrate:

Adapted from Fig. 2 – Psorakis et al., J. R. Soc. Interface  (2012)

This obviously has a lot of applications in social network analysis, as we can infer that individuals present in the same event likely have some interaction. I hope people find it useful, and that it was worth me going slightly mad over!

Advertisements

# NAOC Washington

Passing the last ISBE location on the way

After a stint in the UK I was off to a different conference, this time closer to my (current) home in Washington DC. The North American Ornithological Conference 2016. My boss had been invited to speak in a social network symposium, and sent me along instead.

I was staying in the same hotel that the conference was in. It was somewhat fancier than the previous conference’s accommodation, which had been university halls.

I had a day before I started, so I did some sightseeing. Most of Washington’s tourist hot spots are conveniently within walking distance of each other.

.. and so forth.

Before the conference officially began I attended a workshop on social networks, which was incredibly helpful in term of introducing me to new ways that social networks could be used and plotted. As I’ve said, I still feel I’m very new to social network analysis, and had been very focused on my own system. Both this workshop and the symposium I spoke in had so many different ways of applying network techniques, that would not have occurred to me otherwise.

The conference proper began the next day with a plenary from a NASA astronaut candidate

There were an enormous number of simultaneous talks, not to mention the hundreds of posters. It was a very well organised conference.

I wasn’t the only chickadee talk

The banquet was in the zoo!

The day after the conference finished I managed to get around a few more of the Smithsonian museums (I’d done the air and space on the day before the conferences starts). Then it was time to head back to Ottawa. I’d been away for about a month.

I found both of these summer conference extremely valuable, in terms of thinking about future work and in generally being inspiring. Now however, back to work..

# Translating between Matlab and R

I mentioned in a previous post that I’d write a guide about translating Matlab code to work in R, so that others can avoid the same mistakes I made. This should also function as an R users guide to learning Matlab syntax and vice versa. I hope some people find it useful!

Full article below.

# Translation Issues

Where on earth did the time go? One minute I was looking at ice sculptures and wading through snow drifts and now the snow is gone (mostly) and various wildlife has emerged from hiding.

Look, beavers!

I’ve also managed to get out on the water myself. And eat bacon while doing it. Canadian bacon is Different.

River bacon!

Perhaps the main reason I’ve lost track of time in the last few weeks is that whether I’ve been awake or asleep, this keeps on flashing in front of my eyes.

I haven’t been able to escape it. My office desktop has looked like this on and off for the whole month. I hope to later show what this has resulted in, but in the meantime I’m going to moan about the amount of pain it’s caused me.

The main source of trouble was that this code was originally written in Matlab. I decided to save us from having to acquire a Matlab license by translating the many scripts that make up the code into R.

Initially this was tedious. There are enough syntax differences (not to mention different names of functions etc) between R and Matlab that this required me to go through the code line by line. Then, even when I done this a number of errors arose simply due to the differing ways that the two programs handle data. I’ll post a guide based on what I learnt in separate post, featuring less pictures of beavers.

Much hair pulling later I got the code running, fed it my data and got a result. These results were consistent with some previous findings obtained using simpler methods. So far so good.

I then decided that instead of feeding my data to the code all in one go, it would be useful to give it one day at a time and then collate the results. “Fine” I thought. “Just modify my overarching processing code, no trouble”.

I was wrong.

(Found via googling evil Matlab)

Once again the way in which the two programs handle data required me to make a lot of modifications to the various scripts. Cue more hair pulling. I should also mention that I’ve written this code to be run in parallel, utilising all of my computers cores to increase speed, which means R’s normal debugging tools don’t work.

Finally I got the code to run again and got a result. However, something had changed. A previously suggested relationship had completely reversed in direction. Was this simply due to the new way of feeding the data in? Or was it due to a bug in my code? Or due to me deleting some faulty data? I ran the code again using the original way of processing the data.

Even using parallel processing, this code can take anything from several hours, to all night to run. This meant that getting results was a slow process. So after waiting several hours for the code to run again using the original data processing, once again I got results.

The relationship had flipped direction in these results too.

This suggested that the changes I’d made to accommodate the new data processing method had resulted in a COMPLETELY DIFFERENT RESULT. On the one hand, this was good. It meant that the biologically unrealistic result was due to my error rather than a fundamental problem with the methods. On the other hand, this is the sort of thing that can wake a scientist up at night screaming. A series of small changes in the way data was analysed leading to completely misleading results. In this case we’d caught it before we went too far, but if we’d approached this naively it might have been very easy to miss.

So, now I needed to work out which of my changes had caused this change. Luckily I save all my working files in Dropbox, which keeps a backup of all previous versions. I found a word document containing graphs I’d made to show my supervisor before I’d made changes and reverted all my code to a date before then. Then one by one I reinstated my changes.

In the end I pinned it down to one file. In that file, one line of code.

as.matrix(Y)

One line of code had resulted in huge, significant changes to my final result. As I said, the stuff of nightmares.

In the end I stripped out all the changes I’d made and carefully rewrote the scripts to deal with the new method of data processing. So my tale of woe has a happy ending, the code now works and perhaps I’ll even have some results soon. For everyone who made it this far, here is a view of Gatineu Park:

# the Final Official (anticlimatic!) End and what happens after

Today I received this e-mail:

So that’s definitely it. More or less four years exactly after starting, the PhD is officially completed. I’d say that’s the end of shags for me except, as I’ve commented before, there are bits of my thesis that I’m keen to write up as papers. That process is still ongoing and I might enthuse about things  (like how pretty the graphs that python can produce are) at some point in the future. There is however, now an extra incentive to get those papers finished.

Back when I wrote about my viva experience I finished on a question:

“…but for the most part I now have to think about some big questions. Namely, what on earth do I do next?”

At the time that was a huge and scary question. With the viva over I felt like I was lacking in purpose. I had corrections and papers to work on, but those were distractions from the sudden looming nothingness that came from the sudden end of what had taken up three and a half years of my life. I’d stop being paid a while back and was relying heavily on the kindness of my friends, subsisting on my savings and whatever demonstrating work I could pick up, anything to avoid having to leave Falmouth.

This was important to me. I’ve lived here for over eight years now, and as such was reluctant to leave. I wanted to try and remain near the university in some capacity, where I could benefit from collaborations on papers and get advice from others’ experience while applying for jobs. I also felt it was important to try to maintain my independence.

However, I knew I was going to have to leave eventually. Eight years is quite a long time to remain at one institution and so any job I applied for would definitely not be down here. In some ways, the further away the better. My main objectives with finding a post-doc was that it would allow me to continue studying something relevant to my research interests (social information use, group behaviours etc.) and that my skills were adequate. Aside from that, I would go anywhere and study anything.

So here I was, malingering, doing teaching work, making pretty figures in python, trying to write papers, applying for some post-docs and getting used to rejection.

Then a friend in the office e-mailed me a job that they had seen advertised that they thought sounded relevant to my interests. Which it was, enormously. I spent a good long while checking my CV and cover letter were as good as they could be. There was a lot of proof reading by various people. To cut a long story short I got a skype interview and then, to my infinite surprise, came home late one night to find I’d been offered the job.

I always said that if I were to move away from Falmouth that I’d rather move a long way so as to make it a clean break. I definitely succeeded with this job. The reason that I was interviewed using skype was because this particular post-doc happened to be at the University of Ottawa, Canada.

I am very excited about this.

Canada was a place that was definitely on my list of places that I’d like to visit, but at the time of application this seemed of secondary importance compared to the project. I deliberately avoided thinking about it, as I didn’t want to get too excited about a job I might not get. I received quite a lot of mockery upon telling the office that I’d got that job, when they realised I hadn’t even looked at where  Ottawa was on a map. I have to admit, I always pictured Canada to be something like this:

With apologies to Three Panel Soul

or

Still, now I have to face the reality of moving there. I moved down to student accommodation in Falmouth eight years ago. I’ve never properly moved to a new city, let alone a new country. There are a million things to do before I go and it’s all somewhat terrifying. Very exciting as well.

I’ll definitely try to keep this blog going. I’m not quite done with studying shags just yet and there will no doubt be amusing trials and tribulations as I move to Canada and start my new job.

I may have to consider a new name though. The chickadee project doesn’t really roll off the tongue.

# The viva

This sign has been at the  bottom of my road all week. Constantly reminding me that my viva is imminent (SOON. DON’T FORGET. DON’T STOP THINKING ABOUT IT FOR EVEN A SECOND. DID YOU STOP THINKING ABOUT IT? LOOK AT THIS SIGN! etc). Making it impossible to forget that on Monday I was going to have to defend my thesis.

This translates into me sitting in a room for several hours while experts in my field discuss my thesis in great detail. One of these was Matt Witt from my own university, who I at least knew. The other was Ian Couzin, who I did not. I had seen him give a plenary at Behaviour in Newcastle. This talk basically seemed to consist of one extremely clever experiment after another, all flowing naturally into each other.

Now almost all the people I’d spoken to about their own vivas had told me that I should enjoy it, as it was the only opportunity I’d ever have to discuss my work with experts in my field and the only time anyone would ever care about my work that much. I told them they were filthy liars and went back to worrying about having to talk to these people, both of whom had done a ridiculous amount of fantastic science, about my little corner of shag-related study.

The final week of fretting eventually ended and the big day arrived. People in my office were treated to the strange and comical sight of my head protruding from a suit.

After about twenty minutes of wearing a hole in our office carpet I finally headed to the room where the viva was due to take place. My thought process at this point basically consisted of white noise and the occasional “This is it!”. I’d been doing this all day yesterday as well: “By this point tomorrow, it will be over”, “One more sleep” and so on.

So as I lurked in the corridor waiting to be asked in, all I could think about this was going to be the culmination of about three and a half years worth of work.

I am not sure how much I can say about the actual viva. It’s all a bit hazy. Facts:

• I was in there for about three and a half hours (hey, that’s about an hour for each year of PhD!).
• The examiners tried to put me at my ease immediately (which definitely  helped, but I was probably still extremely tense throughout the whole thing)
• We opened with a discussion about working with shags. I think I may have chuckled ruefully as some point.
• We discussed a lot of interesting ways that the field of collective behaviour might go and how technological development might help that.
• There were biscuits at one point.

Then it was over. That was it, done! I picked up my thesis and proceeded to the traditional cake and bubbly celebration in our common room.

Cake! (By Sheridan)

There were also presents, the main one being a rather nifty knife/multitool with a custom message on it:

And a shag’s skull in amber from my friend Sarah:

Then, it being a Monday lunchtime, everyone shuffled off back to work. Leaving me wondering what to do next. I went and had some lunch myself, then hurled myself into the sea to try to counteract the effect of the bubbly I’d been drinking. It felt extremely odd to have finished.

In the evening we celebrated by visiting my normal haunt, The Waterfront which actually appeared on the front of my viva card, courtesy of Emma Wood:

The Front!

Followed by a curry, and then more drinks at the Front. I did my best to cope with the many drinks and not fall asleep before midnight.

Falmouth on a Monday night being what it is (not to mention that I was feeling extremely sleepy at this stage) after midnight we returned to my house for some viva whisky. This ended up taking a while.

So it’s all over. I have some minor corrections to make but for all intents and purposes I have finished my PhD. Naturally I’d like to try and get some of the other chapters published, which will mean going back through the data and writing I’ve already collected/written, but for the most part I now have to think about some big questions. Namely, what on earth do I do next?

# What do you think you’re doing?

Oh look, it’s going to be February this weekend.

This week I have been:

• Trying to write a chapter, which involves putting words on a page in an order that doesn’t make my supervisors want to murder me.

• Writing a detailed methodology for the chapter, which involves putting ALL the words about what how my simulation works. Which is a surprising amount. However, I also need to avoid writing too many words on the bits of my simulations which have already been written elsewhere, by someone else. This can be frustrating.

• Wrestling with LaTeX, which can turn this:

\overrightarrow{f}\! \! _{i,int}= \omega _{rep}\overrightarrow{f}\! \! _{i,rep}+ \omega _{al}\overrightarrow{f}\! \! _{i,al}+\omega _{att}\overrightarrow{f}\! \! _{i,att}+\omega _{front}\overrightarrow{f}\! \! _{i,front}

Into this:

• Producing surprisingly pretty excel spreadsheets:

• Trying to work out how best to quantify which model is better than another, and how to convince others about this (“Trust me!” just doesn’t cut it).

• Not Panicking.