In the previous blogpost I described the ECIR conference and the workshop and tutorials day. Now, I will summarise some of the recurrent topics during the main conference. The first session of the conference focused on reproducibility and the challenges to obtain the same results as they were reported in previous research.
Being able to reproduce a research paper to verify its findings, or to apply the same ideas to new data is critical to the scientific nature of Information Retrieval, and any other fields. However, we are known for our lack of it. There were multiple tips on how to address some of this challenges, but the main ones that resonate more with the audience were to share both data and code, as well as to design the experimental process in a deterministic matter. For instance, if an algorithm uses randomised sampling, it will be impossible to replicate. An easy solution is to use some type of deterministic sampling from the experiment point of view, while being equivalent to a random sampling from the data distribution perspective. Other more obvious suggestions were to publicly define all the preprocessing steps in the data (this can be done by sharing all the code). Partially because of this session, a lot of posterior presentations said that the code (and data in some cases) will be shared for the community. The reproducibility idea was also picked up by the discussion panel about Evaluation in IR, where Norbert Fuhr even said that “we need to open our data to become a science“. The panel, formed by Forbert Fuhr, Udo Kruschwitz, Diane Kelly and Jaap Kamps, debated the current state of evaluation on IR. One especially important point was the role of the users (e.g., with living labs evaluation) vs a more system-oriented approach (following the Cranfield paradigm). The panel had diverse opinions about Crainfield as a process and some panelists (mainly Diane Kelly, if I remember correctly) were advocating to abandon it. On the other hand, Jaap suggested that we should incorporate more user simulations. My personal opinion is that we should focus on more realistic metrics. For instance, we should be much more interested on the usefulness of articles rather than relevance. Why? Because that is a metric much closer to the real way in which the users will judge the system. Lastly, one of the “elephants on the room” with the current state of IR evaluation experiments is that the people used in the large majority of user studies are usually “picked” from the student population. This is a huge bias on our experiments. The conclusions based on a mostly white, male population between 20 to 30 years old (as an hypothetical example) might not be able to generalised to the general population. The rest of the main conference went on in the usual fashion with multiple sessions focusing on specific tasks such as efficiency, summarisation or social media, … I won’t be going into the details of the sessions. Nonetheless, I recommend to check the programme and look for papers that are relevant to you. The other part of the conference I would like to spend some time on is the Industry Day. Traditionally, Industry Day is the day where some of the biggest companies in the field (e.g., Google, Microsoft, Yahoo, LinkedIn, …) present their last developments, and do some PR and hiring at the same time. However, this year was a breath of fresh air as Jussi Karlgren and Paul Ogilvie changed the style of the Industry Day to atract small companies and start-ups within the community by focusing on the experiences to transform an idea to a business. The main difference I felt was that the big companies are usually much more secretive with their solutions and processes while start-ups are, in general, more open on some of this matters. In addition to this, there was a presentation by a Venture Capitalist and a lengthy discussion about the creation of companies from academic ideas, the role of universities in the process and how we should focus on the user first of all. To be honest, I truly enjoyed this session. We had some very good presentations from seasoned researchers in the field such as Arjen De Vries, Jeremy Pickens or Vassilis Plachouras. In addition, two very interesting start-ups also presented some of their work: Joao Graça introduced UnBabel, a SaaS company that provides translation on demand by combining algorithms and human intervention; and Martin Stiksel who presented Lumi.do, a news personalisation company based in London. Signal was also represented as I was one of the speakers in this session. It would be very difficult to repeat the same style next year, as there is a limited number of start-ups from people in the community. Nonetheless, I hope this experience has shown that the small players have as much to share, if not more, to the community and I would love to see more start-ups and small enterprises are involved in the Industry Day of ECIR in the future. As a summary, ECIR 2015 it was a fantastic conference (as it has been every year I have attended) with amazing discussions in the coffee breaks and new potential collaborations. Getting the best demonstration award in the conference for one of our presentations was just the icing on the cake (see the picture below). 
Looking forward to see you all again in Padua…
The Practical Academic 