Open Source Research!

One of the recurrent issues that has been plaguing the ML research community is irreproducibility. An issue that I have stumbled upon many times, both in my latest degree project on Meta-Reinforcement Learning (a field known to have considerable irreproducibility issues) and in the NeurIPS 2019 reproducibility challenge. Unfortunately, in many cases, there is no simple solution to it. Immense computational costs, private data, proprietary code… However, in some cases, I believe there is a way to fight back.

Coming from a computer science background, I think the importance of open-source software is often undervalued. Remember that most of the global IT infrastructure sits on the open source project, linux :).

If most programmers agree that “bug-free software” is rare (if not just outright impossible). How can we assume that all the research published comes from bug-free code?

Open-sourcing code, whenever possible, is beneficial to all. It is for our personal growth. It is to thousands of others researchers in the field that are stuck in the implementation process.

Being nervous that people might find bugs in our code is not the way to go. I understand the fear of someone finding a game-breaking bug in the code you have been building for weeks but we need to understand that this is above us. It’s about research going forward, even if that means refuting, rethinking and rebuilding previous work.

From a learning perspective, we learn best from our mistakes.

From a software development perspective, the more sets of eyes we have, the faster we detect bugs.

From a ML perspective, we need ensembles of experts to help each other build upon current work and improve, instead of starting from scratch, speculating and working self-contained.

Let’s own up to our work, be responsible and share code whenever we can :)