Nikola Stikov, is an Associate Professor of Electrical Engineering at Polytechnique Montreal.
In his interview with Paul Weavers, he talks about the importance of open science and his realization throughout his studies and research that the way we communicate needs to become more transparent.
Paul Weavers (PW): Nikola, what I see most publicized and most associated with you is the open science push. Can you describe the goals behind your side job?
Nikola Stikov (NS): For me it has become more than a side job – you could even say it is my main job – and it comes from this feeling that the way we do science should be more transparent! Not only becoming more transparent on data processing but also on the way we communicate our research findings.
Initially, my research was on developing techniques for MRI. However, throughout my PhD at Stanford and my postdoc at the Montreal Neurological Institute – which took about 10 years total – I realized that there is not enough transparency in the way we do our experiments and definitely not in the way we communicate our results to the public. Therefore, when I got a faculty job, I decided: transparency will be one of the most important poles of the research for my lab. Whatever we do, we will try to make it easy for people to reproduce it.
PW: Was it the inability to replicate other people’s work or the desire for other people to replicate your work that pushed you into open science?
NS: It was just the frustration with the PDF format of publishing papers. My first realization of this happened while I was working on an ISMRM (International Society for Magnetic Resonance in Medicine) conference abstract. The abstract was accepted, I presented a poster and it was very well received.
I asked my PhD supervisor John Pauly if we should publish this paper and his counter question was if I have calculated the opportunity costs of writing that paper. I realized that the price you pay for writing a paper is not being able to work on something else. Therefore, I decided not to write a 10-page PDF on my abstract and focus on something that I wanted to write about – and that took a long time. I spent about 5 years after that not publishing a single paper.
Given that I have managed to make a career out of that very long path of calculating opportunity costs and not publishing PDFs, now I want to change the way we communicate research. I want us to take our time and only write a paper when we are actually ready. And the way to do it is by complementing the paper with data, code and hopefully with transparent post-publication review.
PW: That is a very lofty idea to have. The metric for success as I understand it is not how well someone else could do this work or how well someone understands it – it is how many papers get published. How do we make these two rewards compatible?
NS: I believe that three 10-page PDF’s should not be the norm to get a PhD. There is a lot of exciting research that is using new technology and a lot of data, yet we are still communicating it using a 17th century artefact that comes from the printing press. At a time when everything is electronic, if you want to make it a three-article requirement to get a PhD, people should feel free to get creative. Many things can be an article: a data article, a software article, an interactive review tutorial that remixes other people’s work into something new and educational. I have a strong belief that the way we publish is very wasteful and we could be doing much better. But somebody needs to lead by example.
PW: Do we need to redefine the quanta of science into science units if we change from the static immutable papers? How can we rebrand that?
NS: It would be easier to make a case if you show how widely cited and consumed your research object is by showing alternative metrics and statistics, such as how many times a particular dataset was downloaded or a particular software has been used. If the software gets cited in regular papers, then the people who have written that software should get just as much credit as the people who wrote the paper that cited the software.
Honestly, what I would rather like to see than people spending time to “salami slice” one work into three PDF’s, is for them to divide their work into a preregistration, a data paper, a software paper, and a wrap up PDF discussion. Every research object should have at least a couple of components that introduce transparency in their experiment.
PW: Can you share some particular example of data sharing or discussion openness that you have been part of. How has that turned out?
NS: My first paper was published in 2011, and it was on measuring the g-ratio using magnetic resonance imaging. That paper was primarily theoretical and I would consider it to be my preregistration for g-ratio measurement using MRI. It was theoretically sound, exciting and we decided to submit it for publication. Then I did my postdoc, which took four years and resulted in an experimental g-ratio paper in 2015. By that time, we already had data and validation which was done with histology. It was kind of a follow-up that said that what we proposed in 2011 actually makes sense. We then made the data and the software that we used publicly available, and submitted a data paper and a software paper. Finally, after a few replication studies conducted by other researchers, we wrote a review article that summarized the g-ratio field and all the work that went into it from 2011 to 2018. I feel that the g-ratio project is an example of “salami-slicing” that can work. The project is still alive, different people are working on it and I feel that down the line it could show the way for slow science that is high quality and reproducible.
PW: High quality and reproducible – would it be possible to take a hardware or software project and add an open framework to get to same result? Have there been efforts to pitch this to vendors and say that you can still do something different and exciting even though we are pushing to use this highly open framework?
NS: You are getting to the last part of this pipeline that I am trying to build for reproducible science. It is true, a lot of the vendor work is not open. Making sure that sequences are compatible is not easy. I have many people in the lab that are actually working on improving this.
One way we see it happen – I am not saying it is the best solution but it is one that we are passionate about – is working with a third-party software that can run on different vendor platforms. qMRLab, the software we are developing for bringing quantitative MRI under one umbrella, is vendor agnostic. We are working with a Silicon Valley startup called HeartVista to deploy an instance of qMRLab both on Siemens and GE scanners. Of course there will be differences in the hardware implementations, but it will bring us closer to qMR maps that can reproduce across sites, which is the holy grail of quantitative MRI.
PW: What recommendations would you have for someone who is thinking whether to write a paper, versus going towards creating something that others can use or recognize and understand?
NS: The approach I take here is trying to get people to get involved. Building a community is the hardest part. It is about standardization, agreeing on the way that we give credit and get promoted, which is difficult to do without some social engineering. There is a saying that a lot of work on standardization is 20% technical and 80% social.
Over the past 4-5 years I have kickstarted a couple of grassroots outreach initiatives (MRM Highlights, the OHBM blog) that try to get people to talk about their science in lay language in a forum that is endorsed by a society, such as the ISMRM or OHBM. I am also organizing and participating in hackathons where people can come together and work collaboratively on an MRI-related project. All of this culminated in an initiative called the Canadian Open Neuroscience Platform (CONP) where I am leading the platform’s communications and publishing branch together with Pierre Bellec and Rachel Harding.
Among the CONP communications initiatives is a series of podcasts about open science, as well as the NeuroLibre platform for publishing reproducible research objects. We are also working with the journal MRM to promote reproducibility. We recently revived MRHub, to make it easy for authors of papers in MRM to submit supplementary materials, including their data and their code. I feel like there is a critical mass of people that are finally coming together to build something new. And hopefully the community listens and contributes.
MAKING THE LEAP IN IMAGING PERFORMANCE
Department of Electrical Engineering