The changing face of Open Science (and the OHBM Open Science SIG)
Next year marks 10 years of the OHBM Open Science Special Interest Group (OSSIG). In that time, the landscape of open science has evolved and so have the priorities of the OSSIG. In preparation for OHBM 2024 in Seoul, the OSSIG embarked upon a reflective journey through conversations with its past and present OSSIG chairs.
Here we met with: Cameron Craddock, first Chair of OSSIG (2015-16; Currently Machine Learning Engineer at Meta), Kirstie Whitaker (Chair from 2019-20; Currently Program Director for Tools, Practices and Systems at the Turing Way), Janine Bijsterbosch (Chair from 2021-22; now Assistant Professor at Washington University, St Louis), and the present Chair Nils Muhlert (Senior Lecturer at the University of Manchester). We found out what drove them to get involved, what they think they achieved, and how they see the future of open science.
What initially motivated your involvement in open science and the OSSIG?
Cameron Craddock (2015-16):
Nolan Nichols filled out the paperwork and put my name down as the first OSSIG Chair. I visited Daniel Margulies in Berlin, where he told me he was working on ‘the NeuroBureau': an idea for couchsurfing meets neuroscience and academia. If you wanted to collaborate with someone in Berlin, you would find somebody who had a couch you could sleep on, then go there and do that. When you're a postdoc, or PhD student, those opportunities to go stay someplace, when you have no money, can be golden. You meet new people; it can be invigorating, exciting, and you can learn a lot.
After visiting Daniel, I went to Magdeburg and I thought more about these ideas. I thought, “if we want to do a NeuroBureau, we should have a party.” So we organized the first NeuroBureau party at OHBM 2010 in Barcelona on the beach at a little kiosk, which had a really good turnout. That's how the NeuroBureau began and through that, we began to think more about open science.
There was much discussion about how we do science when we don't have access to money. That's really what it comes down to. You could say it's access to data, or access to a scanner. You can say it's access to an expert in the field, you can say it's all those things. But at the end of the day, it really comes down to money. For a postdoc or PhD student the question is how do we get exposure to the things that will help build our careers? There are large segments of the world who aren't able to participate in neuroscience or any sort of science, because no matter how smart they are, they don't have the right resources available.
Historically, data was a way that some scientists could differentiate themselves from others, and maintain that differentiation by having access to better data. Our thoughts on this were carried out very much in concert with Mike Milham and Bharat Biswal, and their effort to do the 1000 functional connectomes. That was a lot about the science, but not from my perspective of 'we're broke, how do we get data'. Instead Mike and Bharat asked, ‘how do we build 1000 datasets, so that we can do things like look at inter-individual variability in a meaningful way? How do we get those datasets, knowing that if one individual lab tried, it would take a really long time to do, cost a lot of money, and risk having issues with the data.’ It's really hard to do. But by crowdsourcing it, by getting different people to add their data together, you can build a much larger dataset much more quickly.
We then had the first BrainHack at OHBM in Seattle, led by Nolan Nichols. Daniel and I, and others in the BrainHack community, paired together with OHBM and started organizing these activities. Later, JB Poline, who was Education Chair, contacted us; there was consternation about how brainhack and OHBM fit together. BrainHack was getting funding directly from OHBM but we wanted to leverage BrainHack independently of the OHBM conference. JB Poline encouraged us to create an open science SIG. This gave OHBM a permanent home for open science. It would give us a way for the OHBM hackathon, and the Open Science Room, to get money from other people, and to make it a serious thing.
Kirstie Whittaker (2019-20):
I was a postdoctoral researcher at the University of Cambridge in the Department of Psychiatry when I really leaned into open science. That was when I was a member of the leadership team for the OSSIG.
I received a fellowship from the Mozilla Science lab to be an Open Science fellow in 2016, and in the same year became a research fellow at the Alan Turing Institute: the UK's National Institute for Data Science and AI. My applications for those fellowships, whose deadlines fell 2 days apart, both said that I wanted to make all neuroscience research reproducible. I was specifically thinking about cognitive and structural brain imaging, the areas I worked in. But really I had high ambitions for reproducibility.
I would say, very little is reproducible, because it's incredibly difficult to make work reproducible. I work on the skills required to make something reproducible. I think a lot about community building, about making sure that the work we create is reused and actually demonstrates a real world impact rather than just finishing in a nice written paper. I care about building open source tools and I care very much about continuing the work that I started as a member of the OSSIG for OHBM, thinking about data standards, and the governance of communities that are coming together to try to build something through consensus.
Janine Bijsterbosch (2021-22):
I often visited the Open Science Room at OHBM, and attended the BrainHack once. It was a really inviting, welcoming group of people to be around which was the initial attraction for getting involved. The second thing was that the values of the group—data and code sharing—aligned with the values I prioritized as a scientist. In general, I've been open to other people rather than being competitive. That’s something I really like in science: the team science attitude. It's where the most fun happens and where the most value is created.
When I entered OSSIG in 2021 as Chair I had been in a faculty position for a couple of years. I saw the email pop up about signing up for the SIG. I nominated myself for treasurer or secretary. When I interviewed with Aki Nikolaidis I was asked if I would be interested in being chair. I didn't know. They said "we think you'd be great as chair." So I thought, you think so? I didn't expect to be chair at all.
I was happy to step up to that. The whole experience exceeded my expectations. The community was even stronger than I already knew. Especially in terms of the willingness to come together and get something done. The volunteers on the committees, but also at the BrainHack and OSR, are really impressive. They show real commitment and self organization. The community exceeded my, already high, expectations.
Nils Muhlert (current Chair):
I originally applied to become what I thought would be a ‘general member’ of the OSSIG. I’m the section editor for preregistrations at Scientific Reports, and had previously been heavily involved in the Communication Committee (ComCom), ultimately serving as Chair, so I was keen to help promote open science initiatives particularly within OHBM. After an interview with Koen and Andrea (then Chair and Secretary, respectively) I was asked to take on the Secretary-elect role. Through the course of the year, the Chair-elect, Cassandra Gould van Praag, realized her career path was taking her elsewhere. So I helped out by taking up the Chair-elect role.
There was a lot of initial trepidation. Many past chairs had been leading lights in the Open Science world. I felt a weight of expectation on having big ideas and a dramatic impact on ongoing activities. But my role in OHBM has always been more pragmatic, providing the setting to facilitate others. In ComCom, I helped set up the ‘How to’ series of free educational blogposts, the podcast, and many blogposts (~50 in one year). My plan on becoming Chair-elect, and now, is to work out how we can improve the reach of the OSSIG, to make more people aware of what happens at the BrainHack and OSR, and to get them involved in open science activities.
2. How have Open Science goals and initiatives evolved since your engagement began?
Cameron Craddock:
The influence of Open Science has become much more central to OHBM than it was before. The Open Science Award is a great way to recognize people for their contributions and comes with a sizeable monetary award. It's an accomplishment to get it. It’s also an accomplishment that the OSSIG still exists after 10 years. It hasn't been run into the ground, which is what everybody thought. Not only that, but it exists under its own power. It’s great that the torch has been passed and other people are involved in creating it. OSSIG has its own mind. That's the biggest compliment you can have, actually creating something.
In terms of how Open Science has evolved, now it has some very large institutional support from the Montreal Neurological Institute and others. When I woke up in Open Science, there was all the excitement, all the free tools, free data, the raw data, the processed data, free journal publications. I got fully satiated on all those things. But now the major issue comes down to money and sustainability-particularly for jobs.
I once gave a talk at the American Academy of Child & Adolescent Psychiatry (AACAP). A postdoc asked ‘how do you make a career out of open science.’ I said, “well, it's easy, it worked out for me.” Russ Poldrack was there and said, “but Cameron, do you think it'll work out for everybody? Do you think there really are enough positions for people to make their career out of this?” Russ’ observation back then was very true. Open science requires an investment. Somebody has to give away their data, to spend their time curating their data, to write the software tools, to debug and fix them. All these things are required in order to sustain open science, and that's a really big issue.
Early on with the NeuroBureau and the OSSIG, I used to advocate that people who don't contribute money, contribute their time. There are many great tools that people develop and put online, but often the tools starve because nobody uses them. Somebody's got to pay the bill to maintain these tools. The question is who?
In the US, the federal grants we rely on are not built to sustain long term projects or tools. The whole goal is, if it's good enough, then you should be able to make money out of it on the free market. That's the long term thing. If you say: “this is important, but I can't make money from it.” I'll say: “well, obviously, it's not that important. If it was, people would be willing to pay for it.” A grants manager once asked me: “tell me a fundamental technology that we need that isn't capable of generating its own money.” I said, “libraries.” His reply to me was: “libraries are screwed.” He didn't say that, he said a word which I turned into the acronym for fundamental, unsustainable, critical technology.
We do have a few exceptions, for example AFNI was developed and is maintained in-house at NIH; that's special. Similarly, there’s FSL at FMRIB and SPM at UCL. The reason why we have so few tools like that is because there's not a lot of people willing to put in enough money to really create the tools. The people involved in these tools have received criticism in the past about open sharing. But sustainability has always been top of mind. Selling it to the people is a good sustainability model. I agree - some people who use open science are flush with cash, it's reasonable to expect those people to pay to use those tools.
Kirstie Whittaker:
There's this really fascinating tension around open science. Just the way that science is supposed to be done. I used to joke that I would go back to doing brain network analysis after I'd solved this reproducibility thing. Reproducibility is where my skills are best focused. But it's also a precursor to doing good brain imaging analysis. Until we can figure out how to do reproducible brain imaging analysis, you don’t need another person doing more non-reproducible or small scale analyses.
I love doing brain network analyses. I love thinking about brain development in teenage years, how it relates to mental health. It was a fascinating space to be, and I loved the conversations that we had. But I thought, ‘I'll try and fix this infrastructure situation first.’
I don't make that joke anymore because I think I'm at a point where I don't really think we're anywhere close to leveraging openness in a way that maximizes the societal benefit. My belief is that the solution to having the whole scientific and research ecosystem working for the social good requires an anti-capitalist stance. That's the realization that makes me think I'm probably not going to be returning to brain network analysis anytime soon. That anti-capitalist stance is really important and becomes harder and harder to hold when you have more people that are relying on you for their job. It becomes much harder to be outspoken.
But I'm really proud of the fact that people listen when I say things. It was definitely not like that in the past. Being on the Open Science SIG was a platform for me, it elevated me up to allow my advocacy work. But how do we make space for the next generation to come through while also remembering to keep advocating for these long, slow infrastructure changes that we're looking for? I don't think anyone's got a solution for that.
My advice is to ask yourself a question: “How do I work inside the current system, where I’m only one person but expected to make a difference?” My answer, since 2016, is to do what you can to the best of your ability. Bear witness to the compromises that you're making and change the system so others don't have to make those compromises again. I have huge amounts of privilege that allow me to do my work. I can listen to others who point out areas of privilege that I might not recognize and make sure to dismantle those barriers for others, so they don't have to have those same privileges as me.
Janine Bijsterbosch:
I joined at a weird time. My first year, as incoming chair, was during the pandemic, it was online only. When I was acting chair, it was the first year back after the pandemic. It evolved a lot during that time, in that we were thinking about hybrid. We purchased equipment for live streaming, so speakers and viewers can attend remotely, and spent a lot of effort thinking about how to make hybrid work, how to engage people both in and outside the room.
Since finishing being chair I have become an editor at Imaging Neuroscience. For me, that's part of my personal open science journey. I was part of the negotiations with Elsevier, asking them to reduce their author fees, and then part of the team that quit that position when Elsevier wouldn't, even though I was really proud to be an editor at NeuroImage. I share those values, even though I'm sad about it. Then we started a whole new journal, Imaging Neuroscience, and I'm proud of that. It was amazing to see the outpouring of support on twitter, to see its impact even in entirely different academic communities. So for my personal involvement in open science, that was a really exciting chapter.
Nils Muhlert:
Even from 2015, there was tension and resistance to open science practices, like preregistration. People were forcefully saying, “you're putting science in chains, taking out the spontaneity, the ability to discover fortuitous associations that turn out to be meaningful.” Senior academics argued: “if you have to preregister, you'll miss out on those findings/opportunities.” The counterargument was: “just clearly label posthoc findings.” There was a strong sense of there being two sides: people pushing both for and against it.
Over time, there has been more acceptance of these changes in scientific practices —acknowledgement that the traditional practices were far from optimal. Some journal reviewers now regularly ask whether studies were preregistered.
So there was a sense of a revolutionary wave of people, fighting against the establishment to promote open science initiatives. My sense is that some people almost wish those revolutionaries were still around. People who were coming up with great, but sometimes controversial, ideas.
If you look at the OSSIG now you see a huge number of people from all over the world. From them you find out that scientific practices have moved on at a global scale, that these open science practices are more accepted, but that it differs by country and by region. Some work with funding bodies that are less enthusiastic about data or code sharing. They may not be as motivated to ensure that's part of the funding process. There have been huge shifts in attitudes towards open science, but there's still regional disparities and progress to be made.
3. What are your aspirations for the future of open science, both personally and for the next generation of researchers?
Cameron Craddock:
For me, the most exciting opportunities in open science are the BrainHack space, the open collaboration, tearing down walls that prevent collaboration. When I look at the preprocessed connectomes projects that I was involved with, I always loved seeing the number of PhDs or masters students using that data, and doing very different things with it. Some people were doing neuroscience, others more computational projects. People also came from countries where they may not have been able to do anything with neuroimaging data, based on infrastructure in those countries. To me, that's the huge thing about open science: the ability to educate people, and allow them to participate when they don't necessarily have that background. Going to a BrainHack, you see people learn through that experience. The thing they learn the most, in my opinion, is confidence. More confidence in the work they're doing, more confidence in engaging with other people. Many people built relationships through BrainHacks that ended up being mini clusters of people that have worked together for the rest of their lives in collaboration.
Keeping the OHBM BrainHack is critical. The Open Science Room, in the beginning, was a bit murky in terms of what we would do there. It's become a bit of a town hall. Towards the end of my time there were some controversial ideas that would get debated, or there'd be ad-hoc talks. I really liked those social aspects of open science. Maybe that gets back to why we started with the party as our first foray.
The future of open science will be about figuring out sustainability. It may be that you end up with a few purveyors of open science. So it almost comes back to the silos: the Open Science places. If you're not working in one of those, then you're not really producing open science.
That's what the funding issue dictates. The NIH is working towards making every grant have a data sharing plan; however, I don't know that they've ever enforced them. I know that when the National Database for Autism Research first started off, which is a NIMH database for autism data, there were several grants that were given, and they specifically wrote in those grants that they had to share their data to this database. There were PIs at that point that turned down the grants. They felt like that was an excessive requirement. So, the problem is how you can have it, how you can sustain it.
Kirstie Whittaker:
The biggest challenge for the future of Open Science is precarity. People have short-term contracts, and even faculty are reliant on relatively short term funding to allow them to do that work. That keeps a focus away from sustainability maintenance, from building a culture where it's okay to change what we learn, or to come up with one answer a year and then bring in a new method or new data. Or even to think deeper about it and change that perspective based on new evidence.
Precarity in the system is probably the biggest problem that sits at the base of everything else. It all comes down to the reason that I don't adopt open science Practice X, because I need to pay my rent and I can't move because I have family etc. That’s probably the biggest challenge.
Another really interesting challenge relates to novelty and skill building. There's not that much new that needs to be done. There'll always be some new elements but there's an awful lot of supporting people to go to varying levels of depth, and understanding within different dimensions of open science.
In an academic career, we tend to draw a very strong boundary between skills and teaching, and research. We need to explore that more, because it maintains a mental model of research that is yoked to individuality, yoked to novelty. That’s toxic for the whole ecosystem. So why don’t we value skill building as a component part of a team? Why is that siloed off away from a successful research career? That’s an interesting challenge.
Ultimately, adopting an open research approach is anti-capitalist, and we live in a capitalist society. So I think we probably need to remain comfortable with being uncomfortable, talking about how a funded research ecosystem sits in the societies that we have around the world. While we pretend that we are not part of a capitalist system, we're kidding ourselves. We won't be able to clearly see the things that need to change. While we stay believing that capitalism is the only way to fund and conduct research, and to incentivize good work, we limit ourselves from options that I think could exist.
Janine Bijsterbosch:
I'm currently trying to shift the culture. Locally, in Wash U, I'm on a culture survey task force — trying to contribute to ways that policies can improve researcher's experiences. The key will be to make changes within leadership. Stepping up to roles as they become available and trying to have a positive impact on how people experience academia. I try to put work into open science — into co-chairing, preregistration, but also bigger picture collaboration and team science, sharing, work-life balance. I’m using my leadership to actively improve culture, that’s the most important thing going forward.
The OSSIG is bleeding edge in pushing toward culture change. OHBM is perhaps being dragged along, and the publishing community is possibly behind that, and grant funding bodies are perhaps behind that, then institutions and they're all being dragged along. How do we make sure these are all getting dragged along? Is there something we can do to tighten the chains so one isn't holding all the others behind? But maybe the slow dragging along approach is right, and will lead to change over the years. The challenge is to ensure all the expectations are aligned.
If I was to give early career researchers advice, I would say: “As an ECR you're not going to change this — power structures aren't going to disappear.” For senior people it’s easier to push for these changes. But activism is helpful at all levels. Still, if you have power, use it. If you don't yet have power, then do what you feel is right, whilst also being strategic.
Nils Muhlert:
Personally, I want findings that replicate. I want insight into behavioral or cognitive phenomena, and a real understanding of why they happen, where the boundaries are. Not in a flippant way — as in, not just being used to explain potential differences between disparate published studies. But careful analysis of how effects differ when you hold variables constant. How can we quantify those effects, using various initiatives in open science to improve how we do that.
As part of the OSSIG, I want to create resources for identifying appropriate Open Neuroimaging datasets, and to reflect on what open science initiatives aimed to achieve, how they’ve done, and discuss future trajectories. This blogpost forms part of that. I also want to promote the ideas of the OSSIG team and broader community. For instance, our Chair-elect, Stefano Moia is promoting cross-talk with other open science groups — e.g. those at ISMRM. How do we usefully combine our work? Sina Mansour has expertly crafted this year’s BrainHack activities, and Selma Lugtmeijer has set out symposia at the Open Science Room that will generate useful reflections and discussion — on how open science is funded, regional disparities and the future of open science. On the latter note, we’re planning an open mic session in the OSR, where you have 5 minutes to convince the audience about your new open science initiative. Should be thought provoking!
So the immediate future is to develop ideas and set-up for the OSSIG teams of 2024, 2025. A key goal for me is to increase awareness of these events. Sometimes that's been tricky. There have been amazing sessions in the Open Science Room, of broad interest, and opportunities for discussion and audience participation (not often available in other OHBM symposia) that get missed. There’s live-streaming of activities, allowing remote participation, that again, many people aren’t aware of. So greater visibility at the OHBM annual meeting would help.
Over the longer term there are pressing problems we need to address. There's attribution and credit, appropriate reward for open science involvement and initiatives in careers. But there's also the challenge of ensuring we're getting new people interested and invested in open science. These issues are of course linked. But there are great benefits for getting involved in society groups and committees — benefits in building networks and gaining new skills, sometimes missed when only working within an institution or with other local groups. Networking to me has never been about approaching very senior names. It's been about finding peers that share my values and research interests. Those have been the most productive relationships I've developed in my career. So for ECRs who find all this interesting, they should consider putting themselves forward, helping to shape how open science—and science more generally—develops into the future.
In summary, the leadership of OSSIG over the years demonstrate a diversity of opinions, reflecting the many voices within open science. Yet they all commonly point out the sense of community and the need for sustainability (on many levels). Join us at the BrainHack and Open Science Room this year to contribute to these discussions, to help drive the change, and to set out paths towards better science. And for those that would like to volunteer — consider signing up here.