Replication as you describe isn’t done in most fields, that’s part of the “serious effort and funding” I mentioned. If I am applying for a grant to do research what do you think gets funded? Novel proposition or rudimentary replication? Funders want to be a part of glory just as much as institutions which is part of the systemic issue here.
There are researchers that aim to replicate but the numbers of them have shrunk across all fields because funders and universities are pushing for novel research.
Aside from this though one does not need to fully replicate a study to disprove it. In both the studies I pointed out people were sounding alarms for years about discrepancies in the data that in wakefields case should not have passed peer review. The lesne paper is more subtle and one could argue it still should’ve been caught in peer review. But in both cases it took ages of people saying “hey hey hey this shit is fucked” and that is the problem. In the case of Wakefield it was more decisive, in the case of lesne it was more insidious (kind of a sunk cost fallacy because the field bought into the hypothesis without verifying so hard)
The thing that always annoys me in this toxic cycle is the insistence on applied research. I’ve seen people across a few different fields run into this problem.
Let’s say that they do some really interesting applied research, where they build on existing basic research to come up with some really cool applications. Yay, science! But this brings them to the boundary of what we know in that area — there’s no more basic research to build upon. What they need to do (and what is very clearly cued up by what they just published) is take this applies research and just do a bunch of structured “fuck around and find out” and see what happens, hopefully producing some additional basic research that they, or other researchers, can then figure out how to apply that in interesting ways.
But noooooooooo. It’s like that meme comic with the dog where it has a frisbee and it says “no take, only throw”. Everything you make has to be useful, or you will struggle to get funding. The area I know most about this is in protein structure stuff, and it drives me mad to see papers complaining about how many potentially druggable targets there could be in the “dark proteome” — the large array of human proteins that we don’t know shit about. Countless papers lamenting how we’re not researching proteins where we’re most likely to find new and useful stuff, but rather we’re just doing more and more research on proteins we already know a heckton about, i.e. “searching in the areas where we have the best light”[1]. But of course people are doing that, when someone who wants to go and search in the dark are expected to produce useful results right away.
The way it’s meant to work is that some people go spelunking in the dark, and they say “hey, I might have found something here”, and that causes other people to head over there to shed light on the area so we can evaluate things better. We need to start somewhere!
[1]: To be clear, I’m not blaming the researchers who write these papers or editorials, because there’s very little that they can do to change it. Hell, writing these papers is likely their attempt to change this unreasonable system of expectations. Unfortunately, the root problem here is how capitalism and our funding model for research leads to toxic cycles such as “publish or perish”.
Replication as you describe isn’t done in most fields, that’s part of the “serious effort and funding” I mentioned. If I am applying for a grant to do research what do you think gets funded? Novel proposition or rudimentary replication? Funders want to be a part of glory just as much as institutions which is part of the systemic issue here.
There are researchers that aim to replicate but the numbers of them have shrunk across all fields because funders and universities are pushing for novel research.
Aside from this though one does not need to fully replicate a study to disprove it. In both the studies I pointed out people were sounding alarms for years about discrepancies in the data that in wakefields case should not have passed peer review. The lesne paper is more subtle and one could argue it still should’ve been caught in peer review. But in both cases it took ages of people saying “hey hey hey this shit is fucked” and that is the problem. In the case of Wakefield it was more decisive, in the case of lesne it was more insidious (kind of a sunk cost fallacy because the field bought into the hypothesis without verifying so hard)
The thing that always annoys me in this toxic cycle is the insistence on applied research. I’ve seen people across a few different fields run into this problem.
Let’s say that they do some really interesting applied research, where they build on existing basic research to come up with some really cool applications. Yay, science! But this brings them to the boundary of what we know in that area — there’s no more basic research to build upon. What they need to do (and what is very clearly cued up by what they just published) is take this applies research and just do a bunch of structured “fuck around and find out” and see what happens, hopefully producing some additional basic research that they, or other researchers, can then figure out how to apply that in interesting ways.
But noooooooooo. It’s like that meme comic with the dog where it has a frisbee and it says “no take, only throw”. Everything you make has to be useful, or you will struggle to get funding. The area I know most about this is in protein structure stuff, and it drives me mad to see papers complaining about how many potentially druggable targets there could be in the “dark proteome” — the large array of human proteins that we don’t know shit about. Countless papers lamenting how we’re not researching proteins where we’re most likely to find new and useful stuff, but rather we’re just doing more and more research on proteins we already know a heckton about, i.e. “searching in the areas where we have the best light”[1]. But of course people are doing that, when someone who wants to go and search in the dark are expected to produce useful results right away.
The way it’s meant to work is that some people go spelunking in the dark, and they say “hey, I might have found something here”, and that causes other people to head over there to shed light on the area so we can evaluate things better. We need to start somewhere!
[1]: To be clear, I’m not blaming the researchers who write these papers or editorials, because there’s very little that they can do to change it. Hell, writing these papers is likely their attempt to change this unreasonable system of expectations. Unfortunately, the root problem here is how capitalism and our funding model for research leads to toxic cycles such as “publish or perish”.
1 ↩︎