In medicine, observational evidence is actually better and far more ethical than the RCT. (Which simply dooms the terminally ill to fake treatment.) You just need large datasets and an agile culture that's responsive to new input.
Don't forget that RCTs are very far from perfect and issues -- sometimes literally fatal issues -- have later turned up via observational evidence in large cohorts. Vioxx, for instance. Many others.
I believe, without the tiniest shred of doubt, that the only trials drugs need to go through are initial safety/toxicity trials (phases 0/1) and that everything else would be much better left to access+observation.
This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case.
Should we demand an RCT before we accept evidence? Of course not. At some point you do have to make a choice on things.
And it should be noted that most drugs do have early cutoff criteria if the evidence is strong enough that it is working. It isn't like people are wanting to withhold good treatments from the world. Adding controls and randomizing them, though, has proven to be highly effective at helping progress.
> "This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case."
If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drugs are used in the real world.
> "Adding controls and randomizing them, though, has proven to be highly effective at helping progress."
I would argue just the opposite. Demands for increasingly byzantine trials have ballooned the costs associated with drug development, and have slowed things to a crawl. There's a reason the field's golden age was in the 1940s and 1950s, and it's not just "low hanging fruit." Today nobody in their right mind wants to work in drug development when they could work in tech or even finance.
Right, but you are just relying on a different form of random, there. The whole point of making controls and then building experiments on changing them, is to get more power from fewer observations. No?
Again, it is off to think that one is automatically superior to the other. Certainly to the exclusion of the other. And that is what feels off with the framing of the parent post. I am perfectly fine saying you should use both observational and controlled trials. But I think it is also wrong to think you don't have to build experiments to test interventions.
This is why you put metrics in your service code. So that you can observe them behave and look for things to change. This is also why you do test cases on your code, so that you can specifically target your change.
Now, I fully back the idea that just A/B testing something doesn't automatically mean you learn something true. But neither does observing a strong outcome on uncontrolled data.
How do you get enough data? If, for example, you need a lot of people in the sample, that might not be so easy. In the abstract, should it not come done to what is the best experimental design for each case?
> Which simply dooms the terminally ill to fake treatment.
I wish people would stop saying this. First, controls aren't necessarily "fake treatment", they are often compared to other standard treatments.
Second, the treatment being tested can actually harm the patient more, therefore the people receiving your alleged "fake treatment" can actually come out better off. Which is the "fake treatment" now?
Experimental designs are critical for obvious reasons but they have a few critical flaws, that mostly all reduce to the fact you can't randomize manipulations with everything. Whether it be due to ethics or practical constraints, you can't conduct a RCT all the time.
This can be more subtly critical than it might seem, in that even if you can manipulate some proxy, often that proxy is insufficient in actually representing the phenomenon of interest, or the conditions under which they actually occur.
I often use the example of videogames and aggression. There were plenty of experimental studies of this but it was always questionable whether lab-induced anger is the same thing as, say, the sort of violence we generally are concerned about societally.
I generally have tried to teach students that experimental designs when done right provide powerful causal evidence of something, but often with limited generalizability; observational designs in contrast provide powerful generalizable evidence of some kind of association, but often with limited certainty about the causal pathways involved.
I've been in a department that was rabidly experimental in its focus and it always seemed sort of short-sighted, because people were idolizing RCTs with proxy manipulations that had questionable generalizability to the real-world phenomena they were trying to model.
Ideally you'd bring both experimental and observational evidence to bear on a question. Your conclusions should be robust to different types of designs.
The reality that individuals and cultures create can be a dark and confining place. Empiricism is our only window to a universe full of possibility and light. Humanity is like a child, standing on our toes to peer through and wonder.
Quite thought-provoking, and connecting it to a related field it seems the (relative) success of LLMs and the likes are indications that enough data can at least learn you something without always needing to interfere with the world first(?)
In medicine, observational evidence is actually better and far more ethical than the RCT. (Which simply dooms the terminally ill to fake treatment.) You just need large datasets and an agile culture that's responsive to new input.
Don't forget that RCTs are very far from perfect and issues -- sometimes literally fatal issues -- have later turned up via observational evidence in large cohorts. Vioxx, for instance. Many others.
I believe, without the tiniest shred of doubt, that the only trials drugs need to go through are initial safety/toxicity trials (phases 0/1) and that everything else would be much better left to access+observation.
This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case.
Should we demand an RCT before we accept evidence? Of course not. At some point you do have to make a choice on things.
And it should be noted that most drugs do have early cutoff criteria if the evidence is strong enough that it is working. It isn't like people are wanting to withhold good treatments from the world. Adding controls and randomizing them, though, has proven to be highly effective at helping progress.
> "This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case."
If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drugs are used in the real world.
> "Adding controls and randomizing them, though, has proven to be highly effective at helping progress."
I would argue just the opposite. Demands for increasingly byzantine trials have ballooned the costs associated with drug development, and have slowed things to a crawl. There's a reason the field's golden age was in the 1940s and 1950s, and it's not just "low hanging fruit." Today nobody in their right mind wants to work in drug development when they could work in tech or even finance.
Right, but you are just relying on a different form of random, there. The whole point of making controls and then building experiments on changing them, is to get more power from fewer observations. No?
Again, it is off to think that one is automatically superior to the other. Certainly to the exclusion of the other. And that is what feels off with the framing of the parent post. I am perfectly fine saying you should use both observational and controlled trials. But I think it is also wrong to think you don't have to build experiments to test interventions.
This is why you put metrics in your service code. So that you can observe them behave and look for things to change. This is also why you do test cases on your code, so that you can specifically target your change.
Now, I fully back the idea that just A/B testing something doesn't automatically mean you learn something true. But neither does observing a strong outcome on uncontrolled data.
How do you get enough data? If, for example, you need a lot of people in the sample, that might not be so easy. In the abstract, should it not come done to what is the best experimental design for each case?
> Which simply dooms the terminally ill to fake treatment.
I wish people would stop saying this. First, controls aren't necessarily "fake treatment", they are often compared to other standard treatments.
Second, the treatment being tested can actually harm the patient more, therefore the people receiving your alleged "fake treatment" can actually come out better off. Which is the "fake treatment" now?
Experimental designs are critical for obvious reasons but they have a few critical flaws, that mostly all reduce to the fact you can't randomize manipulations with everything. Whether it be due to ethics or practical constraints, you can't conduct a RCT all the time.
This can be more subtly critical than it might seem, in that even if you can manipulate some proxy, often that proxy is insufficient in actually representing the phenomenon of interest, or the conditions under which they actually occur.
I often use the example of videogames and aggression. There were plenty of experimental studies of this but it was always questionable whether lab-induced anger is the same thing as, say, the sort of violence we generally are concerned about societally.
I generally have tried to teach students that experimental designs when done right provide powerful causal evidence of something, but often with limited generalizability; observational designs in contrast provide powerful generalizable evidence of some kind of association, but often with limited certainty about the causal pathways involved.
I've been in a department that was rabidly experimental in its focus and it always seemed sort of short-sighted, because people were idolizing RCTs with proxy manipulations that had questionable generalizability to the real-world phenomena they were trying to model.
Ideally you'd bring both experimental and observational evidence to bear on a question. Your conclusions should be robust to different types of designs.
The reality that individuals and cultures create can be a dark and confining place. Empiricism is our only window to a universe full of possibility and light. Humanity is like a child, standing on our toes to peer through and wonder.
Damn fine article, lovely conclusion, a real pleasure to read
Seth Roberts was arguing this ~20 years ago and would have loved the advent of LLMs...
They reference RCT without first defining it.
https://en.wikipedia.org/wiki/Randomized_controlled_trial
I noticed that as well. My eyes feverishly scanning the previous paragraph for the definition.
Quite thought-provoking, and connecting it to a related field it seems the (relative) success of LLMs and the likes are indications that enough data can at least learn you something without always needing to interfere with the world first(?)
> The increasing availability of large datasets should make this an especially good time to reconsider observational evidence in many fields.
This is not happening.
> I was surprised to find that they usually discard papers based on observational evidence wholesale.
He he...welcome to the real world!