The first question to answer is "what is the meaning of replication". A simple naive belief in replication is that it refers to a phenomenon which can be demonstrated at the 95% confidence level in every single experiment.
Unfortunately, in science involving huge numbers of uncontrollable variables such as human beings, this sort of replication almost never happens. Instead, replication is a statistical phenomenon.
In order to illustrate this phenomenon, Dean Radin selects the example of studies on aspirin as a preventative for second heart attacks in his seminal book on the meta-analysis of psi, The Conscious Universe. Today, everyone knows that aspirin is an effective preventative treatment for heart attacks. Why is this an accepted scientific fact? Because a large meta-analysis of multiple studies comparing aspirin to placebo showed an overall significant effect far beyond the chance expectation. See Radin's figure 4.2 below. Note that the vertical line with horizontal endpoints in these charts shows the 95% confidence interval with the actual measurement value in the center.
Notice that only 5 of the 25 individual studies actually returned statistically significant results on their own. If we relied on statistical significance of individual studies, we would say "aspirin's effects on heart disease can't be replicated" because of all these individual "failed studies". In fact, 3 of the 25 studies showed a (non-significant) negative effect from aspirin versus placebo! That is why we need to use a meta-analysis of studies from multiple independent researchers. The combined meta-analysis clearly shows us that aspirin has a statistically significant effect in preventing heart attacks. Aspirin therapy has gone up against the most rigorous examination possible and come out with the scientific seal of approval.
So what happens when we examine the evidence for psi phenomena? Certain categories of psi experiments have been extensively conducted at independent institutions by seperate research teams. These psi phenomena have all been subject to meta-analysis by Dean Radin and other independent meta-analysists, including skeptics such as Ray Hyman. And the effects are astronomically significant. For certain types of experiment such as the Ganzfeld and auto-ganzfeld, the time, effort and expense means that most of the data which could have been collected has been included in these meta-analyses, so no possible "file drawer" effects can even exist.
Below I have reproduced Radin's charts for meta-analysis of dream telepathy experiments, the 1985 Ganzfeld meta-analysis by Hyman and Honorton, an updated Ganzfeld meta-analysis, high-security ESP card tests, RNG PK experiments and dice-rolling PK experiments. Although all of these meta-analyses include data from trials showing non-significant effects, the overall meta-analysis is clear. These phenomena all show enormous, often astronomical deviations from the null hypothesis.
So the answer is clear. Certain psi phenomena have gone up against the most rigorous examination possible and come out with the scientific seal of approval. So why do so many scientists and "rationalists" think that psi is "nonsense", "without a shred of real evidence"? I'm afraid that is more of a sociological question than a scientific one.
6 comments:
Well, I won’t rehash what I wrote before about Radin's ganzfeld meta-analyses, but if anyone’s interested, this thread at the JREF forums is a good place to start.
http://forums.randi.org/showthread.php?t=70950
On a wider perspective, Kennedy has recently written a couple of interesting articles in which he discusses the use of meta-analyses in parapsychology. In 2004 he wrote “Meta-analysis is ultimately post hoc data analyses when researchers have substantial knowledge of the data. Evaluation of the methodological quality of a study is done after the results are known, which gives opportunity for biases to affect the meta-analysis. Different strategies, methods, and criteria can be utilized, which can give different outcomes and opportunity for selecting outcomes consistent with the analyst's expectations.” ("A Proposal and Challenge for Proponents and Skeptics of Psi", Journal of Parapsychology, vol 68)
Meanwhile in 2005 he wrote “One of the most revealing properties of psi research is that meta-analyses consistently find that experimental results do not become more reliably significant with larger sample sizes as assumed by statistical theory (Kennedy, 2003b; 2004). This means that the methods of statistical power analysis for experimental design do not apply, which implies a fundamental lack of replicability.
This property also manifests as a negative correlation between sample size and effect size. Meta-analysis assumes that effect size is independent of sample size. In medical research, a negative correlation between effect size and sample size is interpreted as evidence for methodological bias (Egger, Smith, Schneider, & Minder, 1997).
The normal factors that can produce a negative correlation between effect size and sample size include publication bias, study selection bias, and the possibility that the smaller studies have lower methodological quality, selected subjects, or different experimenter influences. All of these factors reduce confidence in a meta-analysis. However, for psi experiments, the failure to obtain more reliable results with larger sample sizes could be a manifestation of goal-oriented psi experimenter effects or decline effects (Kennedy, 1995; 2003a). Even if these effects are properties of psi, parapsychologists cannot expect that other scientists will find the experimental results convincing if methods such as power analysis cannot be meaningfully applied. Further, for the past two decades, the debates about the reality of psi have focused on meta-analysis. The evidence that psi experiments typically do not have properties consistent with the assumptions for meta-analysis adds substantial doubts to the already controversial (Kennedy, 2004) claims about meta-analysis findings in parapsychology.” (“Personality and motivations to believe, misbelieve, and disbelieve in paranormal phenomena”, Journal of Parapsychology, 69)
More of his work can be found on his site:
http://jeksite.org/psi.htm
One of the most revealing properties of psi research is that meta-analyses consistently find that experimental results do not become more reliably significant with larger sample sizes as assumed by statistical theory
It would be interesting and see if the same holds true with medical clinical studies. I suspect many other areas of science show the same kinds of "anomalies" in their statistics as psi research.
What are you basing your suspicions on?
Kennedy is someone who's been involved in parapsychology for many years and now works in pharmaceuticals. Since the point being made by Radin (and others - I've seen this argument several times) is that compared to the data for aspirin and heart attacks, psi meta-analyses are every bit as robust. But Kennedy is now, with direct experience of both fields of research, saying this is not the case.
He wrote: "One of the most revealing properties of psi research is that meta-analyses consistently find that experimental results do not become more reliably significant with larger sample sizes as assumed by statistical theory" If you take the graph of aspirin experiments and rearrange them so the largest (ie, with the smallest confidence interval) is at one end and the smallest is at the other, then you’ll see that it approximates a funnel. In other words, as the number of trials gets larger, so the results converge on one level. (If I remember right, there is one experiment that sticks out a bit, though). Do that with the ganzfeld data and there’s no such shape. Radin has a funnel graph in The Entangled Mind, but since his meta-analysis has no inclusion criteria, it is largely meaningless.
If parapsychology wants to be measured by the same standards as mainstream science, which is the point Radin is trying to make, I believe, then surely points such as "In medical research, a negative correlation between effect size and sample size is interpreted as evidence for methodological bias" should be given as much weight as the more positive conclusions.
Radin has a funnel graph in The Entangled Mind, but since his meta-analysis has no inclusion criteria, it is largely meaningless.
How do you know that it has no inclusion criteria?
What are you basing your suspicions on?
That experiments in all different areas of probabalistic multifactorial systems show sheep/goat effects, funding bias effects and the like. I suspect psi is itself responsible for this, and not just sloppy science and bogus post-hoc data analysis.
The Entangled Mind m-a is an update of the Conscious Universe m-a. The CU work had no inclusion criteria that was applied across all experiments. As such I don't think it qualifies as a proper meta-analysis. I go into more detail on the JREF forum (link in first comment).
Post a Comment