Avoiding Messiness

According to M. Szlazak on the Skeptiko Forum, clinicians are now running away from meta-analyses in favor of single large-scale experiments to test and make a "final resolution" on questions of clinical effectiveness. He cites this paper and this open letter in support of his position.

Frankly, this does not surprise me at all. After all, the desire to have a simple worldview with black and white answers to every question is very tempting for human beings. And when different researchers conduct the same kinds of experiment, they often get contradictory results, or wildly different effect sizes, eg:



So it is not at all surprising that many people, especially a self-selected group of people who tend towards systematizing and simplifying reality, want to use methods that avoid raising difficult questions.

If the psi hypothesis is true, then the beliefs of experimenters can have a strong result on the results of their experiments, even when the experiments are conducted double-blind, most particularly when the overall experiment is statistical in nature and the experimental subjects are extremely complex and non-deterministic. Medical clinical trials certainly fall into this realm. Given the demonstrated reality that the beliefs of experimenters do effect the results of their experiments, it is certainly possible that psi correlations might help account for this (as opposed to experimenter fraud, biased errors, biased conclusions, etc. which materialists believe account for funding effects on experimental outcome).

Given the inevitable messiness of real science and the often varying results of experimental trials, it should be expected that some people want to avoid all those questions being raised by setting "rules" for science along the lines of what Topher Cooper suggested, tongue in cheek:

1) Don't reach a conclusion.
2) Completely ignore the data and just go with your faith.
3) Kill anyone who ever looks like they might ever do a second test of any hypothesis of interest.

But, of course, a committment to truth and reality demands that we face all of the facts, not just those that confirm our world-view. So long live multiple independent labs and experimenters. Long live meta-analysis!

Controversy between Dean Radin and Andrew Endersby

Andrew Endersby has been commenting about Dean Radin's meta-analyses for some time on this blog and elsewhere as well, and I have remarked that I would like Radin to respond.

Endersby's position is that Radin's meta-analyses are not done properly, that he does not provide inclusion criteria, and that he has left out relevant studies.

I felt that Endersby's points needed to be addressed by Radin, and asked him to do so in an email.

Radin did not wish to be drawn into a debate on this topic, but made the following points in his emails to me, which I will paraphrase.

1) Endersby's ganzfeld meta-analysis also shows staggering odds against the null hypothesis, so Radin doesn't understand the large controversy Endersby is stirring up over Radin's ganzfeld meta-analyses.

2) Radin has conducted two formal psi meta-analyses, published in the literature, and in those papers the inclusion criteria were formally outlined. Those meta-analyses covered PK experiments using dice, and PK experiments using RNG.

3) Radin's meta-analysis of the ganzfeld and his other meta-analyses (besided the two mentioned above) relied on earlier meta-analyses published in the literature, updated with data from more recent studies. They were not full, formal meta-analyses. So studies left out of earlier meta-analyses by other authors are also left out of Radin's meta-analysis updates.

4) Radin mentioned that he invited Endersby to submit his paper to a peer reviewed journal, and that if and when it gets published, he will consider the controversy important and relevant and respond. Until that happens, Radin doesn't believe that Endersby's critique rises to the level of a genuine controversy, and lacking such a controversy, Radin feels it would not be the best use of his time to respond formally.

I want to thank Dean Radin for his responses to my emails, and Andrew Endersby for bringing our attention to some important missing studies in the Radin and previous Ganzfeld meta-analyses. Andrew has also occasionally pointed out problems in my own research for AMNAP, and I appreciate his attention to detail and his penchant for rolling up his sleeves and doing his own investigations.

Scientific proof of psi phenomena (Repost from AMNAP 1.0)

The phrase "Scientific Proof" is a high standard to acheive. In science, a phenomena is considered proven if it has met the standard of multiple independent replications, as determined through meta-analysis of all the data available. And certain experiments demonstrating psi phenomena have easily met that standard of proof.

The first question to answer is "what is the meaning of replication". A simple naive belief in replication is that it refers to a phenomenon which can be demonstrated at the 95% confidence level in every single experiment.

Unfortunately, in science involving huge numbers of uncontrollable variables such as human beings, this sort of replication almost never happens. Instead, replication is a statistical phenomenon.

In order to illustrate this phenomenon, Dean Radin selects the example of studies on aspirin as a preventative for second heart attacks in his seminal book on the meta-analysis of psi, The Conscious Universe. Today, everyone knows that aspirin is an effective preventative treatment for heart attacks. Why is this an accepted scientific fact? Because a large meta-analysis of multiple studies comparing aspirin to placebo showed an overall significant effect far beyond the chance expectation. See Radin's figure 4.2 below. Note that the vertical line with horizontal endpoints in these charts shows the 95% confidence interval with the actual measurement value in the center.



Notice that only 5 of the 25 individual studies actually returned statistically significant results on their own. If we relied on statistical significance of individual studies, we would say "aspirin's effects on heart disease can't be replicated" because of all these individual "failed studies". In fact, 3 of the 25 studies showed a (non-significant) negative effect from aspirin versus placebo! That is why we need to use a meta-analysis of studies from multiple independent researchers. The combined meta-analysis clearly shows us that aspirin has a statistically significant effect in preventing heart attacks. Aspirin therapy has gone up against the most rigorous examination possible and come out with the scientific seal of approval.

So what happens when we examine the evidence for psi phenomena? Certain categories of psi experiments have been extensively conducted at independent institutions by seperate research teams. These psi phenomena have all been subject to meta-analysis by Dean Radin and other independent meta-analysists, including skeptics such as Ray Hyman. And the effects are astronomically significant. For certain types of experiment such as the Ganzfeld and auto-ganzfeld, the time, effort and expense means that most of the data which could have been collected has been included in these meta-analyses, so no possible "file drawer" effects can even exist.

Below I have reproduced Radin's charts for meta-analysis of dream telepathy experiments, the 1985 Ganzfeld meta-analysis by Hyman and Honorton, an updated Ganzfeld meta-analysis, high-security ESP card tests, RNG PK experiments and dice-rolling PK experiments. Although all of these meta-analyses include data from trials showing non-significant effects, the overall meta-analysis is clear. These phenomena all show enormous, often astronomical deviations from the null hypothesis.

So the answer is clear. Certain psi phenomena have gone up against the most rigorous examination possible and come out with the scientific seal of approval. So why do so many scientists and "rationalists" think that psi is "nonsense", "without a shred of real evidence"? I'm afraid that is more of a sociological question than a scientific one.