This blog post from The Calladus Blog is an example of how "positional" skeptics tend to approach any new information that might challenge their worldview. I found a particularly large number of errors and irrelevancies in this post, and thought it would be useful to analyze them here. So I will be excerpting from the "Nutty Professor" post and providing commentary on what I found problematic at each point.
The Nutty Professor: Dr. Rupert Sheldrake and Telephone Telephony
We are off to an inauspicious start with the namecalling ad-hominem right out the gate. . .
"However, his sample was small on both trials -- just 63 people for the controlled telephone experiment and 50 for the email -- and only four subjects were actually filmed in the phone study and five in the email, prompting some skepticism.
Undeterred, Sheldrake -- who believes in the interconnectedness of all minds within a social grouping -- said that he was extending his experiments to see if the phenomenon also worked for mobile phone text messages."
Notice that Dr. Sheldrake didn't address the problem of a too small sample size for his experiment, and instead immediately widened the experiment to include different tests. This is not a sign of good science!
Woah, there, pardner!
Just because the "yahoo news" article made a claim of "too small sample size" doesn't make it true. This is a fundamental misunderstanding of what Sheldrake was studying. If Sheldrake was seeking to demonstrate that everyone has telephone telepathy, then indeed this sample size is much too small. In addition, Sheldrake did not select a random sample of a population -- he actively sought out individuals who felt that they frequently experienced telephone telepathy. Furthermore, Sheldrake only took the most successful participants from the first phase of the experiments for the more detailed and rigorous videotape trials. He was seeking out a very specific population of people with the best possibility of showing strong effects for telephone telepathy -- the exact opposite of What Mark of the Calladus blog is suggesting by complaining about sample size. Sheldrake's sample size was (deliberately) small, and that is not a problem, as long as his effect size and number of trials is sufficiently large, as they certainly are in Sheldrake' experiment.
This isn't the first time Dr. Sheldrake has been accused of making unwarranted claims based on improper methodology. CSICOP took Dr. Sheldrake to task and debunked his Psychic Staring Effect experiment - where he claimed to show that people could tell, better than random chance, when someone was staring at them.
The link Mark provides offers a normal "explanation" for Dr. Sheldrake's results. However Dr. Sheldrake's rebuttal is also found linked to the "debunking" article:
Colwell et al.'s second experiment was designed to test their pattern-detection hypothesis by using "structureless" random sequences. Sure enough, this time there was no significant overall positive score, although in two of the three sessions there was a highly significant excess of correct guesses in the looking trials.
At first sight, the overall non-significant result seems to confirm their hypothesis. But Marks and Colwell (2000) omitted to mention the crucial fact that in Experiment Two there was a different starer, David Sladen. Can we take it for granted that changing the starer made no difference?
Such experimenter effects are not symmetrical. The detection of Schlitz's stares by the participants under conditions that excluded sensory cues implies the existence of an unexplained sensitivity to stares. By contrast, the failure to detect Wiseman's stares implies only that Wiseman was an ineffective starer. Perhaps his negative expectations consciously or unconsciously influenced the way he looked at the subjects.
In Colwell et al.'s Experiment Two, the starer, Sladen, as one of the proponents of the pattern-detection hypothesis, was presumably expecting a nonsignificant result. His negative expectations could well have influenced the way in which he stared at the participants. It would be interesting to know if Sadi Schršder, the graduate student who acted as starer in Experiment One, was more open to the possibility that people really can detect when they are being stared at.
Other Relevant Experiments
Marks and Colwell claimed that their pattern-detection hypothesis invalidated the positive results of staring experiments carried out by myself and others. If these experiments had involved pseudo-random sequences and feedback, as required by their hypothesis, their criticism might have been relevant. But this is not how the tests were done, as they would have seen for themselves if they had read my published papers on the subject.
First, in more than 5,000 of my own trials, the randomization was indeed "structureless," and was carried out by each starer before each trial by tossing a coin (Sheldrake 1999, Tables 1 and 2). The same was true of more than 3,000 trials in German and American schools (Sheldrake 1998). Thus the highly significant positive results in these experiments cannot be "an artifact of pseudo randomization."
Second, when I developed the counterbalanced sequences that Marks and Colwell describe as pseudo-random, I changed the experimental design so that feedback was no longer given to the subjects. Since the pattern-detection hypothesis depends on feedback, it cannot account for the fact that in more than 10,000 trials without feedback there were still highly significant positive results (Sheldrake 1999, Tables 3 and 4).
Conclusions
In spite of their prior assumption that an ability to detect unseen staring must be illusory, both Baker (2000) and Colwell et al. (2000) in their first experiments obtained unexpected positive results consistent with such an ability. They attempted to dismiss these findings with question-begging arguments. In their second experiments, which gave the non-significant results they expected, an investigator with negative expectations acted as the starer. This arrangement provided favorable conditions for experimenter effects, already known to occur in staring experiments (Wiseman and Schlitz 1997). Both Baker and Marks and Colwell also failed to mention a large body of published data that went against their conclusions. In short, their claims were misleading and ill-informed.
To summarize: Sheldrake takes the SI "debunking" and completely shreds it with a devastating recitation of the facts. The hypothesis outlined in the debunking article is completely at variance with the actual data and experimental findings, which Baker and Colwell would have noted if they had carefully read Sheldrake's research before attacking it. I suppose that is the difference between a "debunking", and a scientific investigation. The latter is an attempt to discover the truth, and the former is simply an attempt to win an argument with no particular regard for reality.
In another paper Sheldrake mentions offering to analyze Marks and Colwell's own data in detail and see if it matches their hypothesis of pseudorandom pattern recognition. Not unexpectedly, Marks and Colwell fail to take him up on that offer.
Dr. Sheldrake also wrote a book called, "Dogs That Know When Their Owners Are Coming Home: And Other Unexplained Powers of Animals" (Amazon link) In this book he attempts to show that dogs can somehow psychically 'tell' when their owners are coming home. The methodology described by Dr. Sheldrake shows that he didn't even attempt to create a 'double blind' experiment, where neither owners, dogs, nor observers knew when the owner was coming home. Instead he allowed the owners of these dogs to record the observations of their pets. He again used a very small sample size.
Again, Mark displays a lack of understanding of this kind of research and the relevant statistics. Small sample size is a canard here. His is also completely wrong about the experimental design, which involved both "observational" and double-blind and randomized components. The researcher who analyzed the videotape data was blind to the experimental conditions, as was everyone in the house (including the dog). Either Mark hasn't read the studies he is criticizing here, a cardinal sin in science, or he read them sloppily and is misinformed.
Instead of publishing to peer-reviewed media, Dr. Sheldrake writes popular books and makes claims and announcements pitched to the media.
Like many other scientists, Dr. Sheldrake does write popular books in addition to his admirable record of peer reviewed publication credits, including multiple articles in Nature, Planta, and the Journal of Consciousness Studies. I'm sure that Mark is not intending to imply that Francis Crick, Stephen Hawking and John Wheeler are bogus scientists, because of their popular science works, or that Nature, Planta and JCS are bogus journals, right?
His results are based on a small sample size, which is at the very limit of detection of effects.
Again, the relevant statistic here is the number of trials, not "sample size". In the videotaped experiments, there were of 271 trials, and were 122 (45%) correct guesses (p = 10-12). This is astronomically far from being "at the very limit of detection of effects"!
He bases some of his conclusions on anecdotal evidence (for example, allowing a dog's owner to record their observations.)
Any scientist uses anecdotal evidence as a starting point for designing experiments. Sheldrake's experiments absolutely do not rely on "allowing a dog's owner to record their observations", but rather use videotapes, evaluated blindly by a third party, to determine the measured data.
He claims that Quantum Theory can explain psychic phenomena, which is a proposed new law of nature since Quantum Theory describes the subatomic, not macroscopic, universe.
Now Mark is confused. The quantum nature of matter is responsible for many of the properties of the macroscopic world, such as the very most basic fact that atoms take up space. You cannot wall off quantum mechanics and say that it is completely irrelevant to macroscopic reality!
He works in some isolation, well outside the mainstream science community.
Sheldrake works with many other researchers, including avowed "skeptics", and has been published in a large variety of journals as I described above.
What I find most interesting about Dr. Sheldrake's supposed skepticism is his refusal to cooperate with noted skeptic James Randi in Randi's Million Dollar Challenge.
Randi is a showman, not a serious researcher. And he has already established a track-record of distortions trending to outright lies with regards to Rupert Sheldrake's research:
The January 2000 issue of Dog World magazine included an article on a possible sixth sense in dogs, which discussed some of my research. In this article Randi was quoted as saying that in relation to canine ESP, "We at the JREF [James Randi Educational Foundation] have tested these claims. They fail." No details were given of these tests.
I emailed James Randi to ask for details of this JREF research. He did not reply. He ignored a second request for information too.
I then asked members of the JREF Scientific Advisory Board to help me find out more about this claim. They did indeed help by advising Randi to reply. In an email sent on Februaury 6, 2000 he told me that the tests he referred to were not done at the JREF, but took place "years ago" and were "informal". They involved two dogs belonging to a friend of his that he observed over a two-week period. All records had been lost. He wrote: "I overstated my case for doubting the reality of dog ESP based on the small amount of data I obtained. It was rash and improper of me to do so."
Randi also claimed to have debunked one of my experiments with the dog Jaytee, a part of which was shown on television. Jaytee went to the window to wait for his owner when she set off to come home, but did not do so before she set off. In Dog World, Randi stated: "Viewing the entire tape, we see that the dog responded to every car that drove by, and to every person who walked by." This is simply not true, and Randi now admits that he has never seen the tape.
And Sheldrake is not the first psi researcher who has uncovered a pattern of untruths from James Randi.
Given Randi's history of distortions and lies, why would any serious psi researcher want to work with him on any experiments?
15 comments:
Richard Milton's website describes another incident that shows that Randi's $1M challenge is not being offered in good faith.
http://www.alternativescience.com/randi-retreats.htm
In his paper "The Invisible Gaze", Dr Bierman convincingly (in mind mind anyway, allowing for confirmation bias, of course) demonstrates that those staring experiments using Sheldrake's sheet of random pattern does allow for inflated hit rates.
www.parapsy.nl/uploads/w1/Staring_PA2004.pdf
If there are experiments they need to be taken seperately. This is not easy to do reading the Biology Forum papers on Sheldrake's site. I cannot tell which trials had no feedback (only the London trials seem to be explicitly labelled as such) or which used Sheldrake's stare/non-stare patterns.
Excuse me, the first line of my second paragraph should start:
"If there are experiments that do not give the subject feedback they need..."
The blog states:
Like many other scientists, Dr. Sheldrake does write popular books in addition to his admirable record of peer reviewed publication credits, including multiple articles in Nature, Planta, and the Journal of Consciousness Studies. I'm sure that Mark is not intending to imply that Francis Crick, Stephen Hawking and John Wheeler are bogus scientists, because of their popular science works, or that Nature, Planta and JCS are bogus journals, right?
I think it is clear that Calladus is implying that Dr. Sheldrake's only outlet for his poorly executed research into the paranormal is the popular press (and the few journals that specialize in this area). It must be made clear -- being published in a peer reviewed journal is necessary for scientific credibility, but if the work can not be repeated, or is fundamentally flawed, it loses its credibility. Dr. Sheldrake's ideas on the paranormal have clearly been found to be not credible. Hence the skepticism.
It must be made clear -- being published in a peer reviewed journal is necessary for scientific credibility, but if the work can not be repeated, or is fundamentally flawed, it loses its credibility. Dr. Sheldrake's ideas on the paranormal have clearly been found to be not credible. Hence the skepticism.
First of all, Dr. Sheldrake does not use the term "paranormal" because he feels that telepathy, etc. are a normal and natural part of nature. I agree with him about this.
Most of the experiments that Sheldrake has conducted investigating psi phenomena have been repeated by other investigators, so your criticism there is in error. Not sure what you mean by "fundamentally flawed", you need to be specific here. Again, you need to list what specifically you find "not credible" about Sheldrake's research, or else you are just expressing bias and prejudice.
...Dr. Sheldrake does not use the term "paranormal" because he feels that telepathy, etc. are a normal and natural part of nature. I agree with him about this.
When one expresses opinions (about telepathy) without convincing evidence (such as a body of peer reviewed, repeatable scientific work) as Dr. Sheldrake and yourself seem to be doing, this naturally leads to skepticism on my part. (You seem to be the one's exhibiting bias and prejudice).
..Most of the experiments that Sheldrake has conducted investigating psi phenomena have been repeated by other investigators...
But tellingly, not by scientists, and not be by research published in cited peer reviewed journals. The more evidence you supply in support of Dr. Sheldrake's work, the more you move away from the mainstream science -- the exact opposite of what happens in real science, where the repeated results of research become a fundamental part the reigning theories.
Not sure what you mean by "fundamentally flawed", you need to be specific here.
Dr. Sheldrake's work in the field of the supernatural is fundamentally flawed because is does not build upon existing scientific theory and is not supported by past scientific, peer reviewed research. At this point, I have not reviewed the specifics of his work, it is enough of my time to keep up with research in my own field. But in any case, I think the onus is on you to do the legwork in supplying specifics -- you have made the extraordinary claim that Dr. Sheldrake's work in the supernatural is valid -- I believe should present a detailed, well researched summary of the specifics that support such a claim.
Otherwise my own (and I assume Calladus's) skepticism on this subject remains.
When one expresses opinions (about telepathy) without convincing evidence (such as a body of peer reviewed, repeatable scientific work) as Dr. Sheldrake and yourself seem to be doing, this naturally leads to skepticism on my part.
Psi phenomena are not "produceable on demand", because they rely on the infinite variability of human consciousness. So an insistance that every scientific investigator will be able to reproduce them is bound to fail. That makes them not so much different from most other psychology and medical experiments. Nonetheless, most of his experiments have been successfully replicated multiple times by other investigators.
But tellingly, not by scientists,
Incorrect.
and not be by research published in cited peer reviewed journals.
Wrong.
The more evidence you supply in support of Dr. Sheldrake's work, the more you move away from the mainstream science
"Mainstream" science is a sociological construction, not a scientific one. Science is a method of inquiry, not the pronouncements of a church of science.
the exact opposite of what happens in real science, where the repeated results of research become a fundamental part the reigning theories.
You're describing physics, not psychology.
Dr. Sheldrake's work in the field of the supernatural is fundamentally flawed because is does not build upon existing scientific theory
You haven't even read his theoretical works like The Presence of the Past, and so you have no idea how he builds on existing scientific models. And neither Sheldrake nor I have any use for the term "supernatural" which seems to serve no purpose other than to label "things that materialists do not believe in".
But tellingly, not by scientists, and not be by research published in cited peer reviewed journals. The more evidence you supply in support of Dr. Sheldrake's work, the more you move away from the mainstream science -- the exact opposite of what happens in real science, where the repeated results of research become a fundamental part the reigning theories.
I don't know what kind of "real science" you're engaging in or researching, but this is absolutely not true. In The Structure of Scientific Revolutions Thomas Kuhn very clearly outlines the way that "real science" works. Anomalies occur frequently in science, and theories that work very well in some ways invariably fail in others. Many of these anomalies are ignored, particularly if they are outside the scope of mainstream science. When these anomalies are further explored, the problem inherently becomes that the new ideas move away from mainstream science. This is how "real science" progresses. If this didn't occur our scientific understanding of the world wouldn't have progressed--it would simply stagnate with variations on ideas that simply didn't solve all pieces of the puzzle.
Dr. Sheldrake's work in the field of the supernatural is fundamentally flawed because is does not build upon existing scientific theory and is not supported by past scientific, peer reviewed research. At this point, I have not reviewed the specifics of his work, it is enough of my time to keep up with research in my own field. But in any case, I think the onus is on you to do the legwork in supplying specifics -- you have made the extraordinary claim that Dr. Sheldrake's work in the supernatural is valid -- I believe should present a detailed, well researched summary of the specifics that support such a claim.
When you can get research on any kind of paranormal event easily published in a peer reviewed journal next to the work of stodgy researchers who would tear the work apart simply because they feel it degrades their own work just being next to it, you let me know. The point is that scientific work outside of the mainstream is extremely difficult to get constructively peer reviewed. And also, this whole "extraordinary claims require extraordinary evidence" is really getting tiresome. It requires an extraordinary event merely to get mainstream science to take any of this kind of research seriously to begin with. It sounds to me like a way to keep raising the ante so no amount of evidence ever needs to be taken seriously. If we always required extraordinary evidence to validate claims we would never get anything done. That would be like asking my boyfriend to jump in front of a car for me to prove that he loved me. At some point good old fashioned run-of-the-mill evidence has just got to suffice. Maybe Sheldrake has that. But does that even matter if he isn't taken seriously to begin with?
As an aside, talking about confirmation bias, my attention was recently drawn to an mp3 on Sheldrake's site in which he discusses ESP with Chris French. Sheldrake mentions my articles on the ganzfeld
http://www.skepticreport.com/psychicpowers/ganzfeld.htm
but he calls my findings a "worst case scenario", which is not something I recognise in my own work. Indeed, I even say the figure I end up with can go up or down as people include or exclude experiments in a meta-analysis as they see fit.
Despite that, Sheldrake seemed to want this to be the worst result possible for the ganzfeld work, and so read this into my work.
(I have emailed him to point out his mistake)
Hmm, I just listened again and noticed something else. He describes me as someone who "distrust the whole ganzfeld... er, the meta-analysis procedure which involves a kind of quality control, has put together all studies and done his own analysis".
Where did he get the idea that I distrust meta-analyses? Clearly an invention reflecting his own bias. Or is it the quality control bit that I'm apparently mistrustful of? This too what be a falsehood, plus it implies that Sheldrake thinks that Radin's meta-anlysis has some kind of quality criteria attached to it, which it most certainly doesn't.
Clearly he doesn't know much about the ganzfeld except what he's read in Radin's work, and he's filled in the gaps in his knowledge with things that he finds most palatable (ie, I am mistrustful, Radin has followed standard meta-analytical procedure) without checking on their veracity. Tsk.
The debate in question is here:
http://www.sheldrake.org/B&R/audiostream/
the bit I'm referring to is about seventeen minutes in.
James Randi's institute offers a prize if you can demonstrate (what they define as) the paranormal. So far no one seems to have be able to pass his test. It's eerily similar to the test offered by Christian fundamentalists to prove evolution under laboratory conditions. Actually happened, by the way, this has sort of happened with tests showing how proteins in salt water conditions, simulating the early conditions on planet earth, can be generated. By that's beside the point - evolution is a theory based on observable facts, but is not something that itself can be observed in a normal time-frame (nor can certain ideas in geology).
Clearly, the game is rigged - terms are defined in such as a way that you can never win the money. And usually, these contests do not feature scientific experts, or if they do, these experts already have too many preconceived notions that their agenda is usually just to keep a certain worldview intact.
My point in all this is that this is just human nature, because, after all, science, religion, or whatever are human activities and constructs. Myself, I can live in a normal, paranormal, or perinormal universe. No problem.
Andrew, I agree that Dean Radin needs to explain his inclusion criteria for his meta-analyses for Ganzfeld as well as the other psi-related meta-analyses he has done and written about.
I will see what I can do to get him to respond to your criticism here.
I've already emailed Radin and he explained that the EM meta-analysis was an update of the UC work, with more recent work added. Since he describes the work included in UC we can say with confidence that there's no inclusion criteria.
Meanwhile, Sheldrake replied to my email and explained that:
When I said that you’re analysis represented a worst-case scenario, what I meant was you included all data that you could find irrespective of reservations that some researchers had on the quality of the experiments. I agree this phrase does not express that very clearly and I’ll avoid using it in future when referring to your work.
So it's nice that he's understood my reservations about the phrase.
rboerner3: Randi tested breatharians twice. The first time he basically sat outside a motel and caught the claimant sneaking off to a neighboring fast food place. The second time was on a tv show with some lady determined to starve herself. Doctors monitored various things like fluids and weight and decided to stop the test after a few days since it was clear she was just starving herself.
He just decided the time and effort (not to mention the danger to the claimant) to test that sort of thing wasn't worth it. I'm sure he would make an exception if the claimant was getting media coverage, winning world records, getting vouched by some reputable researchers, etc.
You have a pretty gross ignorance of statistics.
Post a Comment