The Nirmukta Freethought Community

While engaging with self-identified professional astrologers or astrology aficionados, it maybe a good idea to establish mutual consent for ground rules like the following. This list is suggestive and not exhaustive, and therefore, please modify and add to this list. It will be great if the terms and conditions of better-known challenges like the Koovoor Challenge are circulated more widely so that it is a handy resource for freethinkers.

Refusal on part of the defending side to agree with these terms and conditions is tantamount to an acknowledgment of indefensibility of their claims, and obviates any obligation on part of the skeptics to volunteer for formal testing.

1) Sometimes a cigar is just a cigar...And a spade is a spade.

Initially stipulated definitions of terms must be adhered to before and after the prediction. Therefore it goes without saying that one single textual statement unalterable until the conclusion of the test will be treated as the prediction, and no other.
In typical drawing-room discussions on astrology, if a gentleman hailing from Raipur has two children and has built a career in pharmaceutical economics in Mumbai when the family astrologer had predicted that he would be a physician abroad with three kids, then supporters of the astrologer often explain away the contradiction by saying, "Well, a chemist is almost a physician, two is almost three, and for Raipur, Mumbai is 'abroad'"!

If a serious evaluation is to be performed, physician means physician, three means three and abroad means beyond national borders.

2) The goal-posts stay where they are.

Levels of significance in the planned tests and more importantly, the level of chance performance should be stipulated beforehand and cannot be changed following the measurements.
An excuse from the defending side like, "Well, it at least worked 19.45 % of the time and the success rate would undoubtedly have been higher if the other side had co-operated with fuller faith!", is inadmissible.

3) Level the playing field.

The tests performed by the astrological expert must be repeatable by a neutral investigator provided with clearly intelligible instructions of the methods. If no such intelligible statement of the methods is available and resorts to 'intuition' are claimed, then the claims will be declared untestable. The instructions must clearly state the assumptions as well as inference rules employed.
As in engineering practice, it is preferable to have independent teams for drafting the test plan, for conducting the experiment and for administering the test to the results obtained.

4) Plug leaks.

The data available to the claimant as well as the neutral investigators repeating the experiment must be the same in all respects. Leakages of additional biographical information via sabotage or 'cold reading' if discovered shall be considered grounds for declaring the tests null and void.

5) Decide the stakes beforehand.

The results of the test being performed will be interpreted only as an evaluation of the methods being tested, and the result favorable or otherwise will be inadmissible as proof of either side's infallibility or even expertise in tests of other claims or methods. In other words, it is the claims and not the claimants that are being put to the test while in practice and in effect, reputations may well be at stake. This follows from the principle of separating people from ideas.

6) Nightingales sing during daytime as well.

Studies testing predictions for illness must include astrological specimens of illness-free participants besides the patients. Studies explaining road-traffic accidents must present comparisons with successful accident-free trips. The study must be performed with the investigator unaware of which of the groups a specimen comes from. Examining a set of patient horoscopes and ascribing their illness to astrological conditions which might just as well hold for healthy patients, will not do, and hence the necessity of the control group. Nightingales earned their name though they sing often enough in daylight, largely because their songs are easier to hear in the night. This misconception could have been avoided if only daytime control observations had been made besides the night-time ones!

7) Closure must involve full disclosure.

The study will be declared complete in a formal, joint statement by all parties concerned, prior to which a mutually approved and verified statement of the study results must be committed to writing, and after which neither party will be under obligation to service complaints from the other about the methodology or interpretation of the study in question. The joint statement which can take the form of a press release, overrides any other announcement which the parties may make in an individual capacity, and is intended to forestall premature or spurious claims of victory by any party.

8) Customer maybe king, but all experimental subjects must be given equal treatment.

The querying which the different human subjects undergo and the way the treatments their case studies are subjected to in terms of analyses, must be identical for all subjects in the intervention group. Claimants of astrological prowess may not present in the same study, cases of their clients some of whom have undergone 'Standard Analyses' and others have undergone 'Customized/Special Packages' because in the latter case, there might have been additional data acquisition and more meticulousness in application of methods. If there is reasonable doubt that the treatments received by all subjects in the intervention group may not be identical, then the study will be suspended until the cases not conforming to the treatment stipulated in the experimental design are excluded.

9) Public disclosure must adhere to a privacy policy.

Public disclosure of the study results will include information on particular predictor and outcome variables defined in the experimental design and will not include any biographical information of the subjects extraneous to the study which may have been made available to the astrologers. Identities of the subjects shall be protected by default, allowing subjects who are available to be contacted the option of authorizing the investigators to release other details.

Exhaustive as the list is, it's highly unlikely to find any astrologer who will agree to even some of those terms and conditions, let alone all.

I'm sure several astrologers understand the limitations of their falsely perceived 'predictive powers'. They count on the gullible many who fall for simple tricks like weasel wording their talk, etc. , since that is the group that is most likely to visit astrologers. It is only extremely occasionally that they have to deal with curious sceptics(like me).

My first (and perhaps last) encounter with an astrologer was when I accompanied a friend merely as an experiment, since this friend claims to be 'quite sceptical' normally but only trusts this ONE astrologer since he's VERY accurate.

My trip was a dissapointment. Given the enthusiasm my friend built up, I set my expectations way higher and was all prepared to be clever to catch him play on my cognitive biases, but this guy did not even challenge me. It was a dull encounter and he just made some simple unsubstantiated predictions. This experience was similar to the experience outlined in this article . People build an aura of authenticity and superiority around certain particular people while even themselves professing disbelief in other astrologers and then it turns out to be quite normal actually verified. Their use of cheap tricks(such as creating random a sense of hurry whenever required, using red herrings, and also positioning themselves in a socially dominant position such that no questions may be raised in response to what they try to pass of as genuine behaviour) is only too obvious.

The 'ground rules' listed above serve at least one useful purpose, namely, that they lay bare the incompatible epistemic standards of Science and pseudosciences like astrology. The purpose of this exercise will be partly served if at least a few fence-sitters refuse to say by force of habit that astrology is 'an ancient science', when they recognize how it cannot submit itself to scientific scrutiny. A Yale University professor offers lighter take of how no astrological prediction really commits itself to a reality check, in this video clip.

Excellent thread! Thanks to arvindiyer for the great opening post. I describe below, a study that incorporated to the extent possible, the requirements laid out above. I searched the nirmukta.net threads for 'narlikar' and this paper did not turn up, but I hope I am not repeating any earlier discussions, given that the paper discussed below was published in 2009.

Jayant Narlikar published a paper in 2009 in Current Science (journal of the Indian Academy of Science). The paper is very simple to read, and it describes the results of a double blind test of astrology. The summary is as follows:

1. The data consisted of information about the birth details of 200 children. 100 of the kids were 'intellectually bright' and 100 were 'mentally handicapped' (using terminology from the paper.)

2. Each astrologer was given a set of 40 randomly drawn 'birth charts'. (some math, skip if uncomfortable, though IMO it is very important for all freethinkers to know the basics of clinical trials) Their hypothesis test required that the success rate of prediction exceed 2.32σ from the expected value. With a mean of 40 and σ = 3.16 (binomial distribution), success meant that at least 28 be predicted correctly.

3. The astrologers' response is summed up as: "When this framework was announced, the response of the astrologers was varied. Some agreed to take up the challenge, others asked for additional conditions which had no relevance to the nature of the test being conducted, while some called upon the astrological community to boycott the test."

4. Finally, 51 astrologers agreed to participate, and as planned, each astrologer was given 40 randomly chosen birth charts for prediction.

5. 27 of the astrologers replied. The best prediction was 24/40, which was still below the threshold of 28. The average success was 17.25/40, which is even less than what one would get from a random association of birth chart with the two classes.

6. As for institutional (whence I learned there are institutions of astrologers) participation, they were given all 200 records, and the only institution to respond had a hit rate of 102/200 which is still less than the (µ+2.32σ) of 117.

7. To state the obvious, astrology does not stand up to the rigors of statistical hypothesis testing. At least now, no one can make the hollow claim that there is no science to disprove its effectiveness.

Using simple high school physics, one can debunk any claims about stars and planets physically affecting people. But to the incorrigible people who say, 'Hey there may be forces hitherto unknown blahblahblahcrap,' Narlikar's paper is a pretty rigorous riposte. Nutters say, 'It works for many people we know, science cannot explain how.' We say 'Wait a minute! No it doesn't. Screw your anecdotes.'

Narlikar refers to papers by B. Silverman (J. Psychol., 1971, 77, 141–149 and J. Psychol. 1974, 87, 89–95), which did similar studies in the US, forty years ago, presumably around the time when there was a hippie-driven New Age craze for such woo.