Skeptical Reefkeeping 10 – The Power of Anecdote

From ReefsMagazine

by Richard Ross

In the last nine installments of Skeptical Reefkeeping we have looked at varied topics from phosphate to marketing to fallacious lines of reasoning to communication. One of the through lines all along has been the idea of anecdote, and generally, why it isn’t to be trusted. In this installment of Skeptical Reefkeeping, we are going to take another look at anecdote, try to understand why we are dependent upon anecdote in our hobby, and discuss some of its power and how to make it more useful. 

A Brief Reminder to Set the Scene

Skepticism is a method, not a position. It can be defined as a method of intellectual caution and suspended judgment. As a Skeptical Reefkeeper, you decide what is best for you, your animals, and your wallet based upon critical thinking, not just because you heard someone else say it. The goal of this series of articles is not to provide you with reef recipes or to tell you which ideas are flat out wrong or which products really do what they say they do or which claims or which expert to believe. The goal is to help you make those kinds of determinations for yourself while developing your saltwater expertise in the face of sometimes overwhelming, conflicting advice.

These two Dr. Seuss fish have not yet jumped out of their tank, but that doesn’t seem like a reason to jump to the conclusion that these fish aren’t jumpers.

What is Anecdote Anyway?

From Skeptical Reefkeeping – Are you sure that that thing is true, or did someone just tell it to you? (1) Merriam-Webster defines anecdote as “a usually short narrative of an interesting, amusing, or biographical incident.” More hardcore, Ron Shimek says, “Anecdote is unsubstantiated or unverified observation generally made by an unqualified observer who often really doesn’t know what they are looking at.” Essentially, an anecdote is someone telling you what they think happened. The problem with most anecdotes, besides the observation and conclusion being suspect, is how quickly, with no real support, they can be converted to facts. This conversion can have a real and detrimental cost in both animal’s lives and your money.

Better is in the eye of the beholder. People think corals are better under different lights, but how can this kind of subjective assessment help you make decisions about your reef? Photos by Rich Ross.

The second problem is that of confirmation bias. Is the person reporting the anecdote seeing what they want to see? As it turns out, when someone makes a decision about a change to their reefkeeping methodology, they often really want to be right. When convincing themselves to spend money on something new, they really, really want to be right. In a subconscious effort to be right, they can begin to see only results that confirm their rightness – the new salt made my corals look better.

There are a host of unanswered questions that come with anecdote that make it empty of useful content. Do the results go away if the reefkeeper changes back to the original salt? What else was done besides the salt change? Do other people have the same results? Do other people have the opposite results? There is a good reason that these questions are unanswered – to answer them we need to move away from anecdote and into the realm of science.

Why Anecdote Doesn’t Suck

Anecdote can be considered casual evidence, and plays an important role in science because anecdote is really where science often starts. “An apple fell off a tree, hit me in the head and got me thinking about gravity…” (3) “I noticed that when sick people cough on healthy people, they get sick too…” “Hey, all these finches on this isolated island have different shaped beaks, maybe they all came from a common ancestor…” All of these observations led to hypotheses, which led to experiments, which led to results, which led to repeatable results, which led to the development of accepted theories. So, while anecdote informs the hypothesis, the anecdote is not the science in itself. The important thing to notice is that science doesn’t stop with anecdote because anecdote is filled with all kinds of untrustables. The whole scientific method can be thought of as developing ways to support or dismiss anecdote to the point where we think something is true or not. 

VIDEO EMBED: https://www.youtube.com/watch?v=R56gU1-Nmgg
Caption: Many people use the word theory when they really mean hypothesis. 

We are Dependent on Anecdote in our Hobby

Truth be told, real and thorough science takes time, resources and money – all things that are generally in short supply. I can hear you thinking “But Rich, surely the companies that produce additives and products do thorough testing before they bring a product to market?” I have some terrible news for you – there is no Santa Claus. As discussed in Skeptical Reefkeeping 2 – Magic In A Bottle (4), some companies test and some don’t, some do it well and some do it poorly, and some are simply trying to scam you into buying an untested and ineffective product. And while I have previously imagined a reef aquarium testing organization, “Reefbusters” if you will, sadly no such organization currently exists in our hobby (though if you have a pile of money laying around, and are interested in funding such an organization, please let me know). We are generally left waiting for scientists and public aquariums to do the testing, which makes us reliant mainly on hobbyist anecdote for information regarding the effectiveness of new products and methodologies. 

Practically, the hobby is kind of a distributed testing system. Some new idea is presented and there always seem to be some people willing to give it a try, which can lead to a few more people being willing to try it. Most people however, sit back and wait to see what happens, letting other people risk their reefs on new and unsupported ideas or products. In this way, sometimes great advances are made, like the rise of the propeller pump. Sometimes, old monsters are resurrected like using ginger to treat marine ich (5). Most of the time no ground is gained, and we end up with little or no more information than we had before; there is no best salt mix, and the best advice I have for you on which one to use is to use one that is always available in your area. While we have been getting along with this system for decades, there are several real deficiencies with this process. It is slow and inefficient. It is difficult to find results. The “testing” done is often not trustable due to anecdotal messiness. The same bad ideas get tried over and over again, and animals are unnecessarily lost as a result. It seems to me that reef aquarists are some of the most dedicated, inventive and motivated people around, and that with a little planning and effort, our anecdote can be more powerful and more useful. 

How to Make Anecdote Better?

The simplest way to make anecdote better is to never forget that it is anecdote. Don’t let it be converted to fact. Always mention that whatever you are reporting was your experience, that you aren’t really sure what caused the effect you are seeing, and that you think it might have something to do with whatever you think it might have had something to do with. This may sound like a pedantic point, but words are specific and words trigger different understanding in our brains, especially for people looking for quick answers to complex questions (and given the nature of keeping boxes of saltwater animals in our living rooms, almost all reefkeeping questions are complex). There is a world difference, especially to a new hobbyist, between saying “Rich’s Reef Juice makes your corals look better” and “It could be that my corals looked better after adding Rich’s Reef Juice, and the Juice might be the reason why.” 

Try to change only one thing at a time when trying something new. If you are trying a new additive, don’t make any other changes to your tank because those changes will preclude meaningful interpretation of any results of the additive you are adding. If you add an additive and then do a water change, feed some new food, and change the flow in your tank, you can’t possibly know which of those changes, if any, made the difference.

Remove confirmation bias by showing someone else your tank before and after you do the change you are trying to test. If you are adding a product to color up your corals, you are the worst person to determine if a change occurred because you are involved and because you care. You want there to be an effect. This is human nature. A person not directly involved has no dog in the fight and is more likely to give an accurate assessment of the situation. 

If you want your experience to help all other reef keepers, you need to keep records so you know what happened and when.

When you are doing any kind of testing, be methodical and write it down. Adam Savage of Mythbusters said, “Remember kids the only difference between science and screwing around is writing it down.” Keeping a simple log of what you do to your tank can really help support and flush out any ideas you have about why something happened in your reef. 

Simple Science

The best way to make anecdote better is to use it as a starting place for more testing. Reefkeepers are some of the most tenacious and talented people around, and it always confuses me that we will put so much effort into arguing about what a product or methodology might do, and so little into running some more useful tests to generate actual information. I am not demanding rigorous testing, as rigorous testing involves multiple trials, double blind procedures, several controls where nothing is changed, and as large a sample size as possible. While rigorous testing is great, it is impractical because of time and money constraints. But simple testing is well within the reach of most hobbyists.

Doing robust science takes time, money and dedication, but even simple science can yield results that will be useful to other hobbyists. http://www.workboxsystem.com

A simple, scaled down test won’t produce the same kind of robust results a rigorous trial can, but it will produce results that are more useful than simple anecdote. A simple side-by-side test can tell us something about a product, and more importantly, it can point us in useful, productive directions for future inquiry. Additives that are supposed to quickly cycle a tank (build up biological filtration bacteria in days rather than weeks) are a perfect example of where a side-by-side test could tell us something useful: Two 5 gallon tanks set up the same way, the same amount of live rock, sand, the same flow, with the only difference being the addition of the cycle product to one. Test for the relevant parameters, ammonia, nitrite and nitrate over the course of a month and then voila, we would have some actual information that could inform our understanding about what the quick cycle product does or doesn’t do. Sure there are all kinds of variables, but we aren’t looking for rock-soild answers, we are looking for more compelling evidence than simple anecdote. Feeling more ambitious? Set up more than two tanks. How about four with everything the same but two have live rock and two don’t, and one of each gets the product? 

Red Bugs – A Better Than Anecdote Success Story

Red bugs became a real problem in the reefkeeping scene around 2002, and they were a major pain in the butt. They crawl all over your Acropora and can irritate them and stress corals to the point of possibly killing them (6). These Tegastes copepods were a real problem (and still are), and at the time, there was no in-tank treatment. Dips in iodine, betadine, and various other medications were all supported by anecdote only, had mixed results at best, and killed the corals that we were trying to rid of the pests at worst. 
Red Acro bugs are now just an annoyance, but before someone took the time to test out a solution they were a terrifying menace. Photo by Sanjay Joshi.

Dustin Dortin found an in-tank treatment that he thought worked for the eradication of Red Bugs, but he didn’t stop there. He took an extra step. He didn’t just announce that he found a dog heartworm medication and that after he used it his corals looked better. Instead, he really wanted to show that it worked. So what did he do? He took some anecdote and built on it, just like “real” science. 

In Feb of 2004 Dustin posted on Reefs.org (7) that he had found what he thought to be a simple, safe treatment for Red Bugs, that he had done several tests in isolated systems, that the first test in a full blown reef tank was underway, and that he was looking for more people to do more testing in a systematic way. It took 2 to 3 months and 7 different reef systems to finish testing before releasing the name of the medication and the protocol for treating with it. 

“As you can see this medication has gone through a rather small amount of testing and not much time has gone by. All of the results have been positive so far. However, only time will tell if the red bugs will return to treated systems. Time will also tell if there is some sort of sort of side effect that will pop up a year or more from now. I need to make this point clear.If you use this medication it is at your own risk. No one but yourself is responsible for your actions with this medication.”

Holy moly! Even though he did a whole bunch of testing, Dustin still thinks you should be careful and skeptical because he knows what he did was useful, simple science, not rigorous testing. In the thread, he gives a lot of detail regarding what the medication is, where to get it, how to dose it, how many times to dose it, and what to do after treatment. Others have now built on his work, refining dosages, treatment time, and after-treatment procedures. Sure, it took some effort and some time, but now the entire reefkeeping community has a practical treatment for Red Bugs – a treatment that has something more than “I think it works” to support its use.

Your Work is Helpful and Important

Most science today is intimidating because it is so complicated, and there is very little room left for bench-top science. Lucky for us, our hobby is full of room for bench-top science. So, have fun. Use anecdote for what it is, a starting point, and do some testing. When you are done, write it up in one of the many reef magazines or blogs so people can build on what you have done. Your tank, and the hobby in general, will be better for it. 

References

1) Skeptical Reefkeeping 1 http://www.reefsmagazine.com/forum/r…hard-ross.html
2) Tankmates: It works until it doesn’t, by David Perry –http://www.tonmo.com/community/pages/tankmates/
3) Seems like the story of Newton and the Apple is almost true – check outhttp://www.newscientist.com/blogs/cu…eal-story.html
4) Skeptical Reefkeeping 2 – http://www.reefsmagazine.com/forum/r…hard-ross.html
5) The ginger/ich thread that was the basis of skeptical reefkeeping 1 is still going on, 10 years after it first appeared. 10 years later, and the same fallacies that are discussed at the beginning of the thread are being recycled. It has been bizarre to see Skeptical Reefkeeping 1 posted to the threat that Skeptical Reefkeeping 1 is about. http://www.reefcentral.com/forums/sh…181818&page=27
6) The Reef Aquarium Vol 3, by Julian Sprung and Charles Delbeek, page 648,http://www.amazon.com/The-Reef-Aquar…/dp/1883693144
7)The “CURE” for Red Acro Bugs – http://www.reefs.org/forums/topic45859.html