From Reefs Magazine
By Richard Ross
In reefkeeping, there are a million products and techniques, each claiming to be a necessary ingredient for a successful reef tank. The problem is that many of these claims have little, if any, evidence to support them – so how are we supposed to know which ingredient, product or method is useful and which is bunk? Well, we can bellyache that someone else should figure it out and let us know, or we can get up off our collective butts and start producing evidence ourselves by doing some simple experiments.
What Happens in our Hobby
Here is what happens all too often with new ideas and products in reefkeeping: Someone makes a claim about a product or methodology with little or no evidence to support the claim. People argue about the claim with various theories about the claim working or not working. Some people put the claim into practice. Some report incredible success, others report gut wrenching disaster, all based on anecdote (my corals look better, the ich went away, my fish seem happy). Most report no change at all, but they are ignored while the anecdote is embraced, and people argue the merits with startling emotion. After a year or two the idea fades into background noise, only to pop up again in 5 or 10 years when someone runs across it in a old online discussion, and the process grinds back to life with little new understanding reached.
Human-behavior expert Keith Johnstone calls this kind of discussion with lack of evidence the problem of the “red car in the parking lot.” You can argue all you want about if there is or isn’t a red car in the parking lot, but all someone really needs to do is go look and see. Why don’t we look and see? In our case, why don’t we just do some tests? Because we think testing is hard and needs to be perfect, but for the bulk of our pressing reefkeeping needs, I don’t think this is the case.
We are Scientific Creatures
We start testing things almost the instant we are born. Toddlers stick everything in their mouths to see what they can eat. If it tastes really bad, they learn not to stick it in their mouths, but if it tastes good, they stick more of it in their mouths. Though this isn’t academic or planned out, it is basic scientific testing. As they get older, children do start to perform more deliberate tests. We have all seen kids sigh and make a sad face when they are told no about something. Why do they do this? Because often, after they are told no, a sigh and a sad face gets that no converted to a yes. This action is based on the evidence that they have compiled through performing experiments, even if the child doesn’t know they are running experiments. Building structures with blocks is a great scientific experiment, and through experimentation with different blocks in different orientations, children quickly learn about basic structural engineering. Throwing a ball teaches physics. By participating in team sports, they learn what helps and doesn’t help in social situations. We don’t tend to call these things scientific experiments. We call them play. But, the act of play is learning by trying things out, and this is what an experiment is at its core.
The problem with these kinds of experiments is that they are anecdotal. Being the experience of a single person clouds them and often leads to incorrect conclusions. For instance, a child may incorrectly conclude that because whining gets them what they want from their parents, whining will get them what they want from all adults. (It doesn’t.) Thus anecdote, while it has its place, also sucks because it gives us unsupported, unreliable evidence.
The point of this section is that we all have been doing science all our lives, and there is no reason we should ever think we couldn’t continue to do science. The more we understand about the science we’re trying to accomplish, the more useful our results can be. Science need not be overly rigorous and complicated to be useful and practical. If we do some basic experiments, we can get reliable evidence, and evidence is always better than anecdote.
Simple Experiments
In our hobby, often all we really want to know is if there is any basis to believe a particular claim. One of the most common claims discussed in our hobby is that a product or methodology removes phosphate or nitrate from our reefs. Though it seems like using the product on a reef tank and reporting the results is a good idea, this is really just anecdote and is fraught with all kinds of problems: our tanks are uncontrolled soups of chaos with all kind of processes going on, changing over time, and many other ways to lower nitrate and phosphate have been put into play in the recent past and sometimes they take time to have an effect. Attributing success to the new product is really just post hoc ergo proptor hoc (it happened after, therefore was caused by) reasoning – see Skeptical Reefkeeping 1 for a more detailed discussion about this and other fallacies. Anecdote is fine as a jumping off place for further experiments, but it is not a good place to look to determine if a claim or method really does what it is supposed to do.
It should be easy to show, without relying on anecdote, that “Yaiullo’s Magic Juice” (don’t forget that it doesn’t actually exist) removes phosphate or nitrate. Get a sample of saltwater. Test the nitrate and phosphate. Add a known amount of phosphate or nitrate (or heck, even just add some food). Test the nitrate and phosphate again to be sure the levels have risen. Add the product or apply the method. Test the nitrate and phosphate again. Voilà, you will have some actual evidence of the veracity of the claims.
Of course, this super simple way to test has its own problems, because it takes place in such a minimal situation, but by removing as many variables as possible, and by repeating the test several times – we start to generate some real, useful evidence about the efficacy of the product or method. If you share the way you did your experiment, others can try it to, and you get an even bigger pool of evidence to draw from, or perhaps even constructive criticism of your methodology. I would be much more likely to use a product with that kind of evidence over someone saying “I added the product to my 6 month old tank and the corals look better!”
We can make the experiment more robust for the real world pretty easily: Instead of a water sample, set up a small tank like a reef. Test the nitrate and phosphate. Add a known amount of phosphate and nitrate. Test the nitrate and phosphate. Add the product or apply the method. Test the nitrate and phosphate. Already, this is different from the water sample test, because it’s being done in an active reef tank, which we know is a dynamic, changing system. Maybe we’ll find out things we didn’t expect. Perhaps the nitrate and phosphate levels only go down temporarily, or maybe they only go down after a few days or weeks. We also need to consider if the results seen in one tank are due to something unique in that particular system. Perhaps the live rock or the sand is doing the work we are attributing to the method or product we are testing.
To rule this out, we make the test slightly more comprehensive: We set up two tanks as identically as possible. One tank is the experimental tank, the tank in which we implement the product or method as described above. The other tank is the control, the one to which we do not add the product or method. Having a control helps us determine if it is the product or method being added that gives us our result or if we get the same result, without the product or method we are testing. If we get the same result in both the control and the experiment, we may be able to conclude that the method or claim being tested doesn’t have a significant effect.
The experiment can be made even more complicated with more tanks and replications to rule out other testing issues. The only thing limiting you is time and resources, but the take home message is that even such a simple test will give us some actual evidence about the claims being made.
Real World Simple Testing – The Case of Inland Reef and the Eco Aqualizer
In the early 2000s, a product called the Eco Aqualizer hit the market and was supposed to have a plethora of beneficial effects for both fresh and saltwater aquaria. The product claims struck many as over the top. The explanation of how the device was supposed to work left many feeling that the product couldn’t possibly do anything, while others swore that their tanks benefited from the device. Even though a lot of the claims were hard to pin down, and there was much “red car in the parking lot” discussion, at least one claim was clear: The manufacturer said that the device would “improve system redox.” (http://www.saltcorner.com/Reviews/showreview.php?reviewID=19).
Inland Reef Aquaria, sadly no longer in business, decided to test this claim with a very simple experiment.
Since the product was supposed to affect water run through it, they set up a single tank with a closed loop and a bypass so that the water could be passed through the device sometimes and not other times. Then they put an ORP probe and Redox Meter on the system, and ran the tank water through the device at different intervals over two weeks and tracked the results. They found no difference in ORP between using the product and not using the product and had a ton of data to support their conclusion: The product did not “improve system redox” as claimed.
Write It Up
Inland Reef Aquaria didn’t just do the simple experiments. They wrote up what they did and put it all on the web for everyone to see. This is the critical part – like Adam Savage says “Remember kids, the only difference between Science and screwing around is writing it down.” You can still see everything that Inland Reef Aquaria did because the Internet is a wonderful thing. Check out this cached version of their website.
Writing up your results and methods can be intimidating, but it doesn’t have to be. The best advice I ever got about this kind of thing was from the Senior Director of the Steinhart Aquarium, Bart Shepherd: “Just write down what you did.” It really is almost that easy. I mean, make it cogent by breaking it down into sections that make sense like “The problem”, “The claim”, “My methods”, “My results”, and “What I think it means”. Write it with good grammar and stuff (ha!) so it is easy for people to read, understand, and copy. It is actually really fun to put something like this together, and when you are done, publish it somewhere – like your own website, your club’s website, or even in Reefs Magazine perhaps? This will all help everyone that comes after you in the hobby.
No Matter What You Do, People will be Angry
No matter what kind of experiment you do, someone will find fault with it. This is the nature of humanity – people don’t like to have their beliefs questioned. The makers of a claim or product that you test may be especially unhappy if your results show something different than what they have asserted and “red car in the parking lot” discussions tend to create polarized “fanboys” who will fight against anything that makes their side seem wrong.
You might get angry when people question your test or your results. If that happens, fight off that anger and be intellectually honest with yourself because it is possible that others will point out real problems with what you did, and if they do, it is a good thing! The point is to get good and usable evidence, not to defend our own work.
How was the Inland Reef Aquaria test received? Most people were happy to have some evidence, even if they thought the experiment was super simple. Others were not happy at all and pointed out all the flaws in the experiment. The most interesting part of this is, to the best of my knowledge, no one has done any experiment to show how the test was flawed or, even more important, to show that the product actually does what it is claimed to do! The little support that remains for this product, though I don’t believe it is still in production, is still all anecdote and “red car in the parking lot” discussions.
Don’t let the possibility of people not liking your work scare you from doing the experiment. The experiment is about making the hobby better. The work is important to move the hobby forward more efficiently. How many products or methods have you tried that didn’t pan out? How much time and money and coral could you have saved had someone done the simple experiment before you tried the product or method?
Side Note
I should point out that it really does seem like it is the responsibility of the people making the claims to support their claims. Actually, it doesn’t seem like that, it is like that. If someone makes a claim, it is up to them to support that claim, because, well, it is their claim! As the late, great, Christopher Hitchens said, “That which can be asserted without evidence, can be dismissed without evidence.” At the same time, I understand why reef companies might not have comprehensive testing for their products. Thorough testing is hard, and often the margins reef product companies run under are tight. This can result in a rush to market leaving little time for thorough testing. Even worse, as we have discussed in previous installments, we as a community simply don’t demand such testing (though we really should).
In Conclusion
Performing simple experiments allows other people to build on your work. Any simple testing you do makes it possible for people to do more complex testing more easily, and that kind of rising tide does float all boats. If you are met with people not liking something about the experiment you have done, encourage them to do their own experiments instead of just bellyaching.
So, we should all be running experiments to make the reefkeeping landscape more solid. Does this product/claim work to get rid of hair algae? Do corals color up better if you do X? Do corals grow faster? Do fish like one food more than others? Do bananas cure ich? The sky is the limit.
Design a simple test. Come up with some simple metrics to help make it more compelling. Write it up. Get your local LFS or club to help you do it. Do the best you can, but don’t get mired down in trying to do it perfectly. Your results will help inform future tests and the need for future tests. If we all get off our butts and do some work, things will get better faster.
What is Skeptical Reefkeeping about?
Skepticism is a method, not a position. It can be defined as a method of intellectual caution and suspended judgment. As a Skeptical Reefkeeper, you decide what is best for you, your animals, and your wallet, based upon critical thinking, not just because you heard someone else say it. The goal of this series of articles is not to provide you with reef recipes or to tell you which ideas are flat out wrong or which products really do what they say they do or which claims are trustable or which expert to believe. The goal is to help you make those kinds of determinations for yourself, while developing your saltwater expertise in the face of sometimes overwhelming or conflicting advice.
Special thanks to Chris Maupin and Jim Welsh and Scott Meltzer for their help with this article.
