Rich Ross

No matter where you go, there you are

Epic Fail – the Anatomy of a Disaster

It’s every reefkeepers worst nightmare: opening the front door to the house and smelling the pungent smell of the shore that the Yucatecans call ‘lodo’. While pleasant near the ocean, that smell in your house means something has probably gone wrong with your reef. As you rush through the house to the tank you hope you won’t find the milky mess of death that your nose is telling you you will find. Sometimes you are lucky, and the smell is a container of frozen shrimp or macro algae that you left on top of the tank to fester easy-bake-oven-style under your metal halide lamps. Sometimes it is worse – much worse. Last December, I came home to that smell, and it wasn’t light baked shrimp or algae; it was the much worse.

BEF (Before Epic Fail)
First a little backgroundI have a 150 gallon show tank in my living room, mixed reef, but primarily SPS. I love it, and it contains several personally collected pieces. It was a featured aquarium in Advanced Aquarist in 2004. In late 2005 I took it down for a month while we had some work done on the house giving me the opportunity to make it into a in-wall tank with a mini fish room (really just storage) behind it.

The 200 gallons of sumps/coral farms are under the house in a 40 inch crawl space, that, while annoying to move around in, keeps the noise, mess and humidity out of the house. Also, because the crawlspace is cool in the summer, my need for a chiller is very much lessened. The crawl space also gives me room to store all my reef ‘junk’ and allows me to keep 150 gallons of mixed saltwater on hand at all times, which always seemed like a good idea to me.

The rigged out uber-redundant crawl space

In 2006 I got nailed by monti eating nudies, which meant pulling all the giant monties and either dipping them or throwing them away. In 2007 I got the brunt of the AEFW plague when dealing with the wee beasts meant pulling all your corals and dipping them or throwing them away (now its easy to live with them). All this rigmarole meant that I was constantly wishing the tank was 6 months more along so that it could make another appearance as a featured aquarium in Advanced Aquarist. Last November, I started talking about how the tank was finally maturing, how I was removing smaller colonies to get rid of that fruit stand look that tanks go through when stocked with frags and mini colonies. A video Jake Adams took of my tank was posted on youtube. I started fragging to shape the colonies and to stop them from growing together. Then, December came and along with it, the Epic Fail.

A little more background

I love basic automation and redundancy. I don’t run a controller because I don’t like the idea of a single point of failure. There are 4 circuits going to my reef system with lights and pumps and heaters distributed over the different circuits. There are Penn Plax air pumps that come on in a power outage for oxygen and circulation. There is one Vortech with a battery backup. There is a modded maxi jet on a UPS. DI water collects in a 10 gallon reservoir to limit the amount available for possible overdosing, and then is pumped into the Kalk reactor by a pump triggered by a float switch (not valve), and the float switch has a second float switch just above it in case the first float switch fails. The effluent from the Kalk reactor gravity feeds into the sump. The skimmer’s external collection bucket has a float switch on it that controls the skimmer so if the skimmer goes nuts it can only pump out 5 gallons from the tank before shutting off. I also have a pump in that external collection bucket so I don’t have to lug around nasty water – I flip a switch and away it goes.

All in all, I feel my system is pretty sound and able to handle almost anything that isn’t a major disaster or power failure (and yes I have a generator and a power inverter). We get minor power outages sometimes (as I write this, there was one last week) and thus far everything works just fine. Usually the only way I know there was an outage is by the blinking clocks – the tank just chugs along.

It has been important that my system takes care of itself for the most part because my wife’s major hobby is exotic vacations, which means we can be away from the tank for 3 weeks or more at a time. I used to fret about being away for so long, even though I had done pretty much everything I could to automate daily tasks, and had a posse of reefing friends checking in to make sure all was well. I still fretted until my wife gave me the best piece of reekeeping advice ever – ‘assume that you are going to come home to coral soup’. So, every time I leave the tank I make sure my corals are backed up in other people’s tanks, I say good bye to everything, and I go enjoy my vacation and don’t think about the tank. I believe that this advice had the effect of prepping me for coming home to coral soup at any time, and when I finally did, I was able to function and try to control the damage.

Every reefers nightmare


The horror, the horror

I got home Saturday evening, and smelled the smell of the sea. I rushed to my reef but couldn’t see very far into the tank. Some of the fish I could see were being blown around but not moving on their own. Going through my head were lists of dead animals…the double headed Scoli I got at Midwest Frag Fest…the S. wilsoni I got at MACNA…the Picasso Clowns…the Radiant wrasse…colonies that were finally ‘big’. For a minute I stood there frozen in grief. I put all that aside when I saw my pH monitor – 10.5. I knew regardless of what had caused the crash, if I was going to save anything I needed to get that pH down immediately.

I took 150 gallons of water out of the system, put 150 gallons of new water right in, and started filling the container with RO so I could mix more saltwater. That 150 gallon container I keep filled with mixed saltwater had been helpful before, but at that moment it was critical. Sadly, the pH barely moved. I remembered that vinegar would bring down pH. I had a little vinegar in the house, and it brought the pH down, but not enough, so I rushed out to the store to by more. I believe all in all I added a full gallon of vinegar to the system before the pH dropped to 8.6, and some of the fish looked less dying than they had before. I then added new carbon, a nu clear canister filter with a pleated micron cartridge and diatomaceous earth (DE) and ozone. I finally went to bed fully expecting everything to be dead in the morning, and decided to take no rash action, removing nothing from the tank for at least few days to give everything the best possible chance of recovery.

1 day AEF (after Epic Fail)

I woke up, and still couldn’t tell what was going on in the tank because it was still cloudy, but a little less stinky. All that vinegar was probably causing a massive bacterial bloom. Once another 150 gallons of water was ready I did another water change, changed the carbon, rinsed the pleated cartridge and put in fresh DE, posted my tale of woe on a couple forums, and tried to ignore the tank for the rest of the day.

2 days AEF

The next day the tank had cleared enough that I got a decent look at what was going on inside. The ‘true undata’ seemed to look ok, but all the other SPS were white – it was like looking at a show tank from 1982. Some of the fish that I would have sworn were dead were actually alive. Most of the LPS were still sucked tight to their skeletons, so I had no idea what was going to survive.

3 days AEF

The water cleared enough after another 150 gallon water change to take some photos that didn’t look like a tank of milk. More of the fish seemed to be gone and I was braced for a gradual die off of everything else.

4 days AEF

The water clarity was almost back to normal, and it looked like most of the fish made it. One clown, 3 chromis, a hybrid PBT and a mandarin had died, but the Picassos, the radiant wrasse, the flame wrasse, the swales basslet, the remaining pair of mandarins, a home-made banggai cardinal, the cleaner shrimp and the harlequin shrimp pair all were alive and seemed fine. I even fed them and they all ate. I had no idea what to think about the corals.

7 days AEF

As it turned out, every acro was dead as were most montis, poccis, the duncans and I was prepped to lose all the chalices as well. Hanging on were the undata, the double headed scoly, the dendro, some acans, and cespitularia (it had actually grown in the days AEF). I grabbed a 5 gallon bucket and filled it with acro skeletons, but left everything that didn’t have algae growing on it in the tank. My wife pointed out that the tank didn’t really look that bad as tanks go, and that it indeed could have been worse.

24 days AEF

About two weeks later I did a water test and everything came out normal, so I bought a yellow tang to help with any sneaking algae problems. About a week after that I added some ‘canary’ acro frags and they did well. Some of the corals I thought were goners, but that I didn’t remove, came back – most notably the S. wilsoni. Some of the chalices were down to 1 mouth but hanging on.

7 months AEF

The reefing community was nothing but supportive and two months AEF I started adding corals en masse. I got two great boxes of mini colonies from Liveaquaria.com, and tons of frags from local friends. Never was I happier that the attitude of my local reefing community is one of giving your corals away. After 7 months, the tank was a healthy fruit stand again, and about a year later I have to prune colonies back so they don’t fight and I am starting to think about removing corals just to make room.

So, what happened?

Even redundancy in design won't prevent human error

It was all my fault.

I did several stupid things at the same time. First and foremost, I worked on the tank when I was in a hurry (even though I know better). Instead of going to see Mitch Carl speak at Bay Area Reefers, I was going to a production of ‘The Velveteen Rabbit’ with my wife and daughter. We were running late, and just before we left, I added fresh Kalk powder to the Kalk reactor, then I raised the float switch on the auto top off to bring the salinity down a little, and, because I was in a hurry, I left the pump that drains the skimmers external collection bucket on. Then, I left for 7 hours. So, 2 cups of brand spanking new Kalk powder were washed into the tank by the auto top off, and, as if that isn’t enough, any Kalk powder that would have been left was pumped into the tank because once the skimmer started going crazy there was nothing to stop it because the drain pump was on. Had I not been in a hurry, I would have thought those three stupid actions through and not done them. All of this was preventable, and a Sanjay Joshi truism is proven again – “the person running the system is the system’s single biggest point of failure”.

My family was particularly helpful on the discovery of this disaster. Essentially, they asked if they could help, were politely told no, and then got out of the way. I am so lucky to have Libby and Kalin both understand what ‘focus’ means, and to know the best thing to do when someone is focused is to leave them alone so they can get stuff done.

What changes have I made to the system?

Since the Epic Fail, I turn off the auto top off when I add Kalk to the reactor. After the powder is mixed, I turn it back on and make sure that the system doesn’t need too much top off at that time. I also have added a ‘turn past’ timer to the skimmers external collection chamber’s drain pump. When the container is full, I turn a knob that turns on the drain pump, but turns it off again in a few minutes. Of course, I know that neither of these changes are foolproof and in no way will protect my system from me doing dumb things in the future.

If proper and swift action is taken, recovery from disaster is possible


What did I learn?

Despite my overriding the precautions I put in place, my forethought served me well. I had most of what I needed to deal with the disaster on hand – mixed salt water, DI resin, salt mix, carbon, a big canister with a clean pleated cartridge and vinegar. Without those things, this disaster would have been much worse. I can’t imagine how I would have felt coming home to coral soup but not being able to do anything about it because the LFS were all closed.

I learned that fish and animals are way more resilient than we might think. I had corals I was sure were dead recover completely, but if I had tossed them, they would be gone.

Hopefully, I finally really leaned the three main rules to avoiding disaster:

1) If you are in a hurry, don’t do anything to your system.
2) If you are about to leave the house, don’t do anything to your system.
3) If you are distracted, don’t do anything to your system.

In conclusion

I hope this tale of utterly preventable disaster helps you avoid a future reef keeping disaster yourself. I thought a long time about writing this article before I actually put fingers to keys because the majority of articles on reef keeping are about success. Everyone loves pretty pictures of thriving corals and fish, but more importantly, people don’t like to dwell on failures. Failures make people feel bad. Failures make people look bad. People especially don’t like to advertise their failures. But, I think the failures are just as, if not more, instructive than the successes, and think we need more discussion of the dumb things we do so we end up doing less of them. If we own our failures, it seems everything is better for everyone.

I look forward to showing of pictures of my thriving tank in about six months when it is lush and full, unless of course, I pull another epic fail.