Health Update

It’s been eight weeks since my discharge from the hospital and time for an update. For the most part, I’ve been getting steadily better. No more tubes. The only medication is another four weeks of blood-clot slowing Warfarin to make sure I don’t develop any more deep-vein thrombosis (DVT) blood clots. I’m still getting frequent blood tests for that medication as well as a chest x-ray every week or two to check on the one remaining issue: a persistent pleural effusion in my right lung. But I’ve been a bit tired and sometimes somewhat short of breath, particularly when climbing the steep hill at the end of my regular walk.

Yesterday I was supposed to have another thoracentesis procedure to remove more of the liquid in my right pleural cavity. The doctor showed me the most-recent x-rays which clearly showed something about one-third the volume of my right lung was keeping it from fully inflating. He also showed me a before-surgery x-ray. Wow! The pleural effusion back then filled a huge percentage of the cavity. My team of doctors had hoped the remaining effusion would eventually dissipate, but so far it hasn’t done so.

Before the procedure the doctor used ultrasound to get a sense of the size and location of the effusion, but what he discovered was that there was very little liquid in there. The majority appeared to be a non-liquid organized mass which he said was likely to be the consistency of rubber cement. This is essentially what my thoracic surgery in October was supposed to remove (along with a chest drain for any liquid) so it seems what I have is what remains from before the operation. I guess it wasn’t possible to remove it all. (I had a small-incision thoracic surgery. The doctor showed me photos of the scars from the traditional method — a shocking cut from in front of the shoulder all the way down to the hip. What I had was bad enough!)

Since thoracentesis can only be used to remove liquid, the doctor decided not to perform the procedure and instead sent me to radiology for yet another chest x-ray.

At this point it looks like one-third of my right lung can’t fully inflate due to the pleural effusion. I’m going to continue getting chest x-rays every week or two to check. If it gets better on its own…great. If it gets worse, I’ll probably need more thoracic surgery will be required to remove it. Ugh. But it seems likely that I may have to just “live with it.” The surgery isn’t without risks and complications of its own, so the doctors aren’t anxious to cut me open again. I’m generally healthy and active now, so the condition isn’t really affecting my lifestyle. I may just have to get used to being a bit more tired and not able to climb Mt. Everest.

Chronology of a Lost Month

This is one of those too-much-information (TMI) posts. I spent most of October 2011 in the hospital due to a liver abscess and subsequent complications, but due to my condition and medications I remember very few details of what happened during that time. I’ve reconstructed the following chronology based on notes taken by, and discussions with my wife, son, sisters and doctors. I wrote this primarily for myself, so I’d have a better understanding of the events. Warning: I don’t expect it will be of much general interest. And it’s long!

October 1-2

The last week of September 2011 was filled with photography: two photowalks with Google+ friends in Santa Cruz and Half Moon Bay; a third in Golden Gate Park with a group from the Marin Photo Club; and a long afternoon/evening private shoot on Alcatraz with Joe McNally. I was also scheduled to co-lead one of the Worldwide Photowalks with Catherine Hall in San Francisco on Sunday, October 2.

But on Friday it all caught up with me. I was wiped out and felt like I had really done too much. As the day progressed, I felt worse and worse. I knew something was seriously wrong. I had only a slight fever, but the shivers and shakes were severe. I was producing very little urine, and what I did produce was extremely dark brown. Uh-oh, I thought: There’s a problem with my kidneys.

At 3am on Saturday October 1, I called the advice nurse at Kaiser Permanente, our HMO. She asked the appropriate questions and said she’d discuss my symptoms with an M.D. on duty. The advice nurse called back and told me to head to the lab for some blood tests as soon as they opened that morning. My wife had her own health issues (retinal bleeding) and couldn’t drive, so I drove myself to the lab, had the tests and drove home. Within two hours I got a call from the doctor who told me I should come in to see her ASAP and to plan on heading directly to the emergency room from there.

I drove myself to the ER, which probably wasn’t too smart. My wife rode with me, but I think it was a bit of a wild ride. In the ER I was quickly put on IV saline and given lots of tests: a chest x-ray showed some pneumonia; there were kidney problems and bowel problems. (The first doctor was concerned about a possible bowel obstruction.) Finally a CT scan showed a mass “the size of a large grapefruit” in my liver. They didn’t know what it was, but the possibilities included a malignant or benign tumor or an abscess. The only way to tell would be a biopsy, and since it was now Saturday night and the procedure couldn’t be done until Monday morning, I was admitted to the hospital and waited 36 hours. My fever was up to 103.4.

October 3

Monday morning they wheeled me down to a department I’d never heard of: Interventional Radiation, where a team of doctors and assistants perform surgical procedures with the aid of live x-rays, fluoroscopy, etc. It’s pretty wild. There’s a lot of hardware and computer-enhanced imaging and the team members are all wearing lead coverings. They inserted a large needle into my liver. I was quite sedated, but I do remember the point at which one of the doctors said, “We’re getting pus,” or something like that. I realized that was good news. It was an abscess, not a tumor. They drained some of the fluid and inserted a drain tube, which I kept for the next four weeks. They never did perform a biopsy of the liver itself.

It was getting harder and harder to find veins for IVs, so somewhere along the line — I think it was this day — a nurse inserted a peripherally inserted central catheter (PICC) line into my left arm. This is a catheter in the vein that runs up the arm, across the chest and into the vena cava. It’s not as creepy as it sounds. Now I had two ports that could be used for IV medications and even blood draws without having to poke a new vein each time.

I spent the rest of the day in a recovery room, but I was getting sicker and sicker by the hour, so they moved me to the intensive care unit (ICU). The pain in my abdomen was now excruciating when I breathed. I had developed a pleural effusion in my right lung. My breathing was faltering and my arterial blood gases weren’t good. I became acutely septic. In other words, I was one very sick puppy.

They started respiration therapy and told my wife if I couldn’t breathe more deeply they would have to put me on a ventilator. I was holding my breath to avoid the pain, but apparently no one understood that. I wasn’t able to communicate because I was just too out of it. Finally a nurse figured it out, got me started on continuous morphine — it apparently took quite a bit — and she and my wife  talked me into exhaling instead of holding my breath. Ventilator avoided.

The good news at this point was I didn’t have liver cancer. The bad news was I was going downhill fast from a bacterial infection in my bloodstream and messed up blood gases due to my poor breathing.

In the meantime, the labs identified the primary/original culprit: a bacterium known as Streptococcus milleri. This bacterium is one of those that we all have in our mouths, stomachs and intestines, but if it gets into our bloodstreams it’s extremely toxic. Normally the liver can deal with small amounts of milleri, but in my case there was way too much and the liver was overwhelmed. Rather than filter it, as the liver normally does, it created a separate space (the abscess) where it kept the infection segregated. Because of the size of the abscess, the infectious-disease specialists thought this process had been going on for four to six months. They started an IV antibiotic that was specifically targeted to Streptococcus milleri.

October 4

Tuesday the doctors were worried I was developing Acute Respiration Distress Syndrome (ARDS), which along with the acute sepsis was a potentially fatal condition. I had pneumonia, but they weren’t too worried about that since the antibiotics should be taking care of it. I also had serious edema. I had gained 25 pounds of fluids since this whole thing started.

October 5

The next morning my team decided to perform a thoracentesis. Using local anesthetic, they inserted a needle below my right shoulder blade into the pleural cavity around my lung to remove some of the fluid that had accumulated there. They removed 200-250cc of fluid, but another x-ray revealed 600-700cc — two-thirds of a litre! — remained. At this point I was getting a chest x-ray at least once every day, which continued until my last day in the hospital. The fluid they removed was more similar to the fluid from the liver than the doctors liked. They told me it was not uncommon for the infection and fluid from a live abscess to reach the lung, sometimes via a “tunnel”. The liver is just under the right side of the diaphragm, which is why it had become so painful to breathe. The doctors increased my pain medication again.

October 6

By Thursday the size of the liver abscess was dramatically smaller. My fever subsided and my blood gases and blood pressure, which had been very low, returned to normal. In the afternoon, after three days in the ICU, I was transferred to a regular floor. But the doctors continued to drain more fluid from my pleural cavity. The subject of thoracic surgery was raised, but only if the abscess “shelled” out from the liver or the fluid in my lungs developed into too many separate “pockets” such that it couldn’t be easily drained.

Unfortunately, I developed swelling in my left arm and an ultrasound revealed I had developed a blood clot — an acute deep vein thrombosis (DVT) — from the PICC line. I was immediately started on Heparin to thin my blood and reduce further clotting.

October 7-9

Over the next three days my condition improved steadily. I still had a lot of tubes including an IV (saline, two antibiotics, pain meds, diuretic), an arterial catheter (to track blood gasses), the liver drain and the IV PICC line, but with the help of the physical therapists I was finally able to get out of bed (to use the bathroom instead of a bedpan!) and to walk. At first it was just a few steps but by Sunday I was able to climb twenty steps, a requirement for going home.

But the hospital is a quiet place on weekends. A number of departments have minimal staff on Saturday and Sunday and in general it seems procedures and decisions tend to be delayed until the start of the week when the full team returns.

October 10

On Monday things got busy once again. It started with another CT scan, which unfortunately showed more fluid was accumulating in the pleural effusion, which had also become loculated — split into a number of those separate pockets. This meant they couldn’t be reasonably reached with further thoracentesis, so the only solution was thoracic surgery, and for that I’d have to be transported to the larger hospital in San Francisco. Furthermore, the original liver abscess wasn’t resolving as quickly as the doctors had hoped. It hadn’t changed in size since the previous CT scan. To make matters worse, I now had several small blood clots, which could be new or could have broken off from the original one. These would need to be watched.

The medical-transport crew arrived like a SWAT team of paramedics. With a drill-sargent nurse as their crew chief, they whisked me off by ambulance to San Francisco. Once there, the PICC line in my left arm was removed and replaced with a new one in my right arm.

October 11

Tuesday was thoracic surgery day. I learned they would make multiple relatively small incisions and use a camera and tools to remove the material of the pleural effusion. There was a 20% chance it might be worse and they’d have to perform a more complex procedure with a lot more cutting. (That didn’t happen, thank goodness!) Post-op there was some chance of scarring of the lungs and that I could lose 10-20% of my lung capacity.

I woke up after the surgery in the cardiovascular (CV) ICU in a lot of pain and with some new tubes: a catheter in my bladder and drain tubes in my chest. I still had the liver drain as well. To make matters worse, there was a mixup on my pain medication and I was without it for many hours. Of the times I can remember, this was the worst for me. Finally they gave me IV Dilaudid (synthetic morphine) and things calmed down.

October 12

Recovering from the surgery, was tough. Again, this may be because I was also becoming more coherent and aware than I had been for the first eleven days of this ordeal. I was still in the CV ICU. My edema had become quite serious, but the diuretic (Lasix) was finally kicking in. My blood pressure reached 215/145, and it could only be measured on my legs because I had a clot in one arm and the PICC line in the other.

October 13

Thursday was more of the same, but I was stable so they moved me out of the CV ICU to a CV post-surgery floor. This is also pretty much the first day I can remember. Except for the day just prior to surgery, the first twelve days are still pretty much a blur to me at best. I think it’s a combination of medication and the body’s reaction to the infection, procedures, etc.

Along with full consciousness came real discomfort. I still had serious edema. My feet looked like bloated potatoes. My body temperature swung back and forth between cold chills and hot sweats, which was a bigger problem than it might sound. The pain was weird. It was never acute. In fact, it took me a while to realize it was actually pain because it was so non-specific. All I knew was that I was extremely uncomfortable and the only solution was narcotics. After surgery I was given an intravenous Patient Controlled Analgesia (PCA) device. You get a button you can press to get an immediate small dose of Dilaudid, then it’s locked out for some period of time like 10 or 20 minutes. I also had oral narcotics (Norco/Vicodin and Percocet) but these took 45 minutes or more to work. (Actually, the Percocet just made me stupid. It didn’t seem to do anything for the pain.) The PCA was great when the pain broke through the other drugs and I couldn’t wait until the next oral dose took effect. Unfortunately, I worked hard to use it as little as possible, so they took it away! That will teach me. I guess if I’d pressed the button a few times an hour they might have thought I needed it longer.

My other error had to do with food. The regular menu was decent, but they had to test my blood sugar before every meal and possibly give me a somewhat painful insulin injection depending on the results. I’m not diabetic, but I guess the blood sugar/insulin relationship frequently gets weird after surgery like mine. So I made a deal with one of my doctors. I said I’d be willing to eat the diabetic menu if I could stop getting the tests and injections. Big mistake! Whatever you’ve heard about hospital food, there’s nothing as bad as the food they serve to diabetics. I don’t really how to describe it other than to say it has no flavor whatsoever. And of course there’s nothing interesting on the plate to begin with.

October 14

The main event Friday was a transesophageal echocardiogram. From the time of the original diagnosis, the infections-disease doctors were concerned about my heart. Streptococcus milleri often causes endocarditis, an infection of the heart valves. The test sequence for this is rather strange. First they perform a non-invasive (ie, from outside the chest) ultrasound transthoracic echocardiogram. If the results are negative — ie, they don’t detect any endocarditis, as was my case — then they perform the more invasive transesophageal echocardiogram. This is one of those procedures where you have to swallow an ultrasound transducer to get it into your esophagus, which positions it close to the heart valves so the radiologists can get a very clear picture. As with some of these other procedures, it wasn’t as bad as it sounds. My results were thankfully negative for so-called vegetation on the valves.

But I was still quite uncomfortable. I wasn’t allowed to get out of bed on my own until I had the approval of the physical therapist. I had six different IV drip bags feeding the two ports in my PICC line: saline, two antibiotics, blood-pressure medication, pain medication and the diuretic. But for the first time since the earliest days in the emergency room, they removed the supplemental oxygen I’d been breathing. And with the help of the pain medication, I finally got a good night’s sleep.

October 15-16

Finally, on Saturday, the doctors began talking about my going home. Each department had their own requirements for my discharge. The surgeons wanted my chest drainage to stop. They discontinued the active (vacuum) suction and let gravity take over. Other MDs were still concerned about the known DVT (blood clot) in my left arm and possibly one in the right arm, so I was wheeled down to radiology for another set of ultrasounds. The left arm clot was smaller and there was no sign of one in the right. Regardless, they told me I’d be on steady Heparin until my release form the hospital, then three months of Warfarin (coumadin) at home. I was also still receiving intravenous antibiotics (Ceftriaxone) for the original liver infection, which would also be continued after my discharge. As before the lung surgery, physical therapy wanted to make sure I could get in and out of bed on my own and walk up and down stairs. The bed was still a challenge, but I was able to walk about 1,000 feet and climb 24 steps. My edema was still a problem, so the IV Lasix continued.

I was feeling better and getting stronger every day, but still had a lot of pain from the surgery. The pain had shifted from the incisions to my ribs. The surgeon explained that they had to spread the ribs, tweak some muscles and stretch the cartilage to do what they needed to do, hence the pain I was feeling.

Like I said before, not much happens on weekends, so Sunday was just another day of waiting, trying to control the chills, sweats and pain, and wishing I hadn’t opted for that diabetic menu. On occasion I was able to talk a technician or food server into giving me something I wasn’t supposed to have. I was desperate for anything with flavor.

October 17

Monday morning the first string returned to work and I was able to get the procedures and tests I needed to wrap things up. The most significant was another visit to the Interventional Radiation department to check and reposition the drain in my liver, since that one was going to stay in even after I went home.

October 18

After 17 nights in two hospitals, I was finally discharged. At the very last minute, a physician’s assistant from the surgery department came and removed my chest tubes. Just like he said, it hurt slightly for about three seconds and then it was done.

Because of the long wait through the weekend, I was fairly strong and stable on my feet. Physical therapy had given me a cane, but I no longer needed it. I could walk up and down a full flight of stairs, and while it was a bit awkward and painful, I could get into and out of bed on my own.

I went home with a fair amount of paraphernalia and medications including:

  1. the PICC line in my right arm so I could self-administer intravenous drugs and get blood tests without another needle in my vein each time;
  2. a “JP” drain in my liver with an external suction pouch, which I safety-pinned to my clothing;
  3. lots of bandages over about eight incisions and other wounds from various procedures;
  4. intravenous Ceftriaxone (Rocephin), an antibiotic to kill off any remaining Streptococcus milleri;
  5. Metronidazole (Flagyl), another antibiotic to fight an amoebic infection they thought I could have. I was still taking it because a few weeks were required to get the results of the lab tests I had early on;
  6. Warfain (coumadin) for the potential and known blood clots; and
  7. Lisinopril for my blood pressure which had become higher than normal during my hospitalization.

Recovery

Once home, my recovery progressed quite rapidly. My wife, a retired R.N., changed my dressings. Kaiser has an amazing Home Infusion service, which delivered and monitored my IV antibiotics. I went frequently to the local outpatient infusion center for blood tests and PICC-line dressing changes. After about three weeks, I ended the antibiotics and the PICC was line removed.

On November 1 I returned to the Interventional Radiation department in San Francisco. It was supposed to be another “drain check” procedure, but they removed the liver drain altogether without me even knowing it. Never felt a thing.

Because of the DVTs (blood clots) I’m still taking Warfarin daily and getting blood tests once or twice a week to monitor the clotting times. This should end in early January 2012.

I’ve also been getting chest x-rays nearly once a week. I still have a pleural effusion: something (liquid or some kind of gunk) between my right lung and the pleural lining. This reduces my lung capacity by 10%-20%. My doctor says it will eventually dissipate, but it sure is taking a long time. I do notice that I have non-infectious pneumonia-like symptoms. There’s occasional slight pain, and I sometimes get a little short of breath and get tired a bit more easily than I’d like. But I don’t generally notice theses symptoms. Compared to how I felt six weeks ago, I’ll take what I’ve got.

How Did This Happen?

Okay, so how did all this happen? What was the cause of the Streptococcus milleri liver infection in the first place?

As I mentioned, this is a bacterium that we all have in our mouths, stomachs and intestines, but it’s toxic in the bloodstream. My team of infectious-disease doctors were hardcore medical detectives. They were extremely inquisitive about my travel, activities, diet and dental history. One doctor in particular kept asking me about recent dental work. Yes, I’d had a cleaning by a dental hygienist, and I got a new crown during the summer, but that didn’t seem to be it. I had to think back. What happened in the April-May timeframe of when this might have started.

And then it occurred to me. Back in the spring, my hygienist convinced my to start using a Waterpik in addition to flossing, and I did so every day. But there was one area of my gums that always bled. If I’d read the instructions for the Waterpik, I’d probably have found they said something like, “If your gums bleed, stop.” But I just figured I needed to toughen up those flabby gums, so I kept using the Waterpik on them night after night. Every night they bled. And every night that opened a pathway for more Streptococcus milleri to enter my bloodstream.

We’ll never know for sure. The evidence doesn’t give us a provable cause-and-effect relationship, but the circumstantial evidence is so strong, that I and my doctors are satisfied that my constant traumatizing of my gums was the ultimate cause of my live abscess. Not surprisingly, my Waterpik was swiftly and unceremoniously disposed of once I got home.

Cessna’s Vision Problems

My wife, Cessna, has been having some vision problems, and since so many friends and family members have asked for details through email and all the social networks we’re on, I thought it might be easiest to publish the info here and just link to it. This is my version, not hers, so I hope you and she will forgive my inaccuracies.

For a number of years Cessna thought she had more floaters in her eye than normal. Three years ago her optometrist noticed some minor irregularities on her retinas, but no one thought it was anything serious. Cessna teaches Aikido so she rolls and falls a lot, and at one point she noticed a few flashes in her eyes and other irregularities. She had some tests by our HMO’s opthalmology department, which merely confirmed that yes, there was something on her retinas, but no big deal.

Then in June of this year she started having more serious things in her left eye: blobs of stuff, larger than the usual floaters, that more severely obscured her vision. She returned to opthalmology who diagnosed her with retinal vasculitis, inflammation of the blood vessels in the eye. They suspected the inflammation was causing a reduction of oxygen in the blood and the eye to therefore generate additional (undesirable) blood vesels as well. The blood vessels (veins, arteries or both) were leaking some blood into her eye, and that’s what was obscuring her vision.

The doctors started running all sorts of tests: x-rays, MRIs, blood tests, etc. They also performed eye angiograms using injected fluorescein dye, which enhances the image of the blood vessels and apparently can actually show the blood leaking. They confirmed their diagnosis, but Cessna had a pretty nasty allergic reaction to the fluorescein. She had a second angiogram and the reaction was so severe, even with a dose of Benadryl, the assistant was ready to administer an EpiPen.

Although the doctors were moderately confident in the diagnosis of her condition, they still didn’t know what was causing the inflammation so they didn’t know how to treat it. They told Cessna that if it didn’t go away (and it didn’t appear to be) she might need to start using immune-suppressant drugs like Humira to reduce the inflammation. These are nasty drugs for anyone, but Cessna doesn’t have a particularly strong immune system to begin with. The doctors also said they might want to inject Avastin directly into her eyes to halt the growth of new blood vessels. (Yeah, my thought, too.)

The optometrist suggested Cessna get the advice of a nutritionist, which she did. If there was anything that might avoid the immune-suppressant drugs it was worth a try, so Cessna had another slew of blood tests looking for food sensitivities. The result was that she started an incredibly strict diet in mid July. We’re not just talking gluten-free; we’re talking everything-free. It’s a diet based upon her specific sensitivities.

That’s the course she was following until a week ago, when her right eye (which was the good one) suddenly became completely occluded. She couldn’t see anything through the floating mass. We went to the opthalmic experts the same day and they told her it was blood — a lot of it. But the recommended treatment was the same: maybe the Humira, but just wait and see. Well, that’s a bad joke since she now couldn’t see well enough to drive or do many other things. And there wasn’t any indication it was getting better. In fact, it was getting worse.

We decided to go outside of our HMO and get second opinions from other local opthalmic gurus, and that’s what we’ve been doing for the past three days. Here’s what we’ve been told so far:

  • The diagnose is still retinal vasculitis.
  • They don’t know the cause, but there are still a few more tests that will be done.
  • If they can’t find a treatable cause, they may want to give her prednisone. But that’s a steroid with all sorts of bad side effects — Cessna had an aunt who died from taking it long term — so it can’t be used for more than a few months.
  • If that doesn’t work, then they’re talking about Humira, etc.
  • They used ultrasound today to determine that she doesn’t have any retinal detachment. (I got to see this in real time. It was an amazingly clear picture, taken through the eyelid and all.)
  • The blood in the right eye might dissipate by itself, but it’s going to take “months”. If it doesn’t, they’ll have to remove the blood surgically.
  • The doctors really want to see what’s going on in the right eye, but they can’t because the blood is in the way. Once it’s gone (on its own or via surgery) they want to do another fluorescein angiogram, but due to Cessna’s reaction she’ll need prednisone and Benadryl beforehand.
  • Once her eyes clear, the doctors also want to go in there with lasers and zap the extra blood vessels. Apparently she’s already permanently lost vision in those spots anyway, but it doesn’t sound like they’re too critical.

At this point Cessna’s hanging in there. It looks like she’s got another week of tests and doctor visits, and the extreme diet continues. The best news came today from the first doctor to tell her that he didn’t expect any of this to be permanent. He wasn’t sure, of course, but he told her she should expect to recover her vision.

Thanks to everyone for your good wishes. I’ll pass them on to Cessna.

Labs and Papers for Black & White

This post is a review of black-and-white printing on eight different papers from four U.S. photo labs.

I’ve been uploading my recent photos to Google Plus, where I’ve been getting good feedback and meeting great photographers. When I published this b&w image of Bubba’s Diner in San Anselmo, California, the comments were particularly enthusiastic. And then, totally out of the blue, two people said they wanted to buy prints. How cool is that? I didn’t get (back) into photography to sell my images, but why not? If someone can get pleasure from hanging one of my photos on their wall, that would be pretty cool.

Bubba's Diner, San Anselmo, California
Bubba's Diner, San Anselmo, California

How to sell prints to my first two customers? I quickly cleaned up my SmugMug portfolio at DougKaye.com — it still needs a lot of work — and upgraded to a Pro account so I could order the prints through there and even sell them directly. But before I accepted money for my work, I wanted to know what the prints would look like, so I decided to order prints of most of my portfolio images for myself. SmugMug uses two labs, and I opted for BayPhoto, which appears to be their more high-end lab. (The other, ezprints, is somewhat less expensive.) I first ordered a print on Kodak Endura paper, which SmugMug/Bay Photo refer to as their Lustre stock. When the print arrived, I was rather disappointed in the color and texture of the paper. So I turned to other photographers on Google+ and asked them what labs and papers they used for b&w. I got a few recommendations and then ordered prints from four labs (including BayPhoto) on eight different papers. Here’s a summary of my opinions, listed by the coolness/warmth of the papers, starting with the coolest. It’s not an exhaustive test, as I’m sure there are far more papers and labs out there. But if you’re thinking about black-and-white printing, this may be a helpful starting place.

Bay Photo’s Lustre is Kodak’s Supra Endura VC, a resin-based photographic paper finished with a “fine grain pebble texture,” which is too much artificial texture for me. SmugMug recommends it as a compromise between full matte and glossy and as a way to minimize fingerprints. I expect my prints to be matted and mounted behind glass, so fingerprints aren’t really an issue. This is the coolest of all six papers. It actually has a noticeable blue cast to it. I’d say it’s my least favorite of the batch. ($3.23 via SmugMug for an 8×10 color-corrected print. Direct from BayPhoto: $3.50, or $1.79 without color correction.)

Bay Photo’s Metallic (Kodak Endura Metallic VC) is actually a touch warmer than the Endura, which shows how cool/blue the regular Endura really is. The metallic is obviously very glossy and has a bit of a greenish cast to it. The whites and highlights are very reflective/silvery, hence the metallic moniker. I don’t think I’d be likely to use this paper. ($4.12 via SmugMug for an 8×10 color-corrected print. Direct from BayPhoto: $4.03, or $2.06 without color correction.)

Bay Photo’s Glossy (also a Kodak Supra Endura VC) is the third coolest paper, and still not particularly warm. The blacks are deep and there’s pretty good detail in the shadows. I’d probably use this for images where I wanted to emphasize the drama of a contrasty, particularly crisp picture. ($3.23 via SmugMug for an 8×10 color-corrected print. Direct from BayPhoto: $3.50, or $1.79 without color correction.)

MPIX offers a paper they call True B&W, Ilford’s True B&W. This is a silver photographic process, so there are no color dyes or inks at all. It’s yet another cool paper, almost as cool as the Bay Photo papers. Like the Endura Metallic, it has a slight greenish cast. Of all the printer/papers combinations, it’s the lowest contrast. There are no deep blacks and it has the least detail in the shadows. The opposite of Bay Photo’s Glossy paper, I might use MPIX’s True B&W when I particularly wanted a softer, gentler low-contrast look. ($2.49 for an 8×10 print)

I wasn’t really thrilled with any of these combinations, so I asked Matt Russell, a friend who shoots and sells a lot of b&w landscapes, about the high-end labs he uses. He suggested I look into West Coast Imaging and Digital Silver Imaging. WCI has a $250 minimum order, but they were willing to work with me on these tests. Obviously, you don’t want to order one or two 8×10’s at a time from WCI.

DSI uses Ilfospeed Resin-Coated paper with an Ilford Pearl (lustre) surface for their Custom RC prints. This is another lower-contrast combination, but not as low contrast as the MPIX True B&W. The blacks are also deeper and richer than the MPIX, but still not as deep as others. Furthermore the blacks are rather warm. It’s a very nice combination: a neutral paper with slightly warm blacks. One of the best. ($18 for the first 8×10; $9 for prints 2-10.) DSI also offers a less-expensive Direct to Print option (ie, not their Custom service) that delivers Ilford RC Pearl prints for much less ($4.59 for 1-9 8×10 prints; $4.19 for 10 or more).

DSI’s Custom Fiber Base prints are on Ilfobrom Galerie Fiber paper. This is fairly warm paper, but the blacks are actually cooler, similar to the MPIX True B&W. It’s a heavy double-weight semi-gloss fiber paper, about the same weight as the Ilford Gold. It’s in the lower-contrast category like the MPIX True B&W and the DSI Custom RC, but not as low-contrast as the others. The paper is quite warm, but the blacks are cool (again like the True B&W). The depth and richness of the blacks are excellent as are the shadow details. ($38 for the first 8×10; $25 for prints 2-10)

WCI offers Ilford Gold (Ilford Galerie Gold Fibre Silk), warmer still than DSI’s Custom Fiber Base. It’s a very heavy paper made with real rag and has a marvelous rich look to it without sacrificing good, deep blacks. ($12.22 for the first 8×10 of a single image; $10 for prints 2-9; $8.33 for 11-.)

WCI also offers Silver Rag (Crane’s Museo Silver Rag), a 100% cotton paper. It has slightly more rag texture than even the Ilford Gold. This is the warmest of all the papers I tested, with a slightly yellow cast. I would use this paper if I wanted a particularly warm look. Otherwise, I’d stick with the Ilford Gold. (Same price as WCI’s Ilford Gold.)

With the exception of the Ilford True B&W paper used by MPIX, all of the above have deep, rich blacks. It’s possible that a different print on the True B&W might not have such a low-contrast look. But while all the others have solid blacks, all but the two WCI combinations do so by increasing contrast and therefore losing some detail in the shadows.

There’s no question that the more costly prints from DSI and WSI are superior to the others. DSI’s Custom Fiber Base prints are downright expensive.

I’m sure your experiences vary and you probably have used labs and papers not listed here. Leave your reactions in the comments for all to see. DSI’s pricey Custom Fiber Base prints are perhaps the best of all for most of my work, but damn expensive. It’s the one option that starts to become more than a substantial part of the total (including matting and framing) costs. A 12×18 costs $88 plus tax and shipping. For most high-quality work, I’d probably chose Ilford Gold from WSI if I had enough work to justify their $250 minimum order. Otherwise, I’d probably go with DSI’s Direct to Print Ilford RC. For by far the fastest service and the lowest cost (and so long as I wanted a very crisp look), I’d use Bay Photo’s Endura Glossy. WCI’s Silver Rag is an option I’d reserve for those times when I needed very warm (almost toned) whites.

Fair Use?

This is a fascinating case, particularly for me both as a photographer and a fair-use advocate. You should probably read the story for yourself, but I’ll summarize it here. Andy Baio is well-known and respected in the tech world. He produced an album (Kind of Bloop) based on the songs from Miles Davis’ classic album, Kind of Blue. He got all the permissions and rights he needed to the music, but when it came to the album art, he created a somewhat pixelated version of the original image without getting any permission. It turns out the orignal album-art photo was taken by and belongs to a great photographer, Jay Maisel. Jay sued Andy and they settled out-of-court for $32,500. Andy still feels he was right based on the concept of “fair use.” Here are the two versions: Jay’s original and Andy’s interpretation.

kind_of_bloop_comparison-20100701-172352

What do you think? Should Andy have been able to sell his album using the cover on the right without first getting permission from Jay? Would you say that Andy’s version qualifies as “fair use” of the original? It’s a tough call for me.

First, you should know that I’m a supporter of and contributor to the Electronic Frontier Foundation (EFF), who played a role in this case, so I’m a strong believer in the fair-use concept. I believe our copyright laws are severely inhibiting creativity and are increasingly just serving a copyright consortium rather than serving the public good, as originally intended. I have some experience in copyright, trademark and other intellectual-property law, but I am not an attorney. I’m a layperson who has taken an interest in this area for decades. Most notably, I am not up-do-date on the latest details of the fair-use doctrine. In other words, I’m not qualified to give a legal opinion about who is right or wrong in this case — only an emotional one.

Given that disclaimer, I do have an opinion, event though it’s not based in law. To me, I think Andy’s image is a derivative work that goes beyond what I consider to be fair use. From a purely practical point, I can’t figure out why Andy didn’t try to get permission to use Jay’s image in the same way as he did for the music? Did he think it was somehow more incidental? If you’re a photographer, your images are as important to you as a song might be to its composer. This is an iconic album cover, which on one hand suggests that it’s fair game for fair use, but it’s also a work of art and deserves the same protections as any other.

Ultimately, Andy asks an important question at the end of his blog post (scroll to the bottom of the page) where he writes, “Extra credit: Where would you draw the line?” Is there some point in abstracting the image at which the original image is obscured to the point at which the derivative work is no longer infringing of Jay’s copyright? Is this even a legitimate way to evaluate the issue? A fascinating debate in any case. What do you think?

Update: I should have mentioned that I first heard about this from Thomas Hawk, for whom I also have great respect. In this case, however, I disagree with him. But check out Thomas’ blog post and the comments.

Salvaging the Shoot

Once again, I’m determined to get the shot. In this case, it’s the full moon rising behind downtown San Francisco. Last night was my first attempt, but given the horrible results, it won’t be my last. I was about to delete all the images from the session, but first I decided to play with them to see how much I could extract before giving up.

Like all serious shoots, it began with research.

  • The experts told me the best time to shoot is when the moonrise is 30 minutes before sunset. That’s often the night before full moon on the calendar. In this case (June 14, 2011) moonrise was at 7:48pm and sunset was8:33pm. Not a bad spread.
  • To find the best position I used The Photographer’s Ephemeris, an awesome iOS app that shows you the exact position of the sun and moon on any date at any time.
The Photographer's Ephemeris
The Photographer's Ephemeris

Everything was ready, save for the one big fear: the fog, which everyone knows can come barreling in through the Golden Gate during the summer. But fog didn’t turn out to be the problem. Due to a moderate high-pressure system just offshore, there was no marine layer and no wind. And that meant haze and smog: a fairly heavy layer up to about 1,000 feet. Yuck.

But having gone this far, I schlepped all the gear (including a second body+tripod for a timelapse) to the location where I found three other photographers, all with Nikon gear. Two of them had pinpointed the location using The Photographer’s Ephemeris as well. It was so hazy, we couldn’t even see the moon until it was well above the skyline, so the photo below is one of the first of the evening. And one of the best. This was shot about 25 minutes before sunset.

Original from the Camera
Original from the Camera

As you can see, it’s horribly flat and dull. After some tweaking in Lightroom, I was able to recover some of the contrast and clarity:

700_9034
With Global Lightroom Tweaks and Crop

Yes, I could have further lightened the unnaturally dark and saturated water and made a number of other improvements, but I just didn’t want to waste a lot of time on this one.

I posted the tweaked image on Facebook, where photo pal Scott Loftesness suggested I see how it looked as a black-and-white. I popped it into Silver Efex Pro 2, where I spent some time making a number of global and local adjustments and ended up with this:

700_9034-Edit-Edit
Further Tweaked in Nik Silver Efex Pro 2

What do you think? It’s still not at all the shot I’m looking for, but compared to the original, I think it’s at least a serviceable image. If nothing else, it shows that if you keep working at it and consider all the options (b&w in this case) you can sometimes salvage a shot that would otherwise end up in the trash.

Update: I went back and tweaked the moon. First I changed the mapping from RGB into b&w, then I adjusted the contrast. Finally, I used a layer mask in Photoshop to merge the enhanced moon into the original image. It gives the picture an entirely different look, doesn’t it?

700_9034-Edit-Moon-720w

Happy Birthday, The Conversations Network

Yesterday was the 8th anniversary of IT Conversations, the longest running podcast in existence and the flagship channel of The Conversations Network. Since its founding, The Conversations Network has published 2,918 audio programs for an average of one every day for these past eight years.

Thanks to our members,major supporters and TeamITC, the wonderful folks you never hear about that bring you those new programs every day.

I’m a TWiP Again

Once again I had the privilege of being a guest host on the This Week in Photo podcast (#202), sharing the show with Frederick Van Johnson, Syl Arena and Ron Brinkmann, three of my personal photo heros.

On this episode of TWiP, in case of a water landing – take pictures, Getty Images acquires PicScout, Adobe gets touchy feely, and an interview with SnapKnot.com co-founder Reid Warner.

My first appearance was on episode #153, nearly a year ago.

Photography Workshops

Like any other photographer, I’m always looking for ways to improve my skills. There are a lot of options out there: books, magazines, community college classes, online videos (free and $$$) and local photography clubs. And then there are the photo workshops — they’re everywhere. I’ve attended two workshops in the past few months, and while that certainly doesn’t make me an expert, I do now feel like I know what to look for in the next one. (I’m not including the San Francisco stop of the FlashBus 2011 Tour, which was fun, but more of an event than a workshop.)

Artist's Road, Santa Fe, at Sunrise

In March I attended a workshop led by Derrick Story. A good friend, Scott Loftesness, had been to one of Derrick’s earlier workshops and enjoyed it. Since I was able to talk Scott into trying another one with me, and because Derrick’s classroom and studio are in Santa Rosa, California (just an hour from home), it was a low-risk investment. The two-day workshop included eight students and cost $495. Derrick provides lunch both days, but you’ve got to get yourself to Santa Rosa and pay for a hotel room unless you’re local.

Santa Fe Cathedral at Sunset

Two weeks ago I went to a very different kind of workshop: the Mentor Series Photo Trek in Santa Fe, New Mexico. This three-day program had 37 students, two instructors, a bus and driver for the first two days and cost $1,000, which included no food, housing or transportation to/from the event. Mentor Series is owned by Popular Photography and runs about a half-dozen  workshops each year all over the world.

So how did they compare? In the case of the Mentor Series Trek, it’s “trek” that’s the operative word. It’s more about the location and somewhat less about photography. Yes, the attendees are all photographers (some with some very fancy gear) but you spend virtually all your time on the go. The first two days we were on the bus getting from one scenic location to the next a few hours each day, and once we arrived, there were often miles of walking to do. Beautiful scenery to be sure, but more hiking than shooting. And certainly not a lot of time to stop and “work” a subject for an extended period. The best shooting was actually the day they dumped the bus and we walked the city of Santa Fe on foot: once at sunrise and once at sunset. [Santa Fe is one of the best cities I’ve ever shot in. You could easily spend two or three days just walking its streets with a camera. Great art and architecture, terrific light and shadows, and a community that is very accepting of (and used to) photographers wandering around.]

By comparison, Derrick Story’s workshops often include a location such as a local safari park or (as next month) an early morning balloon launch, but there’s usually just one outside event per weekend. The rest of the time is spent in his studio — he usually includes at least one model session — and in the classroom. And it’s the classroom (and the class size) that really sets the two experiences apart. Derrick spends some of his time actually teaching from a podium and he gives the students actual assignments. For example, he might send you into the studio to shoot a model using only a single strobe. That’s something you can do when there are only eight students and they break into groups of four. With 37 students — forget it; everyone is on their own.

This brings up the question of why take one of these workshops at all. Professional photographers on assignment are obviously going to shoot a lot. But we serious amateurs have an interesting challenge. When my wife and I recently went to Egypt, I would have loved to have been able to stop and spend an hour or two studying the light and playing with the composition at each location. I would have given up half or more of the less-visually interesting sites in order to have more time at a few of the good ones. But that’s just me. My wife doesn’t particularly enjoy standing around while I study and experiment, and certainly the 22 other non-photographers in our tour group wouldn’t stand for it.

In one sense this is the role that weekend or weeklong workshops play. They allow the serious amateur to immerse him/herself in photography, surrounded by other photographers in a context where their peculiar habits of stopping, studying and shooting are socially acceptable. I imagine this is why Trekkies go to conventions. Wearing Mr. Spock ears to the grocery store is going to earn you some very strange looks. At a workshop you can truly geek out. Even when you’re on a bus, it’s all photography. All the time.

And what about the other students? Looking back, it’s not too surprising that a group of 37 would include a wider range than one of only eight. But I was surprised that the Mentor Series Trek included same true novices, some with the most expensive DSLRs. There were times when the instructors had to explain the relationship of aperture to shutter speed and ISO, and that surprised me. The instructors were even cornered by students with questions like, “What is ISO and how do I set it on my camera?” or “How do I focus this camera?” (Perhaps not surprisingly, some of these technically naive students sometimes produced some of the compositionally most exciting pictures.) In the smaller group of Derrick Story’s workshop, the range of skills was somewhat narrower although it still varied more than you might expect. Derrick does a good job of giving assignments that are applicable to each student’s skills.

In Santa Fe, I had relatively little access to the instructors given the 1:18.5 ratio as opposed to 1:8 at Derrick Story’s workshop. But even in Santa Fe, they were there if you had an important question. Towards the end of the Mentor Series weekend each student had the chance to show each of the instructors five images for critique (ten images total), and those sessions were quite valuable. We each got four or five minutes of constructive criticism that was appropriate for our skills.

Another benefit of any workshop or joining a photo club is the chance to see how other photographers interpret the same objects and locations. This happens in both the small and large workshops. No matter your level of experience, there are always those moments of, “Wow, I missed that!” that are truly educational.

So which of these two (or any other) do I recommend? It depends on what you want, of course. If pure learning is your goal, then I’d recommend a workshop with the smallest number of students, even a day of one-on-one. And I wouldn’t worry about finding the absolutely best photographer. So long as it’s someone whose work you respect and has been shooting it for a lot longer than you, you’re going to learn. Of course, reviews and opinions of previous students will help a lot.

On the other hand, if it’s a destination you particularly want to shoot or if you particularly want to travel, a larger more-distant workshop might be better for you. Mentor Series, for example, runs treks to places like Switzerland, London, Hawaii, Sedona and Wyoming. If you’re drawn to one of those locations and you want to experience the places in the context of photography, these might be better choices for you.

As for me? My prejudices probably show through in this blog post. I’m signed up for Derrick Story’s Hot Air Balloon Photo Workshop in a few weeks. None of the Mentor Series treks are on my calendar. I’m going to continue looking for small-group workshops that I can get to without hopping on an airplane. I’m also going to spend as much time as possible taking photo walks with friends. For example, tomorrow Scott and I will be shooting at the San Mateo Maker Faire as we did together last year. It’s tremendously visual and there’s enough to keep you engaged for a full day or more.

The Amazon Web Services (AWS) Outage

Like many other sites hosted on AWS, all of The Conversations Network’s websites went down at 1:41am PDT on April 22, 2011. It would be 64.5 hours until our sites and other servers would be fully restored. A lot has been written about this outage, and I’m sure there’s more to come. Don MacAskill, another early adopter of AWS, has posted a good explanation of SmugMug’s experiences during the outage.  Phil Windley and I are hoping to interview our friend Jeff Barr from AWS for Phil’s Technometria podcast once the dust has settled at Amazon.

Many pundits have suggested this event highlights a fundamental flaw in the concept of cloud computing. Others have forecast doom and gloom for AWS in particular. I disagree with both arguments. While it certainly was the most significant failure of cloud computing to date, I predict this event will become not much more than a course correction and a “teachable moment” for Amazon, their competitors, all cloud architects and of course us here at The Conversations Network. For the geeks in the audience, I’m going to describe our architecture, the AWS services we utilize, and give a bit of an explanation about what happened and what we learned.

The Conversations Network utilizes three basic AWS services, plus a few more that aren’t really pertinent to this episode. Our servers are actually instances of AWS Elastic Compute Cloud (EC2) servers. The root filesystem for each server is stored in a small (15GB) AWS Elastic Block Storage (EBS) volume. Not only are these volumes faster than local storage, they’re also persistent. So if/when an EC2 instance stops, the root filesystem for that instance remains intact and will continue to be usable if the instance is re-started. [EC2 instances are booted from Amazon Machine Images (AMIs). In our case, these are based on Fedora 8 (Linux) customized to our standards. The AMIs are identical for all our servers, but the EBS root filesystems, which change dynamically once a server is booted, are unique to each server.]

We also use EBS volumes for non-relational storage. For example, we have one large EBS volume for IT Conversations and other podcast filesystems. This holds all the audio files and images used on the website. We have another for SpokenWord.org, and so on. These EBS volumes are each mounted to one EC2 instance, which in turn shares them with the other servers via NFS. Finally, we use the Relational Database Service (RDS) for our MySQL databases. Like EBS, this is a true service as opposed to a “box” or physical server.

One very important feature of EBS is that you can take snapshots at any time. For example, we make a snapshot each night of each EBS volume. We keep all snapshots of all volumes (other than the EC2 root filesystems) for the past seven days, plus the weekly snapshots for the past four weeks and the monthly snapshots for the past year. The cost of keeping a snapshot is based only upon the incremental differences since the previous snapshot, so it’s quite a reasonable backup strategy even for large volumes so long as they don’t have changes that are both major and frequent.

Designing any server architecture, cloud-based or otherwise, requires that you consider the failure modes. What can fail? What will you lose when that happens? How will you recover? Automatically or manually? How long will recovery take for each failure mode? It’s not about eliminating failures — you can’t really do that. Rather, it’s about planning to deal with them. And like traditional architectures, the cost of the configuration increases geometrically as you increase the reliability (ie, decrease the amount of time it will take to recover from a failure).

We’ve been using AWS for more than four years. During the period when IT Conversations was part of GigaVox Media, we were the basis of one of the first case studies published by Amazon. [Here’s a diagram of one of our AWS-based configurations.] Because The Conversations Network (a non-profit) runs on a shoestring budget and can’t afford the level of redundancy deployed by some commercial enterprises (eg, SmugMug), we’re not looking for a particularly high-reliability architecture. Until last week, we’ve have EC2 instances that haven’t stopped in well over a year. We can’t tolerate any significant loss of data so we need the redundant storage of EBS, but a 99.9% uptime is good enough for us, and that’s what we’ve had from AWS until now. Because of our experience with the high-reliability of AWS, we have never gotten around to automating the re-launching of EC2 instances in case of failure. We do use two separate monitoring services, and there are two of us (me and Senior Sysadmin Tim) who are capable of restarting servers, etc., if something does go wrong.

AWS operates in five regions around the world. We happened to pick US East in Virginia instead of US West (northern California) for no particular reason. Within each region there are multiple physical locations called availability zones. These are probably separate data centers within a metropolitan area. The availability zones within a region are connected by very high-speed fiber. This means you can have some degree of geographic redundancy by deploying servers in multiple availability zones, or achieve even greater protection by also deploying duplicate systems in multiple regions. The latter is far more complex, since the connectivity between regions is not as good as between availability zones. Our needs are humble, so all of The Conversations Network EC2 instances, EBS volumes and RDS databases are located in the us-east-1a availability zone. And of course, that’s where last week’s failures occurred.

Amazon hasn’t yet said what the original failure was. All of our EC2 instances were running and they could communicate with the RDS databases. I think the problem might have been the association between the EC2 instances and the EBS volumes. The volumes used as root filesystems were reachable, but not the others that contained our site-specific files.

After a few hours of downtime, I decided to re-boot our EC2 instances and that’s when things went from bad to worse. All of our EC2 instances entered the Twilight Zone. They were stuck in the “stopping” state. The operating system halted (no SSH access) but the servers didn’t release their EBS volumes. I could have launched all-new EC2 instances, but I wouldn’t be able to connect them to the volumes and hence, no websites.

Because of our backup strategy, however, we did have one more option: We had snapshots of our EBS volumes. I could have created all-new EBS volumes from the daily snapshots, and I could have done so in a different availability zone to get away from the problems. But there was one gotcha. We make the backup snapshots at 2am Pacific time each night. The failure occurred 19 minutes before that, which means our snapshots lacked the most-recent 24 hours of activity: new programs, audio and image files, logs, etc. As with the few previous problems we’ve had with AWS (mostly of our own causing) we thought this outage would be fixed quickly. It was a tradeoff: It seemed better to wait an hour or two rather than to re-launch with day-old data.

Of course “an hour or two” dragged on. Soon the outage was 24 hours old; then 48. It always seemed that the fix was imminent, so we delayed the restart process. Eventually, we decided to go ahead, and that’s when we discovered our one real mistake. Remember that we make snapshots of our EBS volumes every night? Well it turned out that we weren’t making those snapshots of all of our volumes. There was one volume that we somehow missed. The only snapshot we had of that volume was from the date it was created, more than a year ago. That means we would have had to launch our sites with some very old data. In this case, when we finally got access to the most-recent data (on the in-limbo EBS volumes) it would be difficult to reconcile it all. In the end, we decided just to wait it out. Finally, after 64.5 hours, the one EC2 instance that was holding hostage our last EBS volume stopped. We were then able to re-attach that volume to a newly-launched instance. We brought up all-new EC2 instances, attached all the then-current volumes and we were up and running, still in availability zone us-east-1a.

So what did we learn from all this? We re-learned that you have to think through these architectures carefully and understand the failure modes. But most importantly, we learned that once you have a good plan, you have to follow through with it. If we had been making nightly snapshots of that one remaining EBS volume all along, we would have been able to re-start the websites with day-old data at any time, regardless of the problems AWS was having disconnecting EBS volumes from running EC2 instances.

I also have a new strategy for deciding when to stop waiting for AWS to recover and instead switch to the snapshots: Once the length of the outage exceeds the age of the backups, it makes more sense to switch to the backups. If the backups are six hours old, then after six hours of downtime, it makes sense to restart from backups. In this case, we should have done that after the first 24 hours.

But we still know we don’t have ultimate redundancy: We still have to re-start things manually. So long as we accept the downtime, we can survive the total failure of the us-east-1a availability zone and even the entire US East region. That’s because all EBS volumes are first replicated to multiple availability zones within the region, and our nightly snapshots are stores in Amazon’s Simple Storage Service (S3), which is replicated across multiple regions. So our current data can survive a failure within a region and our day-old data can survive a failure of our entire region.

We still have a few things to cleanup and repair from this experience, but all-in-all we remain fairly happy with how things turned out. We didn’t, after all, lose any data. And while we aren’t proud that our sites were down for nearly three days, the world as we knew it did not come to an end. Maybe our team is even glad to have a few days off. (Too bad we couldn’t have told them in advance.) We still have one EC2 instance that refuses to stop, but it’s one of those that used NFS to reach EBS volumes attached to another server. Amazon says “We’re working on it.” Other than that, we’re now better prepared for the next failure, so long as its just like this one. Actually, I think we’re in pretty good shape for most events I can foresee. AWS. It continues to be a great platform for us.