Note: This is a true story, although I’ve elected withhold the name of the company I visited. It contains my opinions, based on my observations. It is totally subjective, and should be taken as such. It is in no way meant as derogatory commentary on the people or organizations mentioned herein. It is, more than anything, a commentary on myself by way of my own experiences and my reactions to them. The opinions expressed herein are mine alone, and are based on over 30 years of engineering experience.
I believe that anyone who is deeply involved in their career, working on high-pressure, high-profile projects, eventually comes to a point where they decide that they probably can’t take it anymore. Something’s got to give, or break. They want to run away screaming into the night, but they know that they can’t—someone has to help pay the mortgage and put food on the table. So they start looking around to see if there might be some greener grass elsewhere. Or they start to quietly go insane (at least temporarily) and stand helpless as their personal lives fall apart. I’ve watched this happen to colleagues more times than I’d care to recount. I myself had a “fight-or-flight” type of experience such as this during my time on the Phoenix Mars lander mission, but fortunately I have a spouse who understands the pressures and demands of my career, and she has on more than one occasion offered sage advice and even some alternate job suggestions that turned out pretty well. She’s also been known to come into the lab and stare down the project manager, and then dragging me home after I’d been working for over 48 hours straight.
So it came to pass that I was offered the chance for an interview with a small company near Seattle that was attempting to build sub-orbital spacecraft. It was an interesting experience, to say the least. The Phoenix lander was still about a year away from launching, but I felt I had some good reasons to start exploring my options (one day I’ll write a book about the Phoenix experience, but not just yet).
I arrived at the SeaTac airport and picked up the rental car that had already been arranged for me. The next morning, after guzzling some free hotel coffee, I proceeded to find my way to the company’s facility. It was still very early, but this was supposed to be an all-day affair, so an early start was deemed necessary.
The building itself was a nondescript converted production facility of some sort, all clean and freshly painted, and except for the people sitting at desks on an upper level, it appeared to be mostly empty. The ground floor lobby was a large and impressive open space with various static displays of sundry space hardware sitting around. I had to tear myself away from an actual Reaction Motors XLR11 type rocket engine sitting on a display stand. I knew what it was, and where it had been used (the X-1 and X-15 programs) but I’d never been able to actually reach out and touch one before, I had only seen them on display at museums, safely out of reach of curious folks like myself. It was almost a religious experience as I admired the fine workmanship, ran my fingers over the fuel feed lines and felt the smooth curves of the thrust tubes under my hand.
Then I looked up toward the upper level where the engineers and programmers sat in an open space arrangement (no offices, no cubicle walls, just desks, shelves and chairs in a semi-organized Silicon Valley fashion). That’s when I saw the full-size replica of a Victorian era “spacecraft” suspended from the ceiling at the top of the stairs. It was like something right out of a Jules Verne novel. When I went inside I was speechless. Everything was real: the gauges, brass levers, copper tubing, and even the period upholstery. There were even authentic period photographs in frames mounted to the walls inside the cabin. It was an amazing object that must have cost a small fortune to build. I was told that some of the people there liked to use it as a quiet place to discuss ideas or relax for a bit.
Then the actual interview process started, and various people took turns asking me questions. All in all, I think I was interviewed by about ten different people, ranging from software developers to control systems engineers.
In an effort to be up-front and honest I had told them earlier during the preliminary phone interview that I wasn’t all that interested in GUI-based end-user type applications, and that the bulk of my expertise lay in embedded systems design and programming. This turned out to be a mistake, as later during the interview process one of their Java GUI gurus took me to task for it. I got the impression that he felt that the embedded stuff was easy, and that the real art lay in the control and data display interfaces. That was unfortunate, from my viewpoint, as I was still heavily involved with the Phoenix mission at that time, and I had come off of some intense aerospace projects just before Phoenix. I knew what was involved in designing and implementing robust and safe embedded control systems, and I had even taught it, but I don’t think he really understood the distinctions between the two programming paradigms.
Later on one of their engineers announced to me that they felt perfectly comfortable with letting a tool generate the guidance control system code for them. I asked if they were even going to review and test the output, and the response was something to the effect that “we’ll look it over, and then we’ll test it in a fixture out on the floor. We don’t need to unit test it—we trust that it works.” Upon hearing that I thought to myself: Well, OK then, I guess I won’t be going for any rides on this thing. I have a hard time trusting anything generated automagically by a tool, and in the world of safety-critical aerospace software, unless the tool itself has been tested and certified to same level as that required for the software it is generating, then the output must be reviewed, tested and verified by a human being before it can be used. Even if the tool is certified, I would prefer that the code it generates be thoroughly inspected and tested long before it ever gets close to the real hardware.
Then there was the programmer who informed me that they didn’t plan to worry about meeting safety-critical testing and verification requirements such as those defined in DO-178B (the official software development guidelines mandated by the FAA for safety-critical avionics software). He claimed that the safety and reliability could be demonstrated by code inspection and some testing with fixtures and jigs. I had to catch myself from showing my shock at this statement. I also had to wonder if they knew that, at some point, the FAA was probably going to get involved and they were probably going to want to see the software and review the testing records. When I mentioned that the FAA was then conducting industry discussions to help determine the rules and regulations regarding things like software on private spacecraft, all I got back was a blank stare.
Finally, I should mention the fellow who arranged for the interview and acted as my guide throughout the day. Although he wasn’t an engineer, or even involved in the engineering management, he was a rather loud fellow with a Texas drawl who apparently thought that he ran the place. I found out that his background was in banking and finance, and that he has been involved with an earlier, and now defunct, private space venture.
After the interview ended later that day I had some time to kill before my flight back to Arizona the next morning, so I drove to the South for a ways, stopped to take some pictures of Mount Rainier off in the distance, and thought long and hard about what I’d seen and heard. Was it worth walking away from the Phoenix mission for? Although I was very unhappy with my job on Phoenix (80 hour work weeks, ineffectual management, constant harassment from JPL, and other issues), I knew that leaving now would mean that I wouldn’t be able to really claim anything about my experience with Phoenix other than that I had been there for a while, and it was obvious that as the new guy at this company I wasn’t going to be able to make much headway against the established mindsets. At least with Phoenix it was ultimately up to me as to whether the spacecraft’s imaging software worked or not, but only so long as I elected to stay with it and make it work.
But more than anything, I think that what really clenched it for me was how I couldn’t shake the impression while in the company’s building that I was walking through the offices of a software start-up, not an aerospace company. I’d been in the production facilities for the huge wing sections of the space shuttle and worked on the automated machine tools that fabricated the wing spars. I had walked through the Vehicle Assembly Building at the Kennedy Space Center and talked with engineers from McDonnell-Douglas and Rockwell, and I’ve seen the wispy clouds that form inside the building up near the ceiling, too. I had walked under, and on, the huge transporter crawler used to haul the Saturn V rockets out the launch pad. I had been in many other facilities as well, such as NASA’s hangers and labs at the Dryden Flight Research Center at Edwards AFB, the spacecraft assembly and test areas at Lockheed, and I had sat in the jump-seat of a Navy LCAC and operated the turbine engines while it hovered just inches above the concrete in order to do tests on the digital engine controllers. This company didn’t feel like any of those experiences, or any of the many other experiences I had accumulated during my career to that point. I realized then that I needed to stick with Phoenix, no matter how much it hurt, and see it through to launch. I struggled with how I was going to explain to the loud guy when he called for my decision the next morning that I wasn’t going to accept the job offer.
Finally, I realized that I could sum it all up in one short sentence: It didn’t smell right.
There was no smell of stale fumes from soldering and welding, no smell of cutting oil from machine tools, no smell of epoxy or paint and no smell of hot laser printers busily cranking out reams of reports and test data. Even the static displays in the lobby had been cleaned to the point where any lingering smells from their origins and history was gone (take a good whiff as you walk through the museum exhibits at KSC, Edwards AFB, Pensacola Naval Air Station, Wright-Patterson AFB or the Pima Air and Space Museum and you’ll know what I’m referring to). The place smelled like a sterile office, not a place where people built rockets, and the people working there that I met, with the exception of a couple of the more senior people, struck me as software contracting types from LA and San Jose with very limited hands-on experience with aerospace hardware. They seemed like a bright bunch, and they were undoubtedly well funded, but they were not the “Steely-Eyed Missile Men” I was used to working with (the steely-eyed folks might have been total jerks on more than one occasion, but that didn’t change the fact that they were generally some of the sharpest people I’d ever worked with).
While saying that it didn’t smell right was, in my opinion, the most succinct and accurate summation I could come up with, I know that the loud guy didn’t get it—it was just a little too Zen for him. But it was all I felt like saying about it, although he did spend a few moments on the phone futilely prying and poking for more. So, as I flew back to Arizona I started to think about ways to make changes in my own life so that I could survive the Phoenix experience. Well, I did survive it, and it would appear that Phoenix worked quite well and took a lot of pictures while on Mars. I didn’t stick around for the “landed” part of the mission, since I really didn’t have any interest in taking pictures of rocks, and JPL had decided to toss most of the work that I had done earlier with the instrument command sequences into the trash in favor of their own processes and tools (like Maslow once said: If your only tool is a hammer, every problem looks like a nail). Fine, they could have it, I’d done my part.
Do I regret not taking the job? No, not at all. In retrospect I’ve come to realize that I would have been just as miserable there as I was with Phoenix, but for different reasons. I do wish the folks in Seattle success in their endeavors. All in all, it was a fascinating experience and I’m grateful for the opportunity to have had it.
It also taught me, once again, that thinking long-term can make the short-term agony more tolerable, and making choices is more than just reflexively jerking away from the heat. Sometimes you’ve got to leave your hand in the fire for a bit and just endure it in order to grasp the prize.