Too Big To Fail?

March 10, 2009 at 12:15 pm | Posted in Economics, Technology | Leave a comment

I’m sure that most of us have been hearing the phrase “Too big to fail” tossed around rather blythely as an excuse for injecting colossal amounts of cash as “stimulus” to keep various corporations able to pay their employees and creditors.  The thinking here is that it is cheaper to the US Government to give a particular company billions of dollars in bailout money, rather than suffer the loss of tens of thousands of jobs and the resulting tax revenue should the company be forced to close its doors.  The corporation is such a giant that should it happen to fall to the ground, it will flatten a huge swath of the population.  This puts the government in a reactionary “fireman” mode, squirting cash out their hoses at any sign of fire.  It’s definitely not a good way to run a railroad.

Railroads make me think of engineers, and engineers make me think of the first couple jobs I had out of college.  I graduated with degrees in electrical engineering and physics, but in my first job, I was given the title of “Systems Engineer”.  Not knowing anything about the field, I started to look into it.  Essentially, Systems Engineering is the study of the management of complex systems.  However, in my firm’s case, “Systems Engineer” was just a way to indicate an analyst with an engineering degree.  I never needed to do any real systems engineering since my job consisted mostly of putting together reports cribbed from other reports and supply catalogs.

A stint at a tech startup gave me a greater appreciation for systems engineering, mostly due to the lack of it.  Our flagship weather station product made a number of trips to and from customer sites until we began to understand all the myriad ways the sucker could break and take steps to remedy them.  We had spent so much effort getting the thing to work that we had not paid much attention to getting it to not fail.  Eventually, we developed some rules of thumb to include parts that, though they added to the cost, prevented more expensive replacement of units down the road.  This included everything from surge suppressors to protect against nearby lightning strikes to blinky lights that would help the customer diagnose problems over the phone.

Later, working for an engineering consulting company to the medical and laboratory industries, I was introduced to a more formal systems engineering process.  One of the mandatory steps of every design was a Failure Mode Analysis.  In the FMA, we needed to document the system’s respose to various environmental hazards (overvoltage, undervoltage, loss of power) as well as the effect on the system for each component that might fail.  At first, I thought that this was pretty muc a waste of time, since in almost all circumstances, the result would be an inoperable system.  But, they told me of an automated blood pressure cuff system that they had designed in the past.  One day, it occurred to them that if the valve controlling the cuff should jam, not only would the system be inoperable, but the poor patient would be stuck with his arm in the machine.  Thus, the single valve became Too Important To Fail.  Their solution was to redesign the system to include a second valve in parallel.  It added cost, but it prevented a more catastrophic problem down the road.  Ironically, the company went out of business when its singlemost important client cancelled their contract.

Speaking of going out of business, this brings us back again to the current state of the American economy.  We seem to have a number of components that are Too Important To Fail.  Not only does their failure render them inoperable, but the collateral damage from their failure is considered to be downright dangerous to the general population.  While the government seems to be busy trying to put out the fire that may reduce the country to ashes, I’m looking ahead to the rebuilding phase.

So, what to do?  I’m taking my inspiration from the design of computer servers.  A server is a computer that is accessed by multiple users and applications simultaneously.  Unlike a desktop, whose failure may inconvenience a single person, a server has the capability to shut down an entire department or building should it go on the fritz. 

One of the things that surprised me when I started working at my present employer was that, to the casual user, a server functioned just like any other personal computer.  The screen was the same, the icons were the same, and the little animated cursors were even the same as the computer on my desk.  This led me to question why someone would pay up to ten times as much for a rackmounted server instead of a cheap desktop, especially when they were capable of pretty much the same operations.  However, when I finally had a chance to take one apart, I learned that there were three important principles that separated servers from desktops: quality, redundancy, and visibility.

Quality  The first thing that is noticeable about a server is that it is heavy, much heavier than a desktop.  This is because the case is made of thick steel, reinforced by solid steel brackets.  I’ve had to repair desktops after shipment, but never a server.  And, unlike many desktops, the interior of a server is well labeled and accessible, with top-notch brand name components in easily replaceable locations.  A server will have the maximum number of connectors it can to allow for complete upgrades.  In short, no corners are cut to ensure that the system is reliable and maintainable.

Redundancy  The next thing one notices poking around the guts of a computer server is that it seems to have many more parts that necessary.  It has multiple hard drives, multiple processors, multiple network cards, and perhaps multiple power supplies.  This is so that should one of these parts fail, the duplicates can shoulder the load and continue on until the broken part can be replaced.  In fact, most components are “hot swappable”, which means that they can be replaced while the system is still running.  Try changing the tires on your car while driving down the highway….

Visibility  After the case is sealed up and the operating system is booted, the third thing one might notice is that there are a lot more icons and gadgets on the screen.  These are monitors and controls for the various systems in the machine.  through these extra programs, one can inquire as to how hot the processor chips are, how busy the hard drives are, even how fast the various fans are spinning.  But, there is no need to sit and watch.  If something should go wrong with a fan, hard drive, or interface, a notice will instantly pop up on the screen.  Many servers even have an LCD panel on the front, since most servers don’t even have screens.  Also, everybody and his brother will be sent a detailed message via email describing exactly what went wrong and maybe even how to fix it.

So, if uninterrupted and failsafe operation is important, a company would gladly pay ten times extra (or more) for a computer system with quality, redundancy, and visibility built in.  Now, why won’t we invest the same care and forethought into our national economy?  Why don’t we have firms investing in solid, documented business opportunities, instead of pipe dreams or comlpex scams?  Why do we allow a few firms to dominate the business landscape, so that risks are concentrated into a small sector?  Why do we still have a “shadow economy”, in which the warning signs appear well after it is too late?

Seems to me Obama should dump his economist friends and hire a couple of good systems engineers.  There should be a few looking for work these days now that the manufacturing industry is being shredded….


Leave a Comment »

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Entries and comments feeds.

%d bloggers like this: