USAir Flight 427, June/July 1999, Page 24

As reporters clacked furiously on their laptop computers and the victims' relatives--some wearing lapel buttons bearing photos of their lost loved ones--clenched their entwined fingers, Chairman Hall declared that his agency had determined that the probable cause of the accident was a mechanical malfunction of a small part in the airplane's rudder control system.

Had Hall, the other Safety Board members, and the NTSB staff not been so determined to take every possible measure to find the cause of the accident--and had ALPA line pilot accident investigators and staff not spent the previous 4½ years working diligently to help the Safety Board in its long and often frustrating search for the truth--the probable cause of the accident likely would not have been determined.

The Boeing 737-300 was being vectored to the downwind leg for a landing at Pittsburgh International Airport on the evening of Sept. 8, 1994. At 7:03 p.m., the airplane rolled to the left and dove into a wooded area near Aliquippa, Pa. The loss of control and dive from 6,000 feet msl (5,000 feet agl) took less than 30 seconds. All 132 persons on board died instantly.

Seasoned ALPA accident investigators were among the first to arrive on the scene.

Capt. Bill Sorbie (US Airways, now retired), then the Central Air Safety Chairman for the USAir (now US Airways) pilot group, and Capt. John Cox (US Airways) were driving back to their hotel in Pittsburgh after dinner at a local restaurant (see "Capt. John Cox, Air Safety Resource," August 1998). Their pagers went off at the same time, triggering the beginning of a long and horrendous night.

Since that terrible night, ALPA line pilot accident investigators and air safety volunteers, backed up by the Engineering and Accident Investigation Unit of the Association's Engineering and Air Safety Department, have spent thousands of hours and more than $781,000 of members' dues to investigate the crash of USAir 427. One of the advantages of belonging to the largest pilot union in the world is being able to garner such resources when required.

The ALPA team included US Airways Capt. Herb LeGrow (now retired), then the chief accident investigator for his pilot group; Capt. Dan Sicchio; Capt. Sorbie; Capt. Cox; Capt. John Brookman; First Officer Jeff Diercksmeier; Capt. Coby Johnson; First Officer Constantine Kleissas; First Officer Pete Lambrou; Capt. John D. Long; Capt. Don McClure (Eastern, Ret.), air safety coordinator for the USAir MEC; Capt. Tom Phillips; First Officer Steve Skupien; Capt. Robert Sumwalt; First Officer Paul Turnquist; and Capt. Bill Weeks. Capts. Sorbie and LeGrow retired during the accident investigation, passing the batons of central air safety chairman and chief accident investigator to Capts. Cox and Sicchio respectively.

The accident bore a disturbing resemblance to the fatal March 3, 1991, crash of United Airlines Flight 585, a B-737-200 that rolled to the right and dived into a park while on approach to Colorado Springs, Colo., killing all 25 persons aboard. After a long and frustrating accident investigation hampered by the limitations of United 585's antiquated flight data recorder (FDR), which recorded only 5 usable parameters, the NTSB was not able to determine the probable causes of the accident. The Board said the two most likely scenarios were (1) a malfunction of the aircraft's lateral or directional control system, or (2) an encounter with an unusually severe atmospheric disturbance, such as a rotor associated with mountain wave.

As in the case of United 585, the data from USAir 427's 11-parameter FDR was maddeningly inadequate. The NTSB and the parties to the USAir 427 investigation methodically eliminated a long list of possible causes. The focus of the investigation eventually narrowed to three potentially interrelated areas--aircraft performance, flight crew performance, and the airplane's rudder control system.

Investigators eventually concluded, through newly developed computer simulations, that the flight path was congruent with the airplane's rudder moving to the blowdown limits (the limit of travel of the rudder at a given airspeed, dictated by air pressure working against hydraulic pressure) and staying there, though the reason for the rudder movement remained a mystery. This new technology involved backdriving FDR data and ATC radar plots.

A question remained: Whatever the reason for the rudder movement, why couldn't or didn't the pilots simply roll the control wheel full right and stop the roll upset? The B-737 turns out not to have enough roll control to counter a full deflection of the rudder at some airspeeds and flap configurations.

Capt. Cox recalls "a sense of urgency" that permeated the accident investigation. After more than a week in the field phase of the investigation, he went home for about 24 hours and then went to Seattle for component testing.

The sense of urgency led to considerable flight testing of the B-737. ALPA pilots participated in flight tests of the B-737's "crossover speeds"--i.e., the minimum speed for a given flap configuration and bank angle at which a real or simulated rudder hardover could be countered by full opposite application of the control yoke. The flight testing revealed that, in certain configurations, the crossover speeds of the B-737 actually were higher than Boeing had previously believed. That discovery led to NTSB and ALPA recommendations for B-737 operators to increase maneuvering speeds for each flap setting.

"You could play with it," Capt. Cox says of the airplane's controllability during the crossover speed tests. "You could control the airplane roll with pitch, going from being cross-controlled but maintaining level flight, to the airplane rolling in the direction of the rudder hardover."

Capt. Paul McCarthy (Delta), ALPA's Executive Air Safety Chairman, says, "John Cox's great contribution to this aspect of the investigation was in working closely with US Airways management--which deserves a lot of credit--to develop new procedures (i.e., increasing maneuvering speeds) and in convincing line pilots to accept the procedures, despite almost total initial denial by Boeing and the FAA.

"At the time, he had just about no support from the manufacturer and the regulatory agency for the changed procedure. Now it's the industry standard--and the NTSB's called on the FAA to mandate the change for those few B-737 operators that haven't increased maneuver speeds."

(In March 1999, the FAA issued a bulletin recommending that B-737 operators increase, by at least 10 knots, block maneuvering speeds for 1, 5, and 10 degrees of flap.)

Capt. Cox also worked closely with US Airways management to test and validate a procedure to isolate a B-737 rudder jam--i.e., turn off the yaw damper and the B hydraulic system and activate the standby rudder power control unit.

The faster block speeds and the procedure to deal with a rudder jam gave pilots simple, specific, straightforward procedures to cope with whatever emergency the doomed pilots of USAir 427 had faced.

Determining what caused the rudder to move to the blowdown limit, however, continued to prove to be a daunting task.

At the NTSB sunshine meeting on USAir 427 in March 1999, a senior NTSB staffer summarized the most frustrating aspect of the FDR data from the airplane: Because the FDR recorded aircraft heading at a rate of once per second, two completely different scenarios--mechanical malfunction or pilot error--could be fit into the rudder time history (i.e., rudder position versus time) derived from the kinematic analysis based on a curve that was faired through the heading data.

The NTSB's analysis showed that a rudder hardover fit the heading data. Boeing favored the argument that available evidence showed that the pilots responded inappropriately to the initial upset after encountering a wake vortex from a B-727 a few miles ahead.

One could interpolate between the two most critical dots on the heading readout with either (1) a gentle curve, which matches the rudder hardover scenario, or (2) a swift zigzag, like a highly compressed capital letter N, to fit the Boeing hypothesis of inappropriate pilot response (i.e., wrong rudder input) to the wake vortex encounter.

If the FDR had recorded heading four times per second, instead of once per second, one hypothesis or the other could have been rejected early in the accident investigation.

Capt. Sicchio, who served initially on the NTSB's maintenance records group and then on the systems group, remembers well the occasion when the accident investigators first discovered that the power control unit (PCU) servo valve from the rudder control system of USAir 427 had a previously unknown failure mode.

The servo valve in the B-737 main rudder PCU is used to control rudder direction and rate of movement. The valve is a cylindrical unit a couple of inches in diameter and several inches long (see photograph, page 25). It consists of a pencil-sized primary slide that moves within a secondary slide that, in turn, moves within the servo valve housing. Through small grooves and holes machined into them, these slides direct hydraulic fluid to the rudder actuators.

The dual-concentric valve was intended to provide two independent, redundant pathways for properly directing hydraulic fluid to the rudder actuators. The valve depends on very tight tolerances--less than the thickness of a human hair-- rather than using O-rings to keep hydraulic fluid loss to a minimum.

Tests of the accident airplane's main rudder PCU servo valve, however, showed that if the unit was sufficiently cold-soaked, the secondary slide could jam against the servo housing without leaving any "witness marks" (gouges or other telltale marks). Moreover, the primary valve could move a little farther than intended within the secondary valve. If those two events happened together, the PCU servo valve could direct hydraulic fluid under pressure in the direction opposite that of rudder pedal movement.

"We were at Canyon Engineering in Valencia, California, for independent testing of the possibility that the valve could suffer a thermal jam," Capt. Sicchio recalls. "About a dozen or so people were in the room. I was about the only guy there who wanted to continue with the testing.

"The first time we got the unit from [Flight] 427 to reverse from the thermal jam, a couple of people in the room became rather upset. They felt the test was flawed because it looked jerry-rigged--the guy who set up the test used liquid nitrogen in a Coleman cooler to cool the valve. But he got the temperatures he needed. The valve from [Flight] 427 jammed, but a new unit didn't.

"Several weeks later, Boeing tested the valves in their much more sophisticated lab in Seattle. We got repeatable results. That was the big breakthrough we needed."

But it was not enough to show that the PCU servo valve had a previously unknown failure mode. Boeing contended that the temperatures weren't realistic.

"It would be hard to underestimate the contribution of Capt. Robert Sumwalt on the human factors group," Capt. Sicchio says. "The Boeing rep on the group continued to press tiny crew issues throughout the investigation. Robert, because of his background and training, was able to refute every one of them, every time. He was able to show factually that these claims were not credible.

"The Safety Board agreed," Capt. Sicchio continues, "and during the ‘sunshine meeting,' Dr. Malcolm Brenner, a senior human performance investigator in NTSB's Office of Aviation Safety, presented detailed and thoughtful arguments very much along the same lines to answer the Board members' questions about the pilots' actions."

Capt. Sumwalt recalls, "The Human Performance Group was part of the full field investigation, but by mid-October '94, our work pretty much had wound down."

In March 1995, however, Capt. LeGrow called Capt. Sumwalt to report that Boeing had convinced the NTSB to reopen the Human Performance Group to look at a lot of human factors issues.

"The situation was definitely intimidating for me. I felt like David going up against Goliath," Capt. Sumwalt confesses. "Everybody on the [NTSB Human Performance] Group except me had M.D. or Ph.D. after their names."

He is quick to add, however, "I don't think David could have gone up against Goliath if there wasn't a sincere commitment on the part of the Safety Board to make sure no stone went unturned in trying to find the cause or causes of the accident. The credit needs to go to Chairman Hall.

"I think the IIC [Investigator-in-Charge], Tom Haeutter, deserves a lot of credit. Dr. Brenner deserves a lot of credit for making sure we used every technique available--including some new investigative techniques--to find the truth."

Among those new techniques was a sophisticated analysis of the breathing and speech patterns of USAir 427's pilots as recorded on the CVR.

"We brought in an expert from Russia, Dr. Belan," explains Capt. Sumwalt. "He's been personally involved in more than 150 accident investigations. The Russians studied the breathing and speech patterns along with the groaning of pilots involved in accidents to determine their stress levels and specific muscular exertions."

Boeing's hypothesis, says Capt. Sumwalt, was, "‘Well, maybe these pilots just panicked and put in the wrong rudder.' We were able to show, through Dr. Belan's analysis, the stress levels the pilots were undergoing at each phase of the upset.

"During most of the upset, the pilots were in Stage I and II stress, which enhances performance due to the increased adrenaline flow to the brain. Not until a person gets to Stage III does performance decrease, and this crew clearly did not panic until the airplane was pointing straight down and death was imminent. So we refuted Boeing's theory."

Capt. Sumwalt and the ALPA team also had to address another theory--i.e., that the pilots were fighting each other for the controls. "On the CVR tape, we could clearly hear the first officer grunting and straining, but the captain wasn't," Capt. Sumwalt points out. "The captain had recently undergone back surgery, and was older and smaller than the first officer. So he couldn't have been fighting the copilot for the controls without straining and grunting."

In some ways, the USAir 427 accident investigation bore the fruit of previous years of ALPA efforts to improve accident investigation.

For one thing, USAir 427 was one of the first accidents involving pilots using boom mikes.

"I can remember [Capt.] Don McClure, some years ago, making a presentation at the annual ALPA Air Safety Forum," Capt. Sumwalt remembers. "He contrasted the high quality of a CVR recording of a British crew using boom mikes during an inflight emergency with the poor quality of a recording made with cockpit area mikes.

"ALPA wanted the FAA to mandate use of boom mikes to improve voice clarity for accident investigation. Eventually the FAA did so. But nobody imagined then that we would be able to use those high-quality recordings for speech and breathing analysis."

Capt. McCarthy credits Capt. Sumwalt with having the foresight to make the case for including human performance protocols in accident investigation as part of ALPA accident investigation policy--3 years before the USAir 427 crash.

"I think this business of ALPA being ‘the conscience of the airline industry' is so true," Capt. Sumwalt asserts. "If it hadn't been for ALPA, I think this investigation would have been wrapped up years ago--and probably not with the correct finding of probable cause. ALPA was almost alone in putting pressure on to keep looking for the real cause of the accident."

On the other hand, he adds, "I've always felt it's not ALPA's job to defend pilots at all costs--we're just here to make sure they get a fair shake. I think we certainly succeeded here in that respect.

"I have a library of 150-200 NTSB accident reports. A lot of them involve pilot error. So I went into this accident investigation with an open mind," Capt. Sumwalt says. "When we started into the breathing and speech analysis, I wasn't sure what we would learn."

He continues, "Not until the spring of '97, when I was trying to put together ALPA's submission to the NTSB, did the jigsaw puzzle come together for me, that, hey, this wasn't a pilot error accident! The speech and breathing analysis dovetailed so neatly with the kinematic study of the aircraft's performance.

"In the final analysis," Capt. Sumwalt concludes, "I think we put together the most credible explanation of what happened to USAir 427."

Capt. LeGrow wrote, "ALPA concludes that the accident was the result of a PCU secondary valve jam resulting in primary valve overtravel. The primary valve overtravel then resulted in uncommanded full airplane nose-left rudder movement. The flight crew was unable to counter this full left rudder due to insufficient lateral control authority available to balance the roll due to sideslip caused by full left rudder."

ALPA Recommendations

In its September 1997 submission to the NTSB regarding the USAir 427 accident, ALPA noted that, since the accident occurred, "the NTSB has issued numerous safety recommendations," and that "ALPA fully supports those recommendations."

The Association added, "With regard to the specific event that initiated the USAir 427 accident upset, malfunction of the main rudder PCU that resulted in uncommanded full rudder deflection, ALPA believes that Boeing and Parker [Hannifin] should work diligently to replace existing B-737 rudder PCUs with improved units as quick[ly] as possible without sacrificing quality.

"In addition, ALPA offers the following recommendations:

"1. The FAA should eliminate the current practice of derivative certification. Newly developed aircraft should be carefully evaluated against FAR criteria in place at the time of aircraft development.

"2. For aircraft [that] were certifi[cat]ed as ‘derivative' models, the FAA should evaluate those aircraft against existing FAR requirements and those aircraft, to the extent feasible, should be modified…to be in compliance with the current FAR regulations.

"3. The FAA should require all FAA-certified repair stations to meet all the standards of the original equipment manufacturer.

"4. [To] increase B-737 lateral control margin to an acceptable level, the FAA should mandate the development of additional operational techniques such as increasing B-737 minimum maneuvering speeds to Boeing-recommended ‘block' speeds plus 10 knots.

"5. The [airline] industry should continue with the development and implementation of ‘advanced maneuver' or ‘selected event' training and…the FAA should require the inclusion of this training in every airline's training program."

The Safety Board eventually determined that "the probable cause of the USAir Flight 427 accident was a loss of control of the airplane resulting from the movement of the rudder surface to its blowdown limit. The rudder surface most likely deflected in a direction opposite to that commanded by the pilots as a result of a jam of the main rudder PCU servo valve secondary slide to the servo valve housing offset from its neutral position and overtravel of the primary slide."

The NTSB also took the unusual step of including, in its official findings on the USAir 427 accident, a number of findings about United 585, which it had revisited in considerable detail, and the June 9, 1996, yaw/roll upset of Eastwind Flight 517, a B-737-200. The Eastwind airplane was in the Richmond, Va., terminal area, flying at 250 knots, well above the crossover speed; the crew regained control and landed without further incident.

The NTSB made 10 new recommendations to the FAA, in addition to many others the Safety Board had made during the course of the investigation.

* "Require that all existing and future B-737s have a reliably redundant rudder actuation system," and

* "Convene an engineering test and evaluation board" of government, industry, and academia representatives to conduct a comprehensive review of the design and certification of the B-737 rudder control system.

Effort on the last item has begun and is expected to take 1-2 years, involving experts from around the world. ALPA's newly created B-737 Flight Control Evaluation Project Team is participating. The team includes Capt. Joe Kohler (Northwest), a member of ALPA's Airworthiness, Performance, Evaluation, and Certification Committee; Capt. Skip Ostermann (Alaska); and Capt. Dixon Smith (United).

The NTSB's full 500-page report on the USAir 427 accident is or will soon be available on the Safety Board's website, www.ntsb.gov/ and the abstract of the NTSB's final report is available on ALPA's website, www2.alpa.org

Capt. Sicchio told his pilot group in the April 1999 issue of US Airwaves, the publication of the US Airways MEC, "For over four years, every action of our crew was under a microscope and attacked at every opportunity.… This investigation was the most extensive that the NTSB and our pilot group have ever been involved with.… Throughout this process, one constant has remained: the dedication of the ALPA team."

Turning to the safety of the B-737, Capt. Sicchio concluded, "The industry has taken many steps over the past two or three years that should allow us to continue safe operations for the short term. These include the modified PCU, which will eliminate the failure responsible for the USAir 427 and United 585 accidents. The increased maneuvering speeds and Quick Reference Handbook procedures will effectively deal with other rudder anomalies.

"In the long term, however, we agree with the NTSB that the B-737 must have a ‘reliably redundant rudder activation system.' ALPA will continue its work with the NTSB, Boeing, the FAA, and others to ensure our long-term goals are met."