The not so boring life in the shipyard

I also worked with ships in overhaul in preparation for the examination either by Naval Reactors or the NPEB prior to initial criticality. The challenges of achieving readiness in the shipyard were much different from those at sea: Most equipment could not be operated. The shipyard was reluctant to provide adequate time for the crew to practice since any crew practice time took away from overhaul work time. The schedule date for the examination was set early, but the shipyard progress frequently did not support that schedule.

This situation frequently became a conflict in which the shipyard wanted to keep the date by shortening the crew training time and I became the bad guy who said no. If the crew did not get adequate training to my satisfaction, I would cause the examination to be delayed that would delay the entire overhaul schedule. I was usually successful in working out a plan to support the crew training without significant impact on the overall schedule.

I also worked closely with the tender commanding officer in preparation for the annual radiological controls evaluation (RCPE) by the NPEB. This was a totally different set of challenges. The Radiological controls division was a relatively small part of the tender crew. However, the impact of the RCPE extended to all parts of the tender in responding to radiological events and drills and conducting nuclear repairs and operations.

One of the classic challenges was connection of an off hull radiological discharge line to a submarine alongside. This action required divers to follow a precise procedure. For divers whose very nature is to improvise, the necessary discipline was a challenge. Radiological spills and other releases were also a challenge since they affected members of the crew who had minimal radiological training. They were also challenging since many tender radiological controls technicians were female and the first step in a personnel contamination event is to remove the contaminated outer clothing. This challenge was handled by declaring training days as swimsuit days. In the actual casualty situations, which fortunately were few, everyone did what they had to do in response to the casualty.

I also got involved in material issues that were high risk or threatened the operational schedule. In one case, during an inspection inside a contaminated reactor plant system, several errors and missteps resulted in serious radiological contamination on the Pier. Contamination on a smooth deck is not hard to clean up. Contamination on a concrete pier is a different challenge. The level of the contamination was much greater than we usually experienced since the source was from inside the reactor plant system. Decontamination techniques included vacuum cleaners, latex paint, and sticky tape. Those methods were generally successful, but a few spots remained in which the contamination was embedded in the concrete. For these few spots, the tender invented a tool consisting of a plunger from a plumbers helper with a hole cut to accommodate a needle gun with a vacuum cleaner connected to the top. The tool was placed over the contamination hot spot, the vacuum started, and the needle gun activated to dislodge the concrete. The pier was successfully cleaned up and there were lots of lessons to be learned both on how not to cause the contamination and how to clean it up.

Another challenge was when I was confronted by a well-meaning civilian who wanted to conduct some significant testing of elements of the electrical distribution system aboard a submarine preparing for deployment. The testing was not routinely conducted on attack submarines but was a routine preventative maintenance process for SSBNs. When queried, the civilian technician acknowledged that the tests would probably show issues with the system and furthermore, he did not have the parts or the resources to fix whatever he found. His goal was to gather data. My goal was to deploy the ship, which had passed all of the distribution and breaker tests that were required. Our goals conflicted in that I knew that when he identified weaknesses in places we were not normally required to look, the deployment schedule would be impacted. The question was whether this was a safety or a reliability issue.

Since these were tests not previously required and since they were for reliability of the SSBNs, I concluded it was not a safety issue that should be considered before deployment. I also knew that if issues were identified it would be very difficult to continue with the deployment without resolution and it was clear that the organization requesting the testing was not prepared to quickly resolve issues they identified. Thus with the concurrence of the squadron commander, I refused to support the testing on that ship and suggested it would be more appropriate to accomplish on an SSN preparing for overhaul during which any reliability issues could be resolved without impact on the operational schedule. There were some unhappy technical folks in Washington from where the test request had originated, but we held firm and the ship deployed on time. I got a call to Washington to improve my understanding, but in the end I think the understanding of others was broadened.

There is work and then there is paper work…

A final anecdote from my time as a squadron deputy concerns dealing with a longstanding problem aboard Lipscomb. Lipscomb had the unique turboelectric propulsion plant. An issue since construction was low electrical grounds in the electrical propulsion loop caused by oil leakage across the bearing seals. During the first few years of operation, many minor modifications were made to the oil seals to minimize the leaks by the construction shipyard, Electric Boat Company in Groton. Lipscomb’s homeport was in Squadron Two in New London. Lipscomb’s first shipyard overhaul was conducted at Norfolk Naval Shipyard and her homeport was changed to Norfolk. Shortly after completing overhaul, the grounds in the propulsion loop became a significant problem. Investigation revealed that the modifications made by Electric Boat had not been properly documented on the ships plans and that Norfolk Naval Shipyard had removed all of the modifications under the rationale that they were not per plan. This was a classic and expensive example of the adage that the work is not done until the paperwork is completed. In order to correct the grounds, the propulsion generator rotors had to be removed from the ship which required a hull cut in the engine room. The repairs would require several months.

One of the rules of the nuclear propulsion program was that if the reactor plant was not operated critical for over six months, a complete requalification of the operators and an NPEB examination was required. Another rule was that the reactor could not be taken critical if containment of the hull could not be established. With a hull cut, containment was not possible. The obvious solution to the first issue was to conduct reactor plant operations alongside the pier prior to the six-month cutoff. However without containment that was not possible. Replacing and then removing the hull patch was prohibitively expensive. However, we determined that we could claim containment with a lightweight metal sheet welded over the hole in the hull from which the hull cut was removed.

After lots of forceful and sometimes heated discussion at various levels of command including Naval Reactors, it was agreed that we could establish containment with a temporary plate and maintain operator proficiency with critical operations alongside the pier. The operations had to be meaningful from a proficiency perspective including drills and exercises, they had to be planned, they had to include all operators for whom proficiency was required, and they had to be monitored. It was particularly satisfying to develop this solution and navigate the various levels of critical review and decision to get it to happen. Ultimately, the propulsion generators were reinstalled, the hull patch was replaced and Lipscomb resumed her operational schedule. This significant disruption to the ship’s schedule was because the ships plans had not been kept up to date as the original oil leaks were resolved.

From Squadron Command to NPEB….

My two years as deputy squadron commander passed quickly. As I had experience as squadron engineer, the tour was enjoyable in that I got to deal with the big challenges and could pass on the mundane. That was not the case as commanding officer when every issue required attention. As I was completing my tour, I was informed that my next challenge would be as a Senior Member of the NPEB. The good news was that we would remain in Norfolk and the family would not be asked to move. It was also good news for the family because while I would be away a lot doing examinations, when I was home, the pace of work was less, few crisis were ever passed to the NPEB and thus the opportunity to plan family time and participate in the life of the boys growing up could continue as it had while a squadron deputy.

The NPEB is part of the fleet commander staff. The technical direction for the NPEB is provided by Naval Reactors and the authority of the NPEB comes from the fleet commander. The task of the NPEB is to conduct examinations of the nuclear activities within the responsibility of the fleet commander which includes both reactor operations and radiological controls in fleet maintenance organizations. The NPEB does not conduct radiological evaluation of shipyards which are evaluated directly by Naval Reactors Staff personnel.

While I was part of the NPEB, every operating nuclear ship crew was scheduled for an ORSE once a year. Each maintenance activity either tender or submarine base repair facility was scheduled for an RCPE once a year. For ships in overhaul that had not refueled, the NPEB conducted a Pre critical operational reactor safeguards examination (PORSE) prior to the first criticality at the end of overhaul. When an ORSE resulted in a failure, repeat ORSE was required before the ship could resume unmonitored and unrestricted operations. In situations where the ORSE or RCPE was passed but significant issues identified, a grade of significantly below average was assigned and another ORSE or RCPE was required within six months.

An ORSE or RCPE is conducted by a team of four—One Captain post command officer and three post engineer officers. In the case of a multi-reactor ship such as a cruiser or aircraft carrier, two teams are required to conduct the ORSE. The ORSE is conducted in two days and one night aboard. An additional day is required for the multi-reactor ships.

Needless to say, it is an intense period in which records are reviewed, each watch section is exercised with drills and evolutions, and selected nuclear trained officers and enlisted operators are interviewed. The ORSE is conducted at sea from the ship’s homeport or at the location where it is deployed. Thus we routinely traveled to each port on the east coast, Holy Loch Scotland, various points in the Mediterranean Sea and infrequently to places like Azores, Bermuda, Gibraltar, and Diego Garcia in the Indian Ocean. We all became frequent flyers on Pan Am and TWA. Unfortunately the perks did not survive their bankruptcy!!

To save on travel funds and wear and tear on our bodies, we frequently scheduled multiple ships in sequence on a single trip to Scotland or the Mediterranean. On occasion, this scheduling resulted in several days between examinations which provided opportunities for wives to join us for short periods of vacation or we just hung out and killed the time between exams. I had the opportunity for Mary to join me on three occasions in England and Scotland which were enjoyable times for both of us.

I had not been on the receiving end of the NPEB since I was engineer on Dace. It was always a stressful but professional experience. I noted the ever-increasing expectations that the NPEB brought aboard. As a result, the capability of the crews on the nuclear ships to combat casualties and system upsets had advanced significantly. When I was engineer, we were not capable or did not understand many of what had become routine responses to failures and upsets by the time I reported to the NPEB. The ability of the ships to respond to the unexpected significantly enhanced the safety of the ships and made responding to the complex or complicated casualty routine. Had the crew of Thresher had the operational proficiency and flexibility I routinely saw on the ships I examined, it is possible they would have saved their ship.

Each NPEB exam was assigned an overall grade: excellent; above average; average; below average; significantly below average; and fail. It was our goal to have a standard distribution of grades with about 10% excellent. As a team, one of our challenges was to find the crews who deserved an excellent overall grade. It was easy to look at the deficiencies we identified and conclude this was not an excellent result, but the question was whether they were in the top 10%. Normally, an excellent ship and crew usually stood out if one was looking with an experienced and objective eye. The lesser grades were usually self-evident.

A failed result could only be given when reactor safety violations were noted. Thus by definition, an RCPE could not have a failed grade. During my two years on the NPEB, I participated in 93 exams. Most were routine but a few were memorable: I only participated in one failure. Not only were there reactor safety violations, but it was our observation that the ship training organization was unable to detect and control the drills to ensure reactor safety would not be jeopardized and operator proficiency enhanced. Thus, not only was the crew not proficient, the training organization gave no confidence that they could provide and oversee the training to improve the crew proficiency. We obviously got the attention of the ship as well as the staff because the next graded ORSE was graded as excellent. In another case, the first ORSE on a newly commissioned ship went very poorly. It was clear the crew and in particular the senior officers had not become self sufficient after leaving the comprehensive technical input and oversight of the shipyard. It was also clear the ship was not proficient at some of the less frequent operations and evolutions. Due to some inappropriate operations during a drill period, it was necessary for the ship to return to port to resolve the issues.

The NPEB challenge was how to deal with this unfortunate set of circumstances and lack of demonstrated proficiency on the part of the crew. Since we were in port during the final portion of the exam, it was possible to discuss the situation with Naval Reactors and the Type Commander. Note… lots of acronyms appearing here…. Hope they are explained. Am not checking for that. Remember…. The reader does not remember this stuff. After much discussion, we determined that the ship could safely complete their operations before returning to the shipyard for a post commissioning shipyard availability (PSA) and that since the ORSE grade was significantly below average (SBA), a PORSE would be required prior to exiting PSA.

Several other SBA results were also very difficult judgments as to whether the ship could continue to operate while gaining improvement or whether they should be taken out of service for focused retraining in safe reactor operations. The excellent results were clearly the easiest and most satisfying. I think the team got almost as much pleasure from finding an excellent result as the crew that received it. Only one RCPE was notable for the challenge of a result. The crew was not proficient to conduct expected evolutions such as discharge hookups or responding to expected casualty situations, and many routine maintenance items had not been accomplished on schedule to ensure that the results of radiological measurements were accurate. Since by definition an RCPE cannot be a failure, this result was graded as SBA with significant critical comments.

My final challenge was the debrief of the results at the end of the RCPE. The debrief was attended by a two star Admiral and a three Star Admiral and they were not happy. My concern was whether they would "shoot the messenger", but fortunately, our facts were sufficiently persuasive that they understood the result and took the appropriate corrective actions. Another challenge was the conduct of an ORSE at a Naval Reactors land based nuclear training prototype. Prototypes were not within the responsibility of the fleet commander and thus not our responsibility to examine.

Admiral McKee who had replaced Admiral Rickover as head of Naval Reactors made a request to the fleet commander that we be directed to conduct an ORSE at the prototype. The rationale was that we understood the capabilities and expectations of the fleet to conduct nuclear operations and that Admiral McKee wanted to benchmark his prototype against that standard to ensure his training pipeline was delivering nuclear plant operators with the proper knowledge and standards. He also wanted a check that nuclear safety was not being jeopardized during prototype reactor plant critical operations.

Adding to the challenge, the prototype was managed and operated by a civilian contractor for the Department of Energy (DOE). Many of the staff operators of the prototype were officers and petty officers who had fleet experienced and returned to the prototype for shore duty. They all had excellent records and experience as nuclear operators. DOE had a comprehensive set of access and information security rules that were different from the ones we usually followed aboard ship or on the base. The civilians did not appreciate or understand our process and did not understand the timeline we followed for the ORSE. However, all of the uncertainty was worked out on both sides and the ORSE was accomplished.

During the operations, there were no trainees in the plant, only staff, some navy and some civilian. Our entire focus was on the staff. We did not examine or evaluate the trainees. The result was interesting in that the operational proficiency of the staff was not at the level we expected. We also noted the continuing training program for the staff did not meet the minimum expectations we had for the fleet. The civilian management was not particularly happy with our results but took action to improve. The conduct of an ORSE at one prototype became an annual event which improved the performance of the prototypes in developing operators for the fleet who had a better understanding of what to expect when they arrived in the fleet. Sometimes our schedules and travel uncertainty became a challenge.

The NPEB nominally had four teams with four senior members. At one point we had only two senior members to support the four teams and the workload that required four teams. As a result, I had a schedule over a two week period from Norfolk for one ORSE in Kings Bay Georgia, to Holy Loch Scotland for two ORSEs to Lisbon Portugal and on to the Azores to meet a carrier for an ORSE, debark the carrier in Bermuda and fly to Norfolk Naval Shipyard for a PORSE on a cruiser and finally home to write the reports.

On another trip we were to meet a carrier in Gibraltar as it completed a deployment. When we arrived in Gibraltar to meet the carrier, we were told to board the C141 transport that was waiting on the runway which immediately took off for Sardinia, Italy. At the air station in Sardinia, we boarded a carrier propeller driven plane and headed farther east. We landed on the carrier in the extreme Eastern Mediterranean where it had been diverted to make a high priority personnel transfer. Since the carrier was supposed to be headed across the Atlantic when we came aboard, it started a high-speed transit across the Mediterranean during which we conducted the ORSE.

Two flag officers aboard were a bit irritated when progress was slowed as we forced one or two reactors off line for drills, but the carrier maintained the planned schedule and we accomplished our operational ORSE schedule. A final note: During the ORSE, I spent time with the engineers reviewing their training records and their management of their department. Most were very good and nearly all were adequate to the task. However, as my time away from the NPEB increased, only two continued to standout in my memory for their excellence. Interestingly enough, 20 years later, they are both Four Star Admirals still on active duty.

Read part 8

Read other articles by Captain William Hicks