Test and Evaluation (T&E) have a critical role in the risk management process. Stakeholders apply T&E in assessing and mitigating risks when working with systems and processes. The T&E procedure is native to the Systems Engineering Process (SEP) but has since received an application in a variety of non-engineering sectors. The strategy is crucial in uncovering the level of performance and correcting any deficiencies within a system (Dahmann, 2012). Through T&E, the project manager is well-suited to learn the technical and operational limitations of the production process. Besides the limitations, the stakeholders of a project are well-placed to learn and appreciate a system's capabilities. The National Security Cutters (NSC) is a prime example of a project that has been managed using the T&E process. The NSC forms a significant part of the Coast Guard's fleet and can undertake challenging operations such as defense missions and maritime homeland security. The United States Navy conducted the initial testing for the NSC in April 2014. The Government Accountability Office (GAO) conducted a follow-up study and provided the recommendations. In both cases, the US Navy and GAO demonstrated the ability of T&E to uncover capabilities and deficiencies within an NSC project.
Background
The NSC is a group of technologically advanced cutters developed to serve security purposes for the Coast Guard. Unlike the previous ones, the cutters would perform some of the most difficult security operations ranging from maritime homeland security to marine missions. The program was launched as part of a process to replace the old high endurance cutters that had existed on the coast since the 1960s (Mackin, 2016). Compared to the old cutters, the NSC came with a wide variety of advantages. First, they have better transit speeds, better endurance, and a greater ability to recover small boats (Mak, 2017). The cutters have an automated weapon system and a large flight deck. The advanced sensors are also critical in providing it with the much-needed intelligence characteristics. Like any other security program, the project was subject to tests and evaluations to detect the capabilities and limitations. The US Navy conducted the first T&E while GAO performed the subsequent one (Mackin, 2016). Therefore, the two processes emphasized the need to test and evaluate programs for their suitability and likelihood to achieve the set objectives.
Delegate your assignment to our experts and they will do the rest.
The NSCs sought to replace the high endurance cutters and guarantee specific capabilities that the former cutters could not provide. The new NSCs have the power to gather, analyze, and share classified information. They can also recover stray aircraft and launch cutter boats, among other sophisticated functions (Mak, 2017). The NSC will conduct the Coast Guard mission, perform drug interdiction, enable defense readiness, and conduct general law enforcement duties. The project sought to replace the 12 high endurance cutters with 8 NSCs (Mackin, 2016). The advanced capabilities and the delicate duties performed by the new cutters calls for an effective risk management strategy. Through T&E, the stakeholders understand the deficiencies, threats, and opportunities that exist within the system. As a potential work in progress, stakeholders at the Coast Guard will understand and appreciate the changes needed to achieve the desired end product.
Acquisition Process and the Need for Testing
The Department of Homeland Security (DHS) and the Coast Guard have specific acquisition guidelines that provide for operational test and evaluation performed by an independent agency. The testing process aims to ensure that the project meets all the predetermined requirements before receiving approval. The Coast Guard primarily relies on the US Navy to perform operational tests and evaluations for all the crucial acquisition programs within its department (Mackin, 2016). The Navy has a subsection known as COFT, which conducts the actual tests and acts as an independent evaluator. More importantly, COFT has conducted all the asset tests and evaluations for the Coast Guard. In the testing, the Navy paid significant attention to two fundamental areas. First, the T&E process looks at the operational effectiveness. Here, the testers are concerned with whether an asset has all it takes to meet the mission's needs (Mackin, 2016). Secondly, the testers are keen on evaluating the operational suitability. Here, the testing team looks at whether the agency has the much-needed logistics to support the infrastructure to acceptable standards. Operational effectiveness and suitability are measured using different metrics. An example of one of the metrics is known as the critical operational issue (COI). During the testing procedure, the COI will tell the project manager whether the system will achieve the mission in question (Mackin, 2016). Also, the testing procedure pays significant attention to the key performance parameters (KPP). The KPPs are the capabilities that testers believe are crucial for the accomplishment of the mission.
Initial Testing
The US Navy performed the initial testing of the project in April 2014. The Navy concluded that the NSC met all the operational and technical requirements and ranked it effective and suitable. However, during the testing, the stakeholders discovered ten main problems or deficiencies. Some of the deficiencies touched on equipment failure, especially on the close in a weapon system. The decoy launcher showed signs of deficiency and a misfire experienced on the 57mm gun (Mackin, 2016). The initial testing process is fundamental for two main reasons. First, it verifies the performance of some of the main systems within a project. Secondly, it provides the project management with information regarding whether assets can meet the mission's requirements. The testing process mainly pays significant attention to key performance parameters and critical operational issues within the system. According to the initial testing, the NSC met 12 of the 19 performance parameters required to certify the project as satisfactory (Truver, 2013). However, during the process, the Navy and the Coast Guard disagreed on the requirements to certify the cutter boat operations as satisfactory to the NSC. The gaps identified during the initial testing prompted a follow-up procedure to assess the functionality of the project.
Operational Issues
Based on the initial testing, the need to perform another T&E remained a critical concern. However, the (DHS) did not clarify nor give any directions on when the testing process would happen again. Therefore, most of these problems started emerging during the operation stage. The NSC faced numerous performance issues that the Navy did not anticipate during the initial testing (Mackin, 2016). The operational and equipment failures compelled the NSC to operate in adverse conditions in several mission areas. In some instances, the cutter experienced high engine temperatures. The consequence of the elevated temperature was a reduced speed of the cutter in specific conditions. The problem of cracked cylinder heads also affected the full operation of the system. Thirdly, the report also recorded that some generators were haphazardly overheating. The Coast Guard initiated programs aimed at uncovering the sources of the problem (Truver, 2013). However, this did not help as the NSC already suffered from immense setbacks that limited its chances of completing the designated missions. Therefore, this prompted GAO to perform another test that sought to improve the much-needed efficiency and resolve any system limitations.
GAO’s T&E and Recommendations
As previously indicated, the NSC was a project to replace the old Coast Guard vessels. The study's main objective was to remove the high endurance fleet and replace them with modern and effective ones. However, the initial testing allowed the project to continue without resolving some of the major problems identified in the T&E. Therefore, GAO reviewed the initial testing event conducted by NSC (Mackin, 2016). GAO primarily paid attention to three fundamental aspects. First, it evaluated NSC’s system performance during the test. Secondly, it also assessed the Coast Guard’s determination to engage in a follow-up test. Thirdly, the organization looked at NSC’s performance during the regular operations. Part of the procedure included analyzing operational reports, test reports, DHS, and Coast Guard policies. More importantly, GAO utilized primary sources such as interviews with the coast guard officials. The second T&E resulted in several outcomes (Truver, 2013). GAO found that the problems identified during the initial testing prevented the coast guard officials from fully operating the NSCs. The issues experienced during operations required immediate retrofits.
In light of the second test's problems, GAO developed several recommendations that would assist the project heading forward. GAO challenged the DHS to strengthen the test and evaluation procedures for major infrastructure and assets. Based on the Navy's inefficient T&E, GAO remained keen on ensuring that all future tests undergo thorough procedures to guarantee accurate identification of capabilities and limitations (Truver, 2013). Secondly, GAO petitioned the Coast Guard officials to ask the NSC program to delineate the performance parameters that defined the cutter boat operations. Developing standard performance parameters is critical in ensuring standardized outcomes during tests and evaluations. It also creates a proper foundation for comparison (Mak, 2017). In the aftermath, the Coast Guard and the DHS agreed with the recommendations. The second evaluation process played a significant role in providing the future of the NSC.
Implications for the Future
The NSC project revealed the value of T&E processes for agencies, especially those dealing with critical issues like security. The project provides several fundamental implications for the future. First, T&E is a crucial procedure that must adhere to rules and standards determined by the authorities. The DHS should develop an elaborate standard operation procedure that determines the performance parameters and metrics for cutters to enable inconsistencies that might be experienced during testing (Mak, 2017). Secondly, the DHS must also provide clear guidelines for the criteria used to certify a project as a pass or a failure. Although the Navy provided the NSC with the go-ahead, it later emerged that the cutters experienced numerous deficiencies that adversely affected it from attaining its mission. Thirdly, a T&E should be a continuous process that occurs after a specified period. The case study has shown that GAO's follow-up testing revealed significant problems and provided recommendations that would help the NSC project in the future.
Conclusion
The NSC project demonstrates the value of T&E in uncovering the capabilities and deficiencies within a project. The NSC project aimed at replacing the old cutters with new and more efficient ones to assist in various security missions at the coast. Mandatory testing was needed to assess whether the new cutters would serve the mission for which they were made. The US Navy performed the initial testing procedure. Despite issuing a pass, the Navy found out 10 significant problems associated with the project. In the subsequent years, this led to numerous operational problems with the new cutters. GOA embarked on a thorough assessment and uncovered some shortcomings with the initial T&E. It went ahead and provided recommendations for testing and evaluations in the future. The case study emphasizes the importance of accurate T&E and the need for a follow-up plan to discover any additional issues.
References
Dahmann, J. (2012, March). Integrating systems engineering and test & evaluation in system of systems development. In 2012 IEEE International Systems Conference SysCon 2012 (pp. 1-7).
Mackin, M. (2016). Coast Guard Acquisitions: Enhanced Oversight of Testing Could Benefit National Security Cutter Program and Future DHS Acquisitions . Government Accountability Office Washington United States.
Mak, M. A. (2017). Coast Guard Recapitalization: Matching Needs and Resources Continue to Strain Acquisition Efforts (No. GAO-17-654T).
Truver, S. C. (2013). The Legend-Class National Security Cutters: Linchpins of the Twenty-First Century US Coast Guard. Naval Engineers Journal , 125 (2), 41-49.
