N112-141 TITLE: Advancing Performance Diagnostics to Support Decision Superiority

TECHNOLOGY AREAS: Human Systems

ACQUISITION PROGRAM: ACAT IV Battle Force Tactical Training

OBJECTIVE: Optimize cognitive performance diagnostic process. Advance instructor and performer in-the-loop machine learning capabilities to support cognitive performance complexities not served in current or planned instructor-led or fielded technology aids for training debriefings. Success will be measured by improved standardization in assessing cognitive performance measures and results (increased adaptive knowledge; increasew in situational recognition and assessment, self-knowledge leading to decrease in decision/judgement errors, improved learning curves to acquire expertise, improved training transfer).

DESCRIPTION: This innovative prototype effort will reorient the integrated performance diagnostic capability to improve and expand the debriefing heuristic. The research effort will establish approaches that extend existing debrief methods for cooperative systems to guide and support facilitator-led and metacognitive level self-assessment debriefs required for the cognitively dynamic and demanding environments. Group and individual performance diagnostics required for individual and team debriefs are presently one of the weakest areas of training, resulting in systemic lost opportunities for performance understanding, feedback and improvement which mirrors other domains [2]. The Navy's standards and metrics for used for unit and individual feedback are primarily focused on task completion and do not generally reflect required cognitive levels or interoperability requirements of performance [3]; therefore, debrief capability and transfer of learning is severely restricted.

While it is recognized that post-training exercise debriefs are the period where much learning can take place, all-too-often post-exercise debriefs fail to be conducted at all or fail to produce desired timely, accurate cognitive-level diagnostic feedback (e.g., interpreting the pattern of data to arrive at a conclusion) [4] or ultimately the desired behavior, skill, attitudinal modifications that support optimal performance. Reasons include failure to facilitate the session appropriately, lack of time, failure to stimulate the trainees to verbalize their experiences, as well as failure to have captured the most relevant variables at sufficient precision to infuse the training feedback/self-assessment with the appropriate level of cognitive and technical content. Training feedback is also problematic when based solely on human observations. Because observations can have a subjective basis, inter-rater reliability becomes an issue in maintaining standardization of evaluation and ability to track training trends. This is exactly the problem we are hoping to avoid; the machine learning approach helps in two ways here: (a) Since performance metrics designed at a gross level that does not provide accompanying cognitive diagnostics, it would be very useful for the crew and instructors to have some form of automated diagnosis backed up with concrete onscreen metrics to inform the performer (s) how close one is to an "optimal" action/decision and (b) with this difference measurement, the objective feedback can then inform people why it's working and not just that it is working.

Despite the computational advances of machine learning, their application is not without risk. The risks include: (a) Feeding the Model with Data; by their nature, machine learning models are very "data hungry," requiring considerable amounts of input, particularly if the model contains a number of predictor variables. (b) Picking the right Model; selecting the right model entails a mix of trial-and-error and expert judgment. (c) Model Convergence and Degree of Fit; depending on the patterns in the data, it may take a given model thousands of iterations to achieve a final solution. Once achieved, the modeler must make a decision concerning the degree to which the best-fitting model has done an adequate job of predicting the outcome measures. There is no hard and fast rule in this area, so the modeler�s experience with previous machine learning environments will be crucial, (d) Updating the Model; requirements for model updating and re-running a given algorithm as new case data are obtained. Also, the modeler must be aware of the impact of new system input variables on his/her model and whether there is a viable need for refitting the data with a revised model.[5]

PHASE I: The research team will identify and create a framework of cohesive cognitive skills that underlie effective "real time" situational awareness and decision-making. These same sources of information will then be used to determine the diagnostic and debrief requirements most deficient in the present cadre of instructors, and in general, debriefing practices. Phase I will conclude with development of a feasibility concept design for the most promising metacognitive measures and diagnostic feedback for improving performance.

PHASE II: The outcome of this phase is the development or adaption of methodologies and technologies that support the most promising performance diagnostic and debriefing strategies identified in Phase I. The impacts of these metacognitive training interventions will be formally evaluated in a field setting, and the results will be used to fine-tune the recommended diagnostic models to advance positive training impacts. Plans for practical implementation of the interventions on a larger scale will be provided as part of the Phase II deliverable.

PHASE III: Training exercises will provide a realistic path to test, evaluate, certify and transition of adaptive and dynamic instructional processes that center on explanation-based learning with diagnostic models is needed to supplement and extend active transfer of learning to the Fleet for an improved training capability to better dynamically identify and support aligned metacognitive training and sustainment of proficiencies.

PRIVATE SECTOR COMMERCIAL POTENTIAL/DUAL-USE APPLICATIONS: While the training community is presently proficient at identifying performance deficiencies and even deficiencies in process, those doing the evaluating are frequently not as effective at identifying underlying performance issues and communicating identified cognitive weaknesses as they are at identifying the shortcomings in the first place. This applies across training environments and any performance assessment and improvement effort. A validated integrated training methodology and technology package with demonstrated ability to enhance training feedback through metacognitive performance diagnostics and debriefing has applicability across the Services and beyond. The need for better individual and team performance diagnostics and debriefs has been established in such diverse fields as commercial aviation, nuclear control rooms, emergency medical teams, and firefighting units. Virtually any industry for which situational awareness, decision making and/or interoperability coordination plays a major role will benefit from meta cognitive-focused improvements in training facilitation and diagnostic capabilities/feedback methodologies.

REFERENCES:
1. Sheppard, J. W. (1992). Explanation-Based learning with Diagnostic Models, ARINC Research Corporation, Annapolis, MD.

2. Dismukes, R., K., Jobe, K.K., & McDonnell, L.K. (1997). LOFT debriefings: An analysis of instructor techniques and crew participation. NASA Technical Memorandum 110442, Ames Research Center: Moffett Field.

3. Spiker, Alan (2009) Interviews with 101st Soldiers at Fort Campbell, Anacapa Sciences, Santa Barbara, CA.

4. Yamamori, H. & Mito, T. (1993). Keeping CRM is keeping the flight safe. In E.L. Wiener, B.G. Kanki, and R.L. Helmreich (Eds), Cockpit resource management. New York: Academic Press.

5. Weiss, S.M. & Zhang, T. (2003). Performance analysis and evaluation. In N. Ye (ed), The handbook of data mining. Mahway, NJ: LEA.

KEYWORDS: Metacognitive debriefing; explanation based learning; cognitive performance measures; instructional methodology; active transfer

** TOPIC AUTHOR (TPOC) **

DoD Notice:   Between April 26 and May 25, 2011, you may talk directly with the Topic Authors to ask technical questions about the topics. Their contact information is listed above. For reasons of competitive fairness, direct communication between proposers and topic authors is not allowed starting May 26, 2011, when DoD begins accepting proposals for this solicitation.

However, proposers may still submit written questions about solicitation topics through the DoD's SBIR/STTR Interactive Topic Information System (SITIS), in which the questioner and respondent remain anonymous and all questions and answers are posted electronically for general viewing until the solicitation closes. All proposers are advised to monitor SITIS (11.2 Q&A) during the solicitation period for questions and answers, and other significant information, relevant to the SBIR 11.2 topic under which they are proposing.

If you have general questions about DoD SBIR program, please contact the DoD SBIR Help Desk at (866) 724-7457 or email weblink.