N102-147 TITLE: Develop Valid Performance Measures for Multi-tasking Environments

TECHNOLOGY AREAS: Human Systems

ACQUISITION PROGRAM: Combat Systems Training ACAT II, Warfare Systems Training PEO IWS

RESTRICTION ON PERFORMANCE BY FOREIGN CITIZENS (i.e., those holding non-U.S. Passports): This topic is "ITAR Restricted." The information and materials provided pursuant to or resulting from this topic are restricted under the International Traffic in Arms Regulations (ITAR), 22 CFR Parts 120 - 130, which control the export of defense-related material and services, including the export of sensitive technical data. Foreign Citizens may perform work under an award resulting from this topic only if they hold the "Permanent Resident Card", or are designated as "Protected Individuals" as defined by 8 U.S.C. 1324b(a)(3). If a proposal for this topic contains participation by a foreign citizen who is not in one of the above two categories, the proposal will be rejected.

The requirement to perform more than one task in a limited period of time is prevalent in many work environments, where multi-tasking (MT) has been associated with stress and burnout in such areas as air traffic control, emergency dispatch, and nursing (Josslyn & Hunt, 1998). Within the Navy, development of new surface ships with reduced manning (e.g., LCS, DDX) has created the conditions for increased MT as fewer crewmembers must perform a greater number and variety of tasks within a time-constrained environment. Observational studies, coupled with incumbent interviews, have identified several major "themes" that characterize MT environments, such as having to manage risk by maintaining situational awareness, making multiple decisions rapidly, monitoring the flow of multiple tasks simultaneously, and monitoring multiple sources of information (Fischer & Mautone, 2005). While we know from anecdotal reports and informal observations that MT demands can be very high under certain operational conditions, and seem to vary considerably across different tactical settings and job domains, we lack validated measures of MT performance that allow us to both substantiate these claims and quantify their impact on overall Fleet readiness, system effectiveness, as well as individual and team performance.
The development of a suite or battery of MT measures of performance will yield enormous dividends to the Fleet. For example, armed with the ability to measure MT performance accurately and precisely, recruiting, training and personnel detailing processes could benefit from identification of jobs that require greater demands in MT. In addition, the MT measures could be used in person-in-the loop simulations to validate Modeling and Simulation (M&S) design efforts where crew models of time-pressured tasking have been constructed. Training resource and personnel selection criteria (via tests of MT ability) could then be allocated to targeted areas of greatest MT. Also, through more judicious application of HSI designs, it would be possible to determine whether, and if so by how much, MT demands have been reduced, resulting in lower ownership costs and improved system/ship operational effectiveness. In short, development of a set of valid measures of MT performance for the Fleet environment will provide empirical information that can be utilized by the manning, personnel selection, HSI design, and training development communities so that better multi-taskers can be assigned to high-demand MT environments, redesigned systems engineering to reduce MT demands, and MT training interventions targeted for jobs and settings where MT skill is most important.

Development of these unique performance measures will require the construction of a measurement model that provides a systematic representation of the dynamic relationship of environmental, temporal, behavioral, attitudinal, and cognitive variables on MT performance. While it is expected that existing theories of cognitive behavior, such as multiple resource theory (Wickens, 2002) and rapid decision-making (Gillan, 2002), will provide useful concepts, original theoretical work will be needed to apportion the candidate measures of MT performance across objective (e.g., amount of work activity performed per unit time, response time, number of errors) and subjective (e.g., self-rated reports of resource "exhaustion", workload) metrics.

In particular, we believe a successful measurement model will have the following characteristics. First, the model must result in measures that are dynamic and which capture the temporal basis of information-based task processing. As such, these should be process measures rather than static. Second, the measures should encompass cognitive meta-competency skills that will underlie many of the tasks that surface ship operators must perform. Examples include, but should not be limited to, such areas as situational awareness, critical thinking, adaptive decision-making, and attention management. Third, the measures, when collected in the context of vignette/scenarios, should yield fairly precise indications of when multi-tasking demands have been exceeded, thereby helping to pinpoint where multi-tasking strategies can be employed. Examples here include task shedding, task prioritization, and task delegation, among others. Fourth, the measures should be sufficiently robust that they can tie into programs aimed at promoting time-critical risk management to improve safety at the rate, mission area, and ship level. A measurement model that helps delineate where and why multi-tasking demands breakdown can then be fed back into strategic-level initiatives concerning talent/manpower management and sustained combat operations at-sea.

In terms of validation, application of a robust MT measurement model will result in content, construction, and criterion validation (Cronbach, 1984). The measurement model should not only guide the specification of existing MT measures of performance to a given work environment, it should also support identification and development of new measures as different job/task settings and new or modified mission requirements are considered.

In Phase I, the research team should perform a detailed literature search to identify candidate measures of MT performance in the areas of objective behavior specification, cognitive processes, and attitudinal factors. A report should be produced that presents a measurement model and a roadmap for performing validation studies in Phase II. A successful Phase I will have established a proof of concept that a cohesive set of MT measures of performance can be created for job domains under various mission contexts and time constraints of interest to the Navy.

In Phase II, the research team should take the measurement model and apply the candidate measures of MT performance to at least two different Navy job domains. Validation studies should be performed to support the extent to which the candidate measures satisfy requirements for content, construct, and criterion validity. The impact of various environmental and HSI factors on MT performance should also be part of the validation studies. Both qualitative and quantitative indices of MT demands should result from the studies, where an economy of scale should be established in taking the measures created in the first job domain and applying/adapting them to the second. A "roll-out" plan for applying both the measurement model and the associated MT performance measures to a broader range of Navy job domains under varying degrees of time-critical performance requirements should be a prominent part of the Phase II final report and out brief.

REFERENCES:
1. Cronbach, L.J. (1984). Essentials of psychological testing (4th ed). New York: Harper & Row.

2. Fischer, S.F. & Mautone, P.D. (2005, August). Multi-tasking assessment for personnel selection and development. (ARI Contractor Report 2005-07). Washington, DC: US Army Research Institute for the Behavioral and Social Sciences.

3. Gillan, C.A. (2002). Aircrew adaptive decision-making: A cross-case analysis. Unpublished doctoral dissertation. University of San Diego.

4. Josslyn, S. & Hunt, E. (1998). Evaluating individual differences in response to time pressure situations. Journal of Experimental Psychology: Applied, 4(1), 16-43.

5. Wickens, C. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3.2, 150-177.

KEYWORDS: Multi-tasking, cognitive skill, performance assessment, job selection, job placement

** TOPIC AUTHOR (TPOC) **

DoD Notice:   Between April 21 and May 19, 2010, you may talk directly with the Topic Authors to ask technical questions about the topics. For reasons of competitive fairness, direct communication between proposers and topic authors is not allowed starting May 19, 2010, when DoD begins accepting proposals for this solicitation.

However, proposers may still submit written questions about solicitation topics through the DoD's SBIR/STTR Interactive Topic Information System (SITIS), in which the questioner and respondent remain anonymous and all questions and answers are posted electronically for general viewing until the solicitation closes. All proposers are advised to monitor SITIS (10.2 Q&A) during the solicitation period for questions and answers, and other significant information, relevant to the SBIR 10.2 topic under which they are proposing.

If you have general questions about DoD SBIR program, please contact the DoD SBIR Help Desk at (866) 724-7457 or email weblink.