Institute for Creative Technologies

Institute for Creative Technologies (ICT)

Established	1999
Location	Playa Vista , California , U.S.
Website	ict.usc.edu
Affiliated With The University of Southern California

The Institute for Creative Technologies (ICT) is a University Affiliated Research Center at the University of Southern California located in Playa Vista, California. ICT was established in 1999 with funding from the US Army.

The Institute's facilities as of April 2021

Dr. Mike Andrews, chief scientist of the US Army is described as "founder of and inspiration behind" the ICT. He followed up on discussions between US Army leadership (four-star general Paul J. Kern) and Disney Imagineering president Bran Ferren, on how to gain access to Hollywood entertainment industry expertise in high-technology areas such as computer-based Modeling & Simulation, and Virtual Reality. The name was derived from Ferren's title at The Walt Disney Company.

It was created to combine the assets of a major research university with the creative resources of Hollywood and the game industry to advance the state-of-the-art in training and simulation. The institute's research has also led to applications for education, entertainment and rehabilitation, including virtual patients, virtual museum guides and visual effects technologies. Core areas include virtual humans, graphics, mixed-reality, learning sciences, games, storytelling and medical virtual reality.

Honors and awards

• 2010 - Scientific and Engineering Academy Award
• 2018 - Second Academy Award for Technical Achievement
• 2022 - Emmy (Technical Achievement)

Army affiliation

ICT is a DoD-sponsored University Affiliated Research Center (UARC) working in collaboration with the U.S. Army DEVCOM Soldier Center. The ICT is one of the Army’s four University Affiliated Research Centers (UARC).  UARCs are a strategic United States Department of Defense (DoD) Research Center associated with an American university.  UARCs are formally established by the Under Secretary of Defense for Research and Engineering (USD(R&E)) to ensure that essential engineering and technology capabilities of particular importance to the DoD are maintained and readily available.  The mission of the ICT UARC is to conduct basic, applied, and advanced demonstration research to develop the new tools, methods, and technologies required to improve military training, education, and combat operations.  The ICT UARC accomplishes this mission by employing a resident faculty that leverages government research with the research capabilities of academia, private industry, and the commercial entertainment industry (e.g., Hollywood, theme parks, and the commercial gaming industry).  

Army program managers include Mr. Mark McAuliffe, Dr. Jim Blake, Dr. Jeff Wilkinson, Dr. John Hart (2008-2022) and Dr. Keith Brawner (2022–present).

Major projects and accomplishments

Year Started	Project	Description
1999	Virtual Humans	ICT advances research in computer-generated characters that use language, have appropriate gestures, show emotion, and react to verbal and non-verbal stimuli.
2000	Emotion Modeling	One of the first, and most comprehensive computational models of human emotion provided virtual humans with the ability to exhibit and understand human emotions.
2000	Mission Rehearsal Exercise	Mission Rehearsal Exercise (MRE) is ICT’s earliest virtual human project combining artificial intelligence software and basic research to create simulated scenarios used to teach young officers to make decisions.
2003	Joint Fires and EffectsTraining System (JFETS)	The JFETS was a suite of state-of-the-art immersive virtual reality environments designed to help Warfighters make critical decisions under stress and provide collective team training and situational awareness. Tasks not only focused on the technical application of skills, but also on the thought processes involved in employing those skills. By leveraging the ICT's mixed reality technology, JFETS recreated environments that placed Warfighters in real world, current operational settings. Stressors included heat, wind, explosions, human distress noise, and snipers. JFETS provided artificial intelligence behaviors to insurgent forces and realistic, reactive behaviors to civilians. Installed at Fort Sill, JFETS trained over 16,000 Warfighters beginning in 2004, and was used by members of the US Army and Marine Corps for training prior to deployment to Afghanistan and Iraq.
2003	Full Spectrum Command	Full Spectrum Command (FSC) is a PC-based training aid that modeled the command and control of a U.S. Army Light Infantry Company in a MOUT environment. As the Captain of a U.S. Army Light Infantry Company, the user interprets a five-part Operational Order (OPORD) for a given scenario, organizes his platoons, develops a multi-phase plan and coordinates the actions of approximately 120 soldiers during the engagement. The scenarios were designed to develop cognitive skills: tactical decision-making, resource management and adaptive thinking. These scenarios focused on asymmetric threats within peacekeeping and peace-enforcement operations. Each scenario was developed in conjunction with subject matters experts from the US Army Infantry Center in Fort Benning, Georgia to ensure both military and pedagogic fidelity.
2004	SLIM ES3	A web-delivered, web-enabled, combat patrol training tool for U.S. ground forces, providing Active Surveillance, Threat Indicator Identification and Information Operations.
2004	BiLAT	Driven by Iraq early security and stability operational needs, ELECT BiLAT was developed as a prototype game-based simulation for Soldiers to practice conducting bilateral engagements in a cultural context. The prototype provided students with the experience of preparing for a meeting including familiarization with the cultural context, gathering intelligence, conducting a meeting and negotiating when possible, and following up on meeting agreements as appropriate. The ELECT BiLAT architecture was based on a commercial game engine that is integrated with research technologies to enable the use of virtual human characters, scenario customization, as well as coaching, feedback and tutoring. BiLAT became a part of the Games for Training Program of Record administered by PEO STRI.
2004	Bravemind	Bravemind is a basic and applied research effort utilizing virtual reality exposure therapy in providing relief from pat-traumatic stress.
2004	Sgt. John Blackwell	SGT John Blackwell, a virtual human, was created for the 2004 Army Science Conference, as a demo of ICT virtual human technology, including spoken question-answering ability, and a mixed-reality presentation on a transparent screen.
2004	High Dynamic Range Imaging	High Dynamic Range Imaging is the ICT Vision & Graphics Lab’s first efforts inbasic research that helped pioneer the trajectory in high dynamic range imaging, a technique used in imaging and photography to reproduce a greater dynamic range of luminosity than what is possible with standard digital imaging or photographic techniques.
2004	FlatWorld	FlatWorld is a virtual reality environment which allows users to walk and run freely among simulated rooms, buildings, and streets. The system integrates Hollywood set construction techniques with state of the art computer graphics and immersive display technology. FlatWorld provides a powerful, flexible tool for experiential simulation and training.
2004	Army Excellence in Leadership (AXL)	AXL provides an engaging and memorable way to transfer tacit knowledge and develop critical thinking through case-method teaching, filmed storytelling and interactive training. The primary products are filmed cases created in collaboration with Hollywood talent to address specific leadership issues and an easily modifiable website, AXLnet. ICT developed five filmed leadership cases addressing complex decision-making skills for the U.S. Army. The films, all based on real-life situations, were brought to life by experienced Hollywood screenwriters and professional Hollywood actors. AXLnet provides a dynamic and interactive experience for students and easy-to-use tools for instructors to author customized lessons. The system draws on ICT's research in natural language processing to allow students to interview characters from the cases through free-text questions. The system can also provide feedback and tailor the learning experience based on student responses. Over 10,000 Soldiers were trained with AXL, including West Point cadets who used it in a Leadership course at the Academy.
2005	Full Spectrum Warrior	Full Spectrum Warrior (FSW) is a squad-based, tactical-action game that places the player into the 21st century set in a Middle Eastern MOUT environment. In FSW, players assume the role of a squad leader, in command of a squad of two fire teams of US Army infantry soldiers. It is their responsibility to achieve specific objectives through the skillful deployment and use of the men under their command. It is the combination of tactical planning and guided execution on modern, asymmetrical battlefield that is the foundation of FSW. FSW was released for Xbox on June 1, 2004 and later released by other entities for Windows and PlayStation 2. Full Spectrum Warrior was the first military training application published for a commercial game console. The game was a critical and commercial success, acquired for commercial release by game publisher THQ. At the 2003 Electronic Entertainment Expo (E3) Game Critics Awards, Full Spectrum Warrior won “Best Original Game” and “Best Simulation Game.”
2005	Full Spectrum Leader	Full Spectrum Leader (FSL) extends the Full Spectrum suite of applications to the Light Infantry Platoon level with a cognitive leadership trainer featuring several innovations. FSL showcases a computer opponent with an adaptive intelligence. Unlike most games and simulations that have a fixed (albeit concealed) OPFOR Plan, FSL’s opponent attempts to recognize players’ actions and respond dynamically. Additionally, FSL includes close air support, artillery calls-for-fire and casualty evacuation to heighten the realism and training value of the game. FSL is the first Full Spectrum application to include a “hasty defense” mission.
2005	Smart Body	The SmartBody basic research program discovered the knowledge and modular framework architecture for virtual human animation that accelerated industry capabilities.
2006	UrbanSim	UrbanSim is a PC-based virtual training application for practicing the art of mission command in complex counterinsurgency and stability operations. It consists of a game-based practice environment, a web-based multimedia primer on doctrinal concepts of counterinsurgency and a suite of scenario authoring tools. The UrbanSim practice environment allows trainees to take on the role of an Army battalion commander and to plan and execute operations in the context of a difficult fictional training scenario. In April 2011, UrbanSim transitioned to the Army and is available at the MilGaming portal. It was named a program of record for two Army programs, Games for Training and the Army Low Overhead Training Toolkit (ALOTT), and has seen widespread application across institutional and operational settings. Key deployment sites include; the School for Command Preparation, Ft. Leavenworth, KS; the Maneuver Captain’s Career Course, Ft. Benning, GA; and the Warrior Skills Training Center, Ft. Hood, TX
2007	Sergeant Star	A recruiting tool developed in collaboration with Accessions Command to support Army recruitment requirements.
2008	DMCTI	Winner of a 2008 Army Modeling and Simulation Award for Army-wide team trammg, the DMCTI prototype application trains US Army logistical planners and supports the understanding of the Army distribution management process. The DMCTI promotes the development of strategies for best exploiting the capabilities of logistics management systems, including the Army's recognized logistics command and control tool, Battle Command Sustainment Support System (BCS3).
2008	CAD-TS	Cognitive Air Defense–Training System, Engagement Control Station Simulation – Trained and assessed soldiers’ abilities to recognize and respond to perceived threats with complete situational awareness. Transitioned to the USAADASCH at Fort Still in 2010.
2008	ECS2	The CAD-TS Engagement Control Station Simulation prepares soldiers to use the US Army Patriot missile defense engagement operations center for the Patriot firing unit. It is designed to help bridge the gap between recognizing the 2D scope of information from the radar interface and understanding that information based on realistic visualizations of the 3D airspace. The CAD-TS ECS2 trains and assesses soldiers' abilities to recognize and respond to perceived threats with complete situational awareness. The system consists of five projection screens and a lecture stage, as well as Blue and Red cells for force-on-force functionalities. The CADTS ECS2 is able to accommodate up to 64 soldiers per training session through combined immersive simulation and digital media-enhanced classroom instruction. It transitioned to the USAADASCH at Fort Still in 2010.
2008	Gunslinger	Gunslinger combined ICT’s virtual human technology with Hollywood and ICT transitioned the TAC-Q system to PEO STRI (PM ConSim) in 2010 where it was further developed and integrated by private industry into the Intelligence Electronic Warfare Tactical Proficiency Trainer (IEWTPT).
2009	MCIT	Sponsored by JIEDDO, the Mobile Counter-Improvised Explosive Device Interactive Trainer (MCIT) introduced lower level enlisted (E1-4/5) and lower level officers (O1-2) to the nature of type, construction, emplacement and attack associated with Improvised Explosive Devices (IEDs). MCIT was a self-paced and self-motivated experience wherein the Soldiers learned how to carry out a successful Red V. Blue IED attack during the capstone exercise. MCIT was implemented in a series of three shipping containers. Each housed interactive exercises that combined ICT’s core strengths in storytelling, videogames and simulations as teaching tools. Story vignettes from an insurgent bomb maker and US soldiers delivered the training materials and guided the trainees through the self-paced experience. Students participated in a multiplayer videogame where they assumed either the role of an insurgent ambush team or coalition military patrol. In the first year alone, MCIT was deployed to Ft. Bragg, Camp Pendleton and Camp Shelby and trained over 15,000 Soldiers and Marines.
2009	INOTS	The Immersive Naval Officer Training System (INOTS) is a blended learning environment that merges traditional classroom instruction with a mixed reality training setting. INOTS supports the instruction, practice and assessment of interpersonal communication skills.
2009	SimCoach	Designed to attract and engage military service members, veterans, and their significant others who might not otherwise seek help. Aimed to motivate users in taking the first step and seeking information and advice with regard to their healthcare and their general personal welfare.
2009	TaC-Q	Tactical Questioning dialogues are those in which smallunit military personnel, usually on patrol, hold conversations with individuals to produce information of military value(Army 2006). Thesimulation training environment can be used to train military personnel in how to conduct such dialogues.
2009	Virtual Human Toolkit	The ICT Virtual Human Toolkit is a collection of modules, tools, and libraries designed to aid and support researchers and developers with the creation of virtual human conversational characters.
2009	Digital Twins	Ada and Grace are a pair of female teenage virtual humans who acted as docents at the Boston Museum of Science. They could answer questions about exhibits at the museum, themselves, and the technology underlying them while delivering a subliminal message about the role of women in STEM.
2010	Light Stage	ICT’s Vision & Graphics Lab pioneered the use of the Light Stage, a high definition capture of actors allowing them to be placed in any environment real or virtual, recognized with an Academy Award® for its use in films including Avatar, Benjamin Button and Spider-Man 2.
2010	COPA	The “Choice of Plausible Alternatives” (COPA) evaluation of commonsense causal reasoning was published and, in the years since, has played an instrumental role in advancing research on automated question-answering and deep neural network language models in both academia and industry.
2010	ELITE	Based on the original INOTS system, Emerging Leader Immersive Training Environment for leader development is a mixed reality environment providing junior leaders an opportunity to learn, practice and assess interpersonal communication skills in basic counseling.
2010	Shared Space	Examining the modes and efficacy of virtual humans and people sharing space through monitors, mixed reality flats, transcreen, and 360 degree light fields
2011	NDT	New Dimensions in Testimony is an initiative in collaboration with the USC Shoah Foundation to record and display testimony in a way that will continue the dialogue between Holocaust survivors and learners far into the future.
2011	MultiSense/SimSensei (DCAPS)	Pioneering efforts within DARPA’s Detection and Computational Analysis of Psychological Signals (DCAPS) project, SimSensei encompasses advances in artificial intelligence fields of machine learning, natural language processing, and computer vision used to identify psychological distress such as depression, anxiety, and PTS symptoms.
2012	Mixed Reality Lab (Headmounted Displays)	ICT’s Mixed Reality Lab’s pioneering work in discovering integrated technology requirements and prototyping hardware and software solutions laid the groundwork for making low-cost VR a reality.
2012	INVRSE	A low-cost platform for mobile-based hardware that integrates casual immersive experiences seamlessly into 2D touch-screen based media.
2012	FOV2GO VR Viewer	Named Field-of-View-2-Go (FOV2GO), this is portable fold-up foam core device that transforms smartphones into 3D viewers. Google Cardboard, Oculus Rift and the Samsung Gear VR all have significant roots in this project.
2012	Redirected Walking	The Redirected Walking Toolkit is a unified development and deployment platform for enabling exploration of large virtual environments. Through redirected walking and lightfield model demos, the lab created software that will let you walk around forever in VR.
2013	DisasterSim	DisasterSim is a game-based training tool focused on international disaster relief. Trainees take on the role of a joint task force staff member coordinating the US Department of Defense’s (DoD) humanitarian aid and disaster relief efforts in a foreign country following a natural disaster.
2013	Dark Networks	Dark Networks is a two-player, web-based game focused on the organizational network structure of covert, violent organizations (or Dark Networks) and how these structures can be altered to make the terrorist group more or less effective.
2014	CounterNet	CounterNet is a single-player, web-based counter-terrorism game that teaches cybersecurity students and personnel how to identify, track, counter and thwart online terrorist activity.
2014	PAL3	Personal Assistant for Life Long Learning is a system for delivering engaging and accessible education via mobile devices.
2014	STRIVE	The Stress Resilience In Virtual Environments (STRIVE) project aims to create a set of combat simulations (derived from our existing Virtual Iraq/Afghanistan exposure therapy system) that are part of a multi-episode narrative experience.
2014	EMPOWER/OmniSense	A virtual human centered multimodal sensing system that unobtrusively assesses fitness and readiness to enhance warfighter performance and resilience.
2015	Rapid Avatar Capture	Generates photorealistic, 3D characters of real people into virtual humans in less than 30 minutes.
2015	ESP	Early Synthetic Prototyping created both multi-player and standalone systems that allowed future combat systems to bevirtually tested by end users, it was transitioned to Operation Overmatch in 2015.
2015	SciTechFutures	An ASA(ALT) funded research project that helps Army leaders ideate in the S&T space while identifying blind spots in Army planning.
2015	Bravemind	Bravemind is virtual reality (VR) exposure therapy system aimed at providing relief from post-traumatic stress (PTS). Currently distributed to over 50 sites, including VA hospitals, military bases and university centers, the Bravemind system has been shown to produce a meaningful reduction in PTS symptoms.
2015	VITA	The Virtual Interactive Training Agent is a virtual reality job interview practice system for building competence and reducing anxiety in young adults with Autism Spectrum Disorder (ASD) and other developmental disabilities.
2015	Project BlueShark	While this “future communications and collaboration environment” was created for the Office of Naval Research, it was tested by a wide range of Army leadership, and helped inform follow-on activities such as Early Synthetic Prototyping and the Synthetic Training Environment.
2016	ELITE SHARP	Three laptop-based training applications developed using the ELITE platform. Bystander Resource Assessment and Virtual Exercise (BRAVE), Command Team Trainer (CTT), and Prevention and Outreach Simulation Trainer (POST).
2016	Balance of Terror	BoT is part of the Naval Postgraduate School’s Global ECCO (Education Community Collaboration Online) initiative. Global ECCO’s mission is to build and strengthen the Regional Defense Fellowship Program's (RDFP) global alumni network of Combating Terrorism (CbT) experts and practitioners through innovative and engaging technologies and techniques that both enable and encourage collaborative partnership between individuals, nations, organizations, and cultures. Balance of Terror is a two-player strategic game centered around terrorism and counterterrorism. Other games built by ICT deployed on Global Ecco include CounterNet and Dark Networks. https://nps.edu/web/ecco/game-center1
2016	One World Terrain	OneWorld Terrain is an Army effort that provides cutting-edge 3D global terrain capabilities and services to the warfighter for training and operations.
2017	CVIT	The purpose of the Captivating Virtual Instruction for Training (CVIT) research effort is to identify expert military instructor techniques and procedures that can be incorporated into the design and development of engaging distance learning applications.
2017	SDC-R	Requested by Army to improve and reduce 40 hour required training course for Army senior leaders (~ 11,000) required to take the course every three years.
2017	IAOC	The Intelligence Architecture Online Course (IAOC) teaches Intelligence Architecture with videos, knowledge checks, missions, achievements, flashcards, and a library of official doctrine.
2017	MxR Drones	This research involves drones small enough to fit in the palm of a hand. The drones can follow and capture a person’s movements so they can be analyzed under a training simulation.
2018	DS2A	Digital Survivor of Sexual Assault is an interactive SHARP training system based on the international recognized Holocaust survivor New Dimension in Testimony application.
2018	RLDP	ICT in collaboration with the U.S.Army-Pacific (USARPAC) RLDP, developed a HA/DR interactive game scenario for student groups to use in a RLDP 19-03 class presentation.
2018	Sigma	Sigma (Σ) is a cognitive architecture and system whose development is driven by a combination of four desiderata: grand unification, generic cognition, functional elegance, and sufficient efficiency.
2018	Open Medical Gestures	Open Medical Gesture (OpenMG) is a tool that enables people to user their hands to directly manipulate objects in virtual reality and mixed reality simulations. OpenMG uses any brand 3D webcam or gloves to create a new level of VR immersion and naturalistic interactions. This resource includes objects and functionality specifically for medical simulations, but can be used for other types of simulations as well.
2018	MentorPal	MentorPal emulates conversations with a panel of virtual mentors based on recordings of real STEM professionals and has developed strategies for the rapid development of new virtual mentors, where training data will be sparse.
2019	Empower	The Enhancing Mental Performance and Optimizing Warfighter Effectiveness and Resilience (EMPOWER) research program brings together multiple technological innovations in instrumentation, behavioral sensing, big data analytics, artificial intelligence, and virtual human (VH) technology to design, develop, and evaluate applications that will revolutionize human-computer interaction and human sensing.
2019	OmniSense	OmniSense is a software platform for delivering an intelligent Virtual Human interaction for studying and measuring psychological measures related to warfighters’ health and performance. OmniSense is able to track and understand verbal and nonverbal behaviors, such as tone of voice, facial expression and sentiment of the spoken words.
2019	VACG	The VACG project was developed by ICT in collaboration with the Army Acquisition Support Center (ASC) in order to provide a digital forum for Army acquisition professionals to quickly and accurately obtain career advice and guidance.
2019	Marine Recon Study (CBC)	The team analyzed data related to prior life experience, self-identified personality characteristics and motivations, athletic experience, military recruitment history and other military experience, as well as physical performance metrics obtained within the Reconnaissance Training Company training itself.
2019	Virtual Patients	The Virtual Patient project uses virtual human technology to create realistic lifelike character avatars and uses speech recognition, natural language, non-verbal behavior and realistic scenarios for both military and non-military issues to train clinicians in interpersonal areas such as rapport, interviewing and diagnosis.
2020	Digital Cora (CBC)	Comprehensive and Continuous Mental and Physical Status Measurement of Pre- and Post-Deployment Warfighters with Mobile Application (Digital cORA). The objective of the research is to use comprehensive and continuous human health and performance data to identify predictive factors of degradation, or “red-lining” events in warfighters, and to investigate the effects of providing the warfighter with mitigation strategies.
2021	Sound (Immersive Audio)	This project examined several key issues in the implementation of immersive audio, addressing the problem from three perspectives: 1) identification of the underlying basic science issues of sound reproduction and perception, 2) delineation of the engineering considerations that impact present and future system design and development, and 3) implementation of an immersive audio system that can be used in virtual environment navigation for mission rehearsal and related applications. Amended in 2022 to build a prototype application showcasing the use of Data Sonification as an alternative to visual inputs for a dismounted user.

See also

• United States Army Simulation and Training Technology Center
• Entertainment Technology Center
• MIT Center for Collective Intelligence
• Technology Innovation Institute
• Creative technology
• Immersive technology

External links

• Official website
• ICT research projects