Graduation Semester and Year
2023
Language
English
Document Type
Dissertation
Degree Name
Doctor of Philosophy in Computer Science
Department
Computer Science and Engineering
First Advisor
Fillia Makedon
Abstract
Cognition is the mental process of acquiring knowledge and understanding through thought, experience, and senses. Fatigue is a loss in cognitive or physical performance due to physiological factors such as insufficient sleep, long work hours, stress, and physical exertion. It adversely affects the human body and can slow reaction times, reduce attention, and limit short-term memory. Hence, there is a need to monitor a person's state to avoid extreme fatigue conditions that can result in physiological complications. However, tools to understand and assess fatigue are minimal. This thesis primarily focuses on building an experimental setup that induces cognitive fatigue (CF) and physical fatigue (PF) through multiple cognitive and physical tasks while simultaneously recording physiological data and visual cues from a person's face. First, we build a prototype sensor shirt embedded with various physiological sensors for easy use during cognitively and physically demanding tasks. Second, participants' self-reported visual analog scores (VAS) are reported after each task to confirm fatigue induction. Finally, an evaluation system is built that utilizes machine learning (ML) models to detect states of CF and PF from multi-modal sensor data, thus providing an objective measure. This effort is the first step towards building a robust cognitive assessment tool that can collect multi-modal data and be used for industrial applications to monitor a person's mental state. For instance, it enables safe human-robot cooperation (HRC) in industrial environments to avoid physical harm when a person's mental state is not good. Another example can be a personalized assistive robot for individuals with motor impairments to perform a task such as preparing lunch with real-time interventions based on the help required from the user.
Keywords
Multimodal, Physiological signals, Cognition, Machine learning
Disciplines
Computer Sciences | Physical Sciences and Mathematics
License
This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License.
Recommended Citation
Jaiswal, Ashish, "AN INTELLIGENT MULTI-MODAL FRAMEWORK TOWARDS ASSESSING HUMAN COGNITION" (2023). Computer Science and Engineering Dissertations. 391.
https://mavmatrix.uta.edu/cse_dissertations/391
Comments
Degree granted by The University of Texas at Arlington