Graduation Semester and Year

Summer 2026

Language

English

Document Type

Dissertation

Degree Name

Doctor of Philosophy in Computer Science

Department

Computer Science and Engineering

First Advisor

Dr. Sharma Chakravarthy

Second Advisor

Dr. Abhishek Santra

Abstract

Due to the proliferation of cameras in handheld devices and the widespread use of CCTV, images and videos have become a preferred alternative for capturing and disseminating information. Automated analysis for understanding image or video contents (e.g., objects, activities, backgrounds, situations of interest, etc.) is critical for many applications such as Civic Monitoring, Surveillance (in general), monitoring activities in Assisted Living environments, and many more. Image and Video Analysis (IVA) research has been ongoing for several decades, resulting in numerous techniques for algorithmically analyzing and understanding image and video contents.

Image Analysis (IA) has advanced in several areas, including object recognition, image classification, pose recognition, and more. Although Video Analysis (VA) subsumes image analysis (i.e., analysis of individual video frames or sequences), it is far more complex than IA because video contents change over time and have to be analyzed as a sequence. Several decades of research in Video Content Extraction (VCE) have led to the development of advanced content extraction algorithms, such as object recognition, object tracking, etc. However, identifying interesting or complex activities (termed situations in this thesis) that are embedded in and span multiple consecutive frames (e.g., a package being stolen from a premises, heavy traffic flow at an intersection, etc.) cannot be achieved by merely analyzing the contents of individual frames. To fully understand the video content, automated analysis of various, diverse situations that span a number of frames is essential. Our focus in this thesis is on identifying situations from the extracted contents of individual frames and piecing together meaningful information across frame contents. For this purpose, the extracted contents need to be represented using appropriate data models, situations need to be expressed on these models using an easy-to-use language (preferably a non-procedural query language or composable analysis expressions), and algorithms need to be developed for accurately and efficiently detecting the expressed situations. However, to date, situation analysis has been accomplished, to a large extent, either manually or with customized algorithms for each situation rather than with a general-purpose, domain-independent framework.

Broadly, two approaches to Video Situation Analysis (VSA) can be identified: manual and custom algorithms. In manual analysis, a human watches a video and identifies interesting or specified situations. Although this approach can be applied for forensic analysis (that is, after-the-fact analysis) or for real-time monitoring, it is impractical due to the significant human capital required. Further, manual situation analysis is subject to human fatigue, skill-set disparities, and human error. In contrast, custom solutions are designed for situations in specific domains and/or contexts and are applicable to videos from those domains or contexts. In addition, custom solutions are not fast enough for real-time situation analysis at this time. Furthermore, different custom algorithms need to be developed, or significant retraining is required (if machine learning approaches are used) to identify various situations. Currently, these alternatives are typically applied after the fact to analyze situations of interest from archived videos. Even for forensic analysis, the custom solutions are not general-purpose. These algorithms cannot analyze all diverse videos (from different domains) or situations, as new algorithms or software packages need to be developed for each situation or video type (from different domains). Hence, a common limitation of the above alternatives is their inability to perform real-time and/or general-purpose situation analysis.

In this thesis, we propose a general-purpose, novel framework that overcomes the limitations of current VSA alternatives. It is a general-purpose solution rather than a customized one, and is amenable to real-time video analysis similar to real-time analysis of stream data. Our approach can also be used for forensic video analysis, which is the focus of this thesis, as it is the first step towards real-time video situation analysis. The proposed approach extracts contents (as much as possible) from video frames using state-of-the-art video content extraction (or VCE) technologies. These extracted contents are represented (modeled) using two alternative expressive data models – relational model with extensions (termed R++) and graph models, including multiple graphs (or layers) per video. Once appropriately modeled, the extracted contents are used as data streams (with the underlying model being the relational model). As a result, Continuous Query Processing (CQP) can be used on streams using the proposed Continuous Query Language for Video Analysis (CQL-VA) using operators specifically extended for video analysis. The use of non-procedural Structured Query Language (SQL) as the basis of CQL-VA provides a much easier way to express situations in terms of “what to identify” (intent) instead of “how to identify” (algorithm), shifting the burden of efficient query processing to the underlying database query processor/optimizer. Unlike custom algorithms, the proposed approach for both representations supports extraction of video contents only once, and evaluation of as many queries or analysis expressions on that as needed, avoiding the development of new algorithms or retraining. A major advantage is that as VCE algorithms improve, they can be used without changing the situation analysis part. More accurate information extraction will lead to a more accurate situation analysis in our approach without any additional effort or change.

Similarly, the graph models provide an alternative representation and facilitate the detection (or analysis) of certain situations that are difficult (or impossible) to achieve using the relational model, even with extensions. Here, existing graph algorithms (e.g., clustering) are used, and additional efficient algorithms have been developed to identify various interesting embedded situations. Together, these two models and analysis approaches support the detection of a wide variety of situations embedded in the videos.

Finally, our approach is domain-independent, meaning that videos from various domains can be processed using the same framework. To accommodate domain independence in the proposed framework, primitive situations are identified and expressed as templates. These templates are used for identifying variants of primitive situations across domains by parameterizing them. Three contrasting domains – Assisted Living (AL), Civic Monitoring (CM), and Surveillance (SL) – have been used in this thesis to highlight domain independence of the proposed approach. Proposed primitive operators or analysis algorithms are used to compose specific queries or analysis expressions to detect more complex and meaningful situations. Finally, extensive analysis has been performed on several videos from the above domains, using the framework to evaluate accuracy, efficiency, and robustness. The scalability of the algorithms has also been tested with videos of varying lengths.

Keywords

Video Situation Analysis, CQL for Video Analysis, Graphs for Video Analysis

Disciplines

Other Computer Engineering

Download

Share

COinS
 
 

To view the content in your browser, please download Adobe Reader or, alternately,
you may Download the file to your hard drive.

NOTE: The latest versions of Adobe Reader do not support viewing PDF files within Firefox on Mac OS and if you are using a modern (Intel) Mac, there is no official plugin for viewing PDF files within the browser window.