Ravi Nishant

ORCID Identifier(s)

0000-0003-3691-7710

Graduation Semester and Year

2017

Language

English

Document Type

Thesis

Degree Name

Master of Science in Computer Science

Department

Computer Science and Engineering

First Advisor

David Levine

Abstract

Logging is the process of storing information for future reference and audit purposes. In software applications, logging plays a very critical role as a development utility and ensures code quality. It acts as an enabler for developers and support professionals by providing them capability to see application’s functionality and understand any issues with it. Data logging has a widespread use in scientific experiments and analytical systems. Major systems which heavily uses data logging are weather reporting services, digital advertisement, search engines, space exploration systems to name a few. Although, data logging increases the productivity and efficiency of a software system, the logging process itself needs to be an efficient one. A logging system should be highly reliable, should support easy scalability and must maintain high availability. Logging infrastructure should also be completely decoupled with the parent system to ensure non-blocking operation. Finally, it should be secure enough to meet the needs of businesses and government as required. In the age of big data, logging systems themselves are a huge challenge for companies, so much so that few corporations have teams dedicated to providing data services such as data storage, analytics and security. They use logging frameworks of varying capabilities and as per their need. However, most of the logging utilities tend to be partially efficient or they face critical challenges like scalability, high throughput and performance. At present, we have few logging frameworks providing analytical capabilities along with traditional functionalities of data formatting and storage. As part of this thesis work, we come up with a logging framework which seeks to solve both functional challenges and problems related to its efficiency and performance. The system demonstrated here combines best features of multiple utilities such as messaging brokers like Kafka, event publishing through SQS and data management and analytics using ELK stack. The system implementation also utilizes efficient design patterns to tackle nonfunctional challenges such as scalability, performance and throughput.

Keywords

Visual logging, Logging, Big data, Messaging queues, AWS S3 appender

Disciplines

Computer Sciences | Physical Sciences and Mathematics

Comments

Degree granted by The University of Texas at Arlington

Download

Share

COinS