DATA DISTRIBUTED MANAGEMENT SYSTEM USING DATA ANALYSIS
Abstract
The rapid proliferation of big data has exposed significant limitations in traditional centralized data systems, necessitating the adoption of scalable and flexible data management solutions. This study presents the design and implementation of a Distributed Data Management System (DDMS) that integrates Kafka, Delta Lake/Apache Iceberg, and distributed compute engines (Spark/Trino) to address the challenges of large-scale data analysis. The proposed system leverages the strengths of modern open-source tools to provide a unified platform for real-time data ingestion, transactional storage, and interactive query processing. The background of the study highlights the exponential growth of data from diverse sources, including IoT devices, social media, and scientific research instruments, and the inability of traditional systems to cope with the increasing scale, heterogeneity, and real-time demands of modern applications. The statement of the problem emphasizes the need for a scalable, fault-tolerant, and unified DDMS model that addresses ingestion, storage, governance, and analytics.
Full-Text Access Notice
In accordance with the NERD Policy on promoting peer-reviewed publication, public access to the full text of a project, thesis or dissertation is restricted for three years, allowing the author and supervisors sufficient time to pursue peer-reviewed publication.
During this period, researchers with legitimate academic or research purposes may request authorisation directly from the author to enable NERD to release the indexed full texts of the work using the form below.