An Unified System for Data Analytics and In Situ Query Processing

In today's world data is being generated at a high rate due to which it has become inevitable to analyze this data efficiently and produce results quickly. But, data scientists and analysts are required to use different systems, because apart from SQL querying relational databases are not well equipped to perform complex data analyses. Due to this, data science frameworks are in huge demand. This may require significant data movement across multiple systems, which can be expensive. Furthermore, with relational databases, the data must be completely loaded into the database before performing any analysis. We believe that it has become the need of the hour to come up with a single system which can perform both data analysis tasks and SQL querying. Ideally, this system would offer adequate performance, scalability, built-in functionalities, and usability. We extend the Python's Dask framework to present DaskDB, a scalable data science system with support for unified data analytics and in situ SQL query processing on heterogeneous data sources. DaskDB supports invoking any Python APIs as User-Defined Functions (UDF). So, it can be easily integrated with most existing Python data science applications. Moreover, we introduce a novel distributed learned index to improve join performance. Our experimental evaluation involve the TPC-H benchmark and a custom UDF benchmark, which we developed, for data analytics. And, we demonstrate that DaskDB significantly outperforms PySpark and Hive/Hivemall.