TALI: An Update-Distribution-Aware Learned Index for Social Media Data

In the growing mass of social media data, how to efficiently extract the collection of interested concerns has become a research hotspot. Due to the large size and regularity of social media data, traditional indexing techniques are not applicable. Our “Learned Index”, which is a part of social media intelligence solutions, uses mathematical principles to summarize the laws from the data. It predicts the location of the data by learning the mathematical properties of the data distribution to build the model. Although existing methods over single dimension and multi-dimension such as setting gaps are proposed to further optimize the performance of index, they do not consider the update-distribution of data. In this paper, we propose an update-distribution-aware learned index for social media data (TALI) to support update operations and handle the data sliding. In TALI, underlying data are learned through machine learning models, and a recursive hierarchical model is built. It also learns the update-distribution of data to adjust the size of each leaf node. Thus, it can more effectively support all kinds of operations in databases due to the decrease of the leaf nodes’ sliding. In addition, TALI uses the model-based insertion method for bulkload and query, resulting in a small prediction error. Thus, exponential search is used to perform secondary lookup to improve query efficiency. Experiments were tested and compared on four realistic and synthetic social media datasets. Through extensive experiments, TALI performed better than the existing state-of-the-art learned index with less space occupancy on four realistic and synthetic social media datasets.