Modern data analytics applications prefer to use column-storage formats due to their improved storage efficiency through encoding and compression. Parquet is the most popular file format for column data storage that provides several of these benefits out of the box. However, geospatial data is not readily supported by Parquet. This paper introduces Spatial Parquet, a Parquet extension that efficiently supports geospatial data. Spatial Parquet inherits all the advantages of Parquet for non-spatial data, such as rich data types, compression, and column/row filtering. Additionally, it adds three new features to accommodate geospatial data. First, it introduces a geospatial data type that can encode all standard spatial data types in a column format compatible with Parquet. Second, it adds a new lossless and efficient encoding method, termed FP-delta, that is customized to efficiently store geospatial coordinates stored in floating-point format. Third, it adds a light-weight spatial index that allows the reader to skip non-relevant parts of the file for increased read efficiency. Experiments on large-scale real data showed that SpatialParquet can reduce the data size by a factor of three, even without compression. Compression can further reduce the storage size. Additionally, Spatial Parquet can reduce the reading time by two orders of magnitude when the light-weight index is applied. This initial prototype can open new research directions to further improve geospatial data storage in column format.
翻译：现代数据分析应用由于通过编码和压缩提高了存储效率,而倾向于使用专列存储格式。 Parquet是专列数据存储的最受欢迎的文件格式,它提供了其中若干好处。 然而, Parquet 并不轻易支持地理空间数据。 本文介绍了空间 Parquet, 这是一种高效支持地理空间数据的Parquet扩展。 空间 Parquet 继承了 Parquet 的所有非空间数据优势, 例如丰富的数据类型、压缩和 列/ 浏览。 此外, 它增加了三个新功能, 以容纳地理空间数据。 首先, 它引入了一个最受欢迎的地理空间数据存储格式, 能够将所有标准空间数据类型编码在与 Parquet 兼容的专列格式中。 其次, 它增加了一种新的无损和高效的编码方法, 即PFP- delta, 它被定制为高效存储以浮动点格式存储地理空间坐标。 第三, 它增加了一个轻度的空间指数, 使读者可以跳过开放文档中非相关部分, 提高阅读效率。 大型实际数据的实验显示, 在与 PasimaParquequet 中可以进一步减少应用的数据大小, 即使不缩缩缩缩缩缩缩缩缩缩缩缩中的数据格式。
相关内容
微信扫码咨询专知VIP会员