Two competing file formats have become the de facto standards for distributing pre-trained word embeddings. Both are named after the most popular pre-trained embeddings that are distributed in that format. The GloVe format is an entirely text based format that suffers from huge file sizes and slow reads, and the word2vec format is a smaller binary format that mixes a textual representation of words with a binary representation of the vectors themselves. Both formats have problems that we solve with a new format we call the Leader format. We include a word length prefix for faster reads while maintaining the smaller file size a binary format offers. We also created a minimalist library to facilitate the reading and writing of various word vector formats, as well as tools for converting pre-trained embeddings to our new Leader format.
翻译:两种相互竞争的文件格式已成为分发预先训练的字嵌入器的实际标准。 两种格式都以最受欢迎的预先训练的嵌入器命名。 GloVe 格式是一种完全基于文本的格式, 它有巨大的文件大小, 读数缓慢, 而 word2vec 格式是一个较小的二进制格式, 将文字表达方式与矢量本身的二进制表达方式混在一起。 两种格式都有我们用新格式( 我们称之为“ 领导” 格式) 解决的难题。 在维持较小的文件大小的二进制格式提供时, 我们包含一个字长的前缀, 用于更快的读数 。 我们还创建了一个最小化的图书馆, 方便阅读和撰写各种字矢量格式, 以及将预先训练的嵌入器转换为新“ 领导” 格式的工具 。