A text can be viewed via different representations, i.e. as a sequence of letters, n-grams of letters, syllables, words, and phrases. Here we study the optimal noiseless compression of texts using the Huffman code, where the alphabet of encoding coincides with one of those representations. We show that it is necessary to account for the codebook when compressing a single text. Hence, the total compression comprises of the optimally compressed text -- characterized by the entropy of the alphabet elements -- and the codebook which is text-specific and therefore has to be included for noiseless (de)compression. For texts of Project Gutenberg the best compression is provided by syllables, i.e. the minimal meaning-expressing element of the language. If only sufficiently short texts are retained, the optimal alphabet is that of letters or 2-grams of letters depending on the retained length.
翻译:文本可以通过不同的表达方式, 即字母、 n 克字母、 音频、 单词和短语的顺序来看待。 在这里, 我们研究使用Huffman 代码对文本进行最佳无噪音压缩, 编码的字母字母与其中的一个表达方式相吻合。 我们显示, 在压缩单一文本时, 有必要对代码簿进行核算。 因此, 压缩总量包括最优化压缩的文本 -- -- 以字母元素的缩写为特征 -- -- 以及文本专用的代码簿, 并且必须包含无噪音( de) 压缩。 对于 Gutenberg 项目的文本, 最好的压缩由符号提供, 即语言的最小表示意涵元素。 如果只保留足够短的文本, 则根据保留长度, 最优的字母或2克字母的缩写方式。