SemParser: A Semantic Parser for Log Analysis

Logs, being run-time information automatically generated by software, record system events and activities with their timestamps. Before obtaining more insights about the run-time status of the software, a fundamental step of log analysis, called log parsing, is employed to extract structured templates and parameters from the semi-structured raw log messages. However, current log parsers regard each message as a character string, ignoring the semantic information included in parameters and templates. Thus, we propose the semantic parser SemParser to unlock the critical bottleneck of mining semantics from log messages. It contains two steps, an end-to-end semantic miner and a joint parser. Specifically, the first step aims to identify explicit semantics inside a single log, and the second step is responsible for jointly inferring implicit semantics and computing structural outputs based on the contextual knowledge base. To analyze the effectiveness of our semantic parser, we first demonstrate that it can derive rich semantics from log messages collected from seven widely-applied systems with an average F1 score of 0.987. Then, we conduct two representative downstream tasks, showing that current downstream techniques improve their performance with appropriately extracted semantics by 11.7% and 8.65% in anomaly detection and failure diagnosis tasks, respectively. We believe these findings provide insights into semantically understanding log messages for the log analysis community.