lightlog.pdf (1.23 MB)
LightLog: A lightweight temporal convolutional network for log anomaly detection on the edge
journal contribution
posted on 2021-12-06, 12:30 authored by Zumin Wang, Jiyu Tian, Hui FangHui Fang, Liming Chen, Jing QinLog anomaly detection on edge devices is the key to enhance edge security when deploying IoT systems. Despite the success of many newly proposed deep learning based log anomaly detection methods, handling large-scale logs on edge devices is still a bottleneck due to the limited computational power on these devices to fulfil the real-time processing requirement for accurate anomaly detection. In this work, we propose a novel lightweight log anomaly detection algorithm, named LightLog, to tackle this research gap. In specific, we achieve real-time processing speed on the task via two aspects: (i) creation of a low-dimensional semantic vector space based on word2vec and post-processing algorithms (PPA); and (ii) design of a lightweight temporal convolutional network (TCN) for the detection. These two components significantly reduce the number of parameters and computations of a standard TCN while improving the detection performance. Experimental results show that our LightLog outperforms several benchmarking methods, namely DeepLog, LogAnomaly and RobustLog, by achieving 97.0 F1 score on HDFS Dataset and 97.2 F1 score on BGL with smallest model size. This effective yet efficient method paves the way to the deployment of log anomaly detection on the edge. Our source code and datasets are freely available on https://github.com/Aquariuaa/LightLog
Funding
Youth Fund Project of the National Natural Science Foundation of China under grant 62002038
History
School
- Science
Department
- Computer Science
Published in
Computer NetworksVolume
203Publisher
ElsevierVersion
- AM (Accepted Manuscript)
Rights holder
© ElsevierPublisher statement
This paper was accepted for publication in the journal Computer Networks and the definitive published version is available at https://doi.org/10.1016/j.comnet.2021.108616.Acceptance date
2021-11-08Publication date
2021-11-10Copyright date
2021ISSN
1389-1286Publisher version
Language
- en
Depositor
Dr Hui Fang. Deposit date: 9 November 2021Article number
108616Usage metrics
Categories
No categories selectedKeywords
Licence
Exports
RefWorks
BibTeX
Ref. manager
Endnote
DataCite
NLM
DC