Assessing the Maturity of Model Maintenance Techniques for AIOps Solutions

AIOps (Artificial Intelligence for IT Operations) solutions leverage the massive data produced during the operations of large-scale systems and machine learning models to assist software engineers in their system operations. As operation data produced in the field are subject to constant evolution from factors like the changing operational environment and user base, the models in AIOps solutions need to be constantly maintained after deployment. While prior works focus on innovative modeling techniques to improve the performance of AIOps models before releasing them into the field, when and how to maintain AIOps models remain an under-investigated topic. In this work, we performed a case study on three large-scale public operation data to assess different model maintenance approaches regarding their performance, maintenance cost, and stability. We observed that active model maintenance approaches achieve better and more stable performance than a stationary approach. Particularly, applying sophisticated model maintenance approaches (e.g., concept drift detection, time-based ensembles, or online learning approaches) could provide better performance, efficiency, and stability than simply retraining AIOps models periodically. In addition, we observed that, although some maintenance approaches (e.g., time-based ensemble and online learning) can save model training time, they significantly sacrifice model testing time, which could hinder their applications in AIOps solutions where the operation data arrive at high speed and volume and where instant predictions are required. Our findings highlight that practitioners should consider the evolution of operation data and actively maintain AIOps models over time. Our observations can also guide researchers and practitioners to investigate more efficient and effective model maintenance techniques that fit in the context of AIOps.