The goal is to build an OpenStack toolchain for monitoring and troubleshooting system and service logs. The toolchain will mainly consist of the ELK (hence the name of the project) stack: ElasticSearch, LogStash, Kibana and Filebeat.
Debugging OpenStack is not easy. Information about the OpenStack events is spread across multiple log files owned by multiple services. Trying to resolve an issue manually becomes a very lengthy and tedious task, that requires tracing information from many sources. The idea of bringing in the ELK stack into OpenStack is not new. Many companies are already successfully using the same tools for troubleshooting. In OPNFV however this space has not been filled yet. So the purpose of Cervus will be to build a flexible monitoring and troubleshooting system that will allow easy navigation and information querying from multiple sources, such as syslog, running OpenStack services, OVS, Fuel, etc. The system will provide a web based user interface for navigation and can be used with multiple cloud deployments at the same time.
ELK Stack has the following generic architecture:
Filebeat is a next generation Logstash forwarder, which tails logs and quickly sends this information to Logstash for further parsing and enrichment.
Logstash is a tool for managing events and logs. It gathers the messages, converts them into json documents, and stores them in an Elasticsearch cluster
Elasticsearch is a multi-node Lucene implementation.
Kibana is a front-end client to filter and display messages from the Elasticsearch cluster.
When it comes to deployment in OPNFV, we can identify two main actors:
- The agent, which basically is a Filebeat daemon configured to work with various logs. The target deployment shall contain an agent on each node.
- The ELK container - a docker container that runs on the Jumphost along with Fuel. It's a passive actor that receives data streams from the deployed agents.
In order to automate deployment of the agents and ELK container the following steps must be in place.
Fuel Plugin for Filebeats
Since an agent has to be deployed on each node during the installation phase, it's best to make it part of the overall installation done by an installer. In this particular case, the installer is Fuel, which has a flexible plugin system for deployment customizations.
The plugin should be able to install and configure a Filebeat agent on each target node. Basic configuration and installation scripts is available here .
Docker Container for ELK Stack
This activity will cover automated deployment of a Docker container with installed and configured Logstash, Elasticsearch and Kibana. The scripts are already implemented , but the Dockerfile has to be written.
[UPDATE] Done. Dockerfile has been added to the elkstack repository.
Integration in CI
Once the above parts are in place, we need to integrate in CI deployment of the ELK container on Jumphost as well as automatically enable Filebeat plugin in Fuel (not sure if this is needed though)