Rainbond integrates ELK/EFK through plug-ins to realize log collection

ELK is：acronym for three open source projects1 Elasticsearch, Logstash and Kibana.However, FileBeat, which appeared later, can completely replace the data collection function of Logstash, and it is also relatively lightweight.This article will cover EFK: Elasticsearch, Filebeat and Kibana :::

foreword

ELK is an acronym for three open source projects：Elasticsearch, Logstash, and Kibana.However, FileBeat, which appeared later, can completely replace the data collection function of Logstash, and it is also relatively lightweight.This article will cover EFK: Elasticsearch, Filebeat and Kibana

Elasticsearch：is a distributed search and analysis engine with high scalability, high reliability and easy management.Built on Apache Lucene, it can perform near real-time storage, search and analysis operations on large volumes of data.Often used as a basic search engine for some applications, giving it complex search capabilities;

Kibana：data analysis and visualization platform.Use with Elasticsearch to search, analyze and display data in statistical charts;

Filebeat：Filebeat is a lightweight transporter for forwarding and centralizing log data.Filebeat is installed on your server as an agent that monitors log files or locations you specify, collects log events, and forwards them to Elasticsearch or Logstash for indexing.

Through this article, learn how to collect application logs and send them to Elasticsearch by enabling the FileBeat plugin for applications running on Rainbond.

Integrated Architecture

When collecting logs, you need to enable the FileBeat plugin in the application for collection. FileBeat collects logs in three ways：

1. Specify the log path
2. Collect all container logs
3. Specify Label Autodiscover

This article uses the specified log path for collection. In this way, we can customize the rules for collecting logs, etc.

We made FileBeat as Rainbond's general type plug-in After the application is started, the plug-in also starts and automatically collects logs and sends them to Elasticsearch. The whole process is non-intrusive to the application container and has strong scalability.Similar methods can be used to connect to other log collection tools. Users can connect to different log collection tools by replacing plug-ins.

The figure below shows the structure of using the FileBeat plugin to collect application logs in Rainbond and send them to Elasticsearch.

Analysis of the principle of plug-in implementation

The Rainbond plug-in system is a part of the Rainbond application model. The plug-in is mainly used to realize the extended operation and maintenance capabilities of the application container.Because the implementation of operation and maintenance tools has a large commonality, the plug-in itself can be reused.Plugins have runtime status only when they are bound to the application container to implement an operation and maintenance capability, such as performance analysis plugins, network governance plugins, and initialization type plugins.

The runtime environment of a plug-in with runtime is consistent with the bound components in the following aspects：

• Cyberspace is a crucial feature. Consistent cyberspace enables plug-ins to bypass monitoring and interception of component network traffic, set component local domain name resolution, and so on.
• Storage Persistence Space This feature enables file exchange between plugins and components through the persistence directory.
• environment variables This feature enables plugins to read the component's environment variables.

In the process of making the FileBeat plug-in, the general type plug-inis used, which can be understood as one POD starts two Containers. Kubernetes natively supports starting multiple Containers in one POD, but the configuration is relatively complicated. User operation is simple.

One-click installation of EK via the Rainbond app store

We have made elasticsearch + Kibana as an application and released it to the application market, users can install it with one click based on the open source application store.

1. Install Rainbond
2. Search for elasticsearch in the open source application store, click install to install it with one click;

3. elasticsearch has the xpack security module enabled by default to secure our cluster, so we need an initial password.We enter the elasticsearch web terminal and execute the command as shown below, run the bin/elasticsearch-setup-passwords command in the web terminal to generate the default username and password：

bin/elasticsearch-setup-passwords parameter
auto generate automatically
interactive fill in manually

4. Enter the environment variables of the Kibana component and modify the default connection elasticsearchenvironment variable ELASTICSEARCH_PASSWORD.

Collect application logs

Use Nginx as the demo application of this article, and use the mirror to create components on Rainbond,

• Mirror address：nginx:latest
• Mount the storage：/var/log/nginxto persist the Nginx log, and the Filebeat plugin can read the log file.

Make a FileBeat plugin

On the Rainbond team interface, click Plug-in to enter the Plug-in interface, click New Plug-in, and create a general type of plug-in.

• Image address：docker.elastic.co/beats/filebeat:7.15.2
• Other customizations are available.

Create the plug-in and build it. After the build is successful, we enable the FileBeat plug-in in the plug-in of the Nginx component.

In the environment configuration of the Nginx component, add the FileBeat configuration file as follows. For more configuration, please refer to official document

• Configuration file mount path：/usr/share/filebeat/filebeat.yml
• Profile permissions：644

filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/*.log
output.elasticsearch:
  hosts: '127.0.0.1:9200'
  username: "elastic"
  password: "elastic "

build dependencies

Establish a dependency between Nginx and elasticsearch, so that it can communicate with elasticsearch through the 127.0.0.1address, and update the Nginx component to make the dependency take effect.

Visit Kibana

Kibana has been localized by default

1. Click Stack Management > Index Management, you can see that our filebeat index already exists.

2. Stack Management > index patterns, create filebeat index patterns.

3. Discover page to see log information.

Summarize

The Rainbond-based plug-in mechanism is combined with EFK, allowing users to quickly collect application logs for analysis throughEFK, and flexibly replace plug-in FileBeat with Logstash.

In addition, Rainbond's plug-in mechanism is open, and the application governance function can be extended through the plug-in mechanism, such as network management and data backup plug-ins. In the case of no invasion of the original application logic, network management plug-ins can be used. Analyze the performance of services and connect to log collection systems such as ELK; for components such as databases, use backup plug-ins to back up data.