How to send logs from Fluent Bit to OpenSearch

Introduction

In this tutorial, you will learn how to send logs from Fluent Bit installed on a managed MongoDB running on an Ubuntu Droplet to a managed DigitalOcean OpenSearch database.

OpenSearch An open-source search and analytics suite that was created by forking Elasticsearch and Kibana. It supports real-time data ingestion, advanced searches, and strong security, making it ideal for e-commerce, IT monitoring, and finance applications. With features such as support for SQL queries, machine learning, and alerting, OpenSearch is continuously updated through active community development.

Fluent Bit Fluent Bit is a lightweight, open-source log processor and sender that collects, processes, and sends log data. It is designed for optimal performance and is suitable for resource-constrained environments such as containers and edge computing. Fluent Bit efficiently aggregates logs and sends them to multiple destinations, making it very popular in real-time logging and monitoring environments.

Prerequisites

Before you begin, make sure you have the following:

• An account on DigitalOcean With access to Managed OpenSearch.
• Basic knowledge of OpenSearch and Fluent Bit.
• Access to OpenSearch Dashboard.

Install Fluent Bit

Fluent Bit can be installed on multiple platforms. Here we are using Ubuntu platform. Run the following command in your Droplet terminal:

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

For more information, you can refer to the official documentation.

Fluent Bit Configuration

The Fluent Bit configuration file is located in the path by default. /etc/fluent-bit/fluent-bit.conf You need to modify this file to send logs to OpenSearch.

Fluent Bit Inputs

Fluent Bit provides a set of input plugins that allow it to collect log and event data from various sources. Since you are going to send logs from log files, you will need to use the input plugin. tail You will use.

File fluent-bit.conf Update as follows:

[INPUT] name tail Path /var/log/mongodb/mongod.log

To read more about input plugins, you can refer to the official Fluent Bit manual.

Fluent Bit Outputs

Like the input plugins, Fluent Bit provides an output plugin that sends the collected and processed logs to various destinations. Since we are going to send the logs to OpenSearch, we will use the OpenSearch output plugin.

[OUTPUT]
Name opensearch
Match *
Host opensearch hostname
port 25060
HTTP_User doadmin #DO Managed OpenSeach username
HTTP_Passwd XXXX #Managed OpenSeach Password
Index mongodb
tls On
Suppress_Type_Name On

For more information about output plugins, see the official Fluent Bit manual.

Once configured, start the Fluent Bit service:

systemctl enable fluent-bit.service
systemctl start fluent-bit.service
systemctl status fluent-bit.service

Access MongoDB logs in the OpenSearch dashboard

• Go to the OpenSearch dashboard in your DigitalOcean account.
• Using the hamburger icon, click on “Dashboard Management” in the admin section.
• Click on “Index patterns” and select “Create Index pattern”.
• Name the index template according to the name used in the file. fluent-bit.conf Enter.
• Once the index template is ready, you can view MongoDB logs from the “Discover” tab.

Result

Congratulations! You have successfully learned how to use Fluent Bit to collect and send self-hosted MongoDB logs to a managed OpenSearch database on DigitalOcean. In this tutorial, you learned how to configure Fluent Bit and create an index template in OpenSearch so that you can effectively monitor and analyze your MongoDB logs.