How to send Python logs to OpenSearch using FluentBit

Introduction

Efficient log management is essential for monitoring and maintaining applications in today's data-driven world. OpenSearch, an open source search and analytics engine, offers powerful log collection and analysis capabilities. By combining it with FluentBit, a lightweight log sender, we can send Python logs to OpenSearch for real-time monitoring and analysis. This guide will walk you through setting up this powerful combination.

This setup is beneficial for developers and system administrators who need to monitor Python applications in real time. By sending reports from Python applications to OpenSearch, you can analyze reports, create dashboards, set alerts, and gain valuable insights into the performance and behavior of your application. This approach is scalable and suitable for small projects and large-scale production environments.

Prerequisites

• Python installed: Make sure Python is installed on your server or local machine.
• OpenSearch Cluster: You need access to an OpenSearch cluster. You can create a new cluster or use an existing one.
• FluentBit installed: FluentBit must be installed on the server or machine where your Python program will run.

Step 1 – Install Python

First, update your repository and install Python. Run the following commands:

sudo apt update
sudo apt install python3

To verify the Python installation, run:

python3 --version

Step 2 – Set up Python logging

Configure your Python program to log messages to a file. Here is a basic setup:

import logging
# Configure logging
logging.basicConfig(
filename='app.log',
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
# Sample log message
logging.info('This is a test log message.')

This configuration writes log messages to app.log with a timestamp, log level, and message format.

Step 3 – Create OpenSearch Cluster

If you don't have an OpenSearch cluster, you can create one using the DigitalOcean command line tool, doctl. Run the following command:

doctl databases create opensearch-doks --engine opensearch --region your-region --size db-s-1vcpu-2gb --num-nodes 1

Replace your region with the region you want. Alternatively, you can use the DigitalOcean control panel to create a cluster manually. For details on how to do this, see this guide to creating OpenSearch clusters via the control panel.

Step 4 – Install FluentBit

To install FluentBit, use the following command:

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Step 5 – Configure FluentBit

Create a configuration file called fluent-bit.conf to define how Fluent Bit should read reports and send them to OpenSearch. Here is a sample configuration:

[SERVICE]
Flush 1
Daemon Off
Log_Level info
[INPUT]
Name tail
Path /path/to/your/log/file.log
Tag python_logs
Refresh_Interval 5
[OUTPUT]
Name opensearch
Match python_logs
Host your-opensearch-host
Port 25060
HTTP_User your-username
HTTP_Passwd your-password
Index your-index-name
tls On
Suppress_Type_Name On

• Path: Specify the path to your app.log.
• Host: Replace with your OpenSearch host.
• HTTP_User: Your OpenSearch username.
• HTTP_Passwd: Your OpenSearch password.
• Index: The name of the directory where logs are stored.

Step 6 – Run FluentBeat

Run FluentBit with the configuration file:

/opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf

Step 7 – Review reports in OpenSearch

To ensure that reports are coming in correctly, check your OpenSearch dashboard. You should see reports sent from FluentBeat.

Step 8 – Create an index template in OpenSearch dashboards

1. Log in to OpenSearch dashboards with your username and password.
2. Open the left menu and click Index Management in the Management section.
3. From the Indexes menu, click Create index.
4. Enter the index name, configure other settings as needed, and click Create.

Result

By following these steps, you have set up a system for sending Python logs to OpenSearch using Fluent Bit. This setup allows you to efficiently manage and analyze logs, and helps you effectively maintain and monitor your applications. With OpenSearch and Fluent Bit, you have a powerful real-time log analysis and monitoring solution designed to meet the needs of any Python-based application.