[mabrarov]

Implementation of Upstream feature for the Elasticsearch output plugin.

This pull request is based on pull request #1560 and Forward output plugin.

It was tested in a local setup with:

1. Fluent Bit without Upstream feature connected to a single node of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.

   Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.

   fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es/fluent-bit.conf and (same in YAML format) fluent-bit-es/fluent-bit.yaml in mabrarov/elastic-stack repository.

   Debug log is available at flb_es.log.

2. Fluent Bit with Upstream feature connected to all Elasticsearch data nodes of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.

   Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.

   fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es-cluster/fluent-bit.conf and (same in YAML format) fluent-bit-es-cluster/fluent-bit.yaml in mabrarov/elastic-stack repository.

   Debug log is available at flb_es_upstream.log.

---

Testing

• Example configuration files for the change can be found in mabrarov/elastic-stack repository under fluent-bit-es-cluster directory.
• Debug log output from testing the change - see above.
• Attached Valgrind output that shows no leaks or memory corruption was found - refer to flb_run_code_analysis.log for the output of command

  TEST_PRESET=valgrind SKIP_TESTS='flb-rt-out_forward flb-rt-out_td' ./run_code_analysis.sh

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature - refer to es_out: support Upstream Servers with configuration overriding fluent-bit-docs#1143.

Backporting

• [N/A] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • HA-aware Elasticsearch output with per-upstream/per-node configuration and upstream-aware formatter flush-context callbacks.

• Integrations

  • Cloud ID parsing for Elasticsearch endpoints.
  • Optional AWS SigV4 support with STS and unsigned-headers handling.

• Configuration

  • Consolidated ES property names, added upstream option, and corrected default Elasticsearch port to 9200.

• Quality

  • Ownership-aware resource handling and improved lifecycle/cleanup.

• Tests

  • Expanded upstream-focused tests covering formatter, flush-context, and config scenarios.

[mabrarov]

Hi reviewers,

Is it possible to approve only workflow for this pull request, so that automated checks and build can start?

Thank you.

[PettitWesley]

@mabrarov sure

[mabrarov]

Hi @PettitWesley,

It looks like all failed checks are around run-macos-unit-tests jobs and caused by the following failed unit tests:

1. flb-rt-in_event_test
2. flb-rt-out_tcp

I feel like other pull requests have the same issues, i.e. it doesn't seem that the failed checks are caused by this pull request changes.

Help of maintainers is appreciated.

Thank you.

[mabrarov]

Hi @PettitWesley,

Is it possible to trigger automated workflow (build) for this pull request one more time? I found & fixed one issue and added tests for the new code since last build happened.

Thank you.

[mabrarov]

Hi dear reviewers,

Is it possible to get this pull request reviewed / accepted sooner? Is there something pending / waiting from my side to start review?

Thank you.

[mabrarov]

Hi @PettitWesley and @edsiper,

It feels like you are code owners for Elasticsearch output plugin. Is there something pending / waiting from my side to start review of this pull request? This new feature was requested 4 years ago and I feel it is something which multiple users of Fluent Bit (not just my team) would like to have.

Thank you.

[mabrarov]

Hi @cosmo0920 and @patrick-stephens,

This PR is pending for more than 2.5 years. I appreciate if you could find time and review it again. I believe I have addressed comment of @cosmo0920 which was the only concern at the time the comment was added.

My winter vacation has ended and I won't be able to spend much time on this PR till summer, especially, considering new pending PRs for the same plugin (e.g. #11422 and #11375) - reabasing such changes can be hard.

Thank you.

[mabrarov]

I believe that similar changes can be implemented / helpful for OpenSearch output plugin (in a dedicated PR).

[edsiper]

closing in favor of #11647

[mabrarov]

Hi @edsiper,

Is performance tradeoff of #11647 (repeating same operations during every flush) - compared to this PR - considered acceptable? Doesn't it contradict the mission of Fluent Bit?

Thank you.

[patrick-stephens]

Hi @edsiper,

Is performance tradeoff of #11467 (repeating same operations during every flush) - compared to this PR - considered acceptable? Doesn't it contradict the mission of Fluent Bit?

Thank you.

I think you meant #11647 @mabrarov :)

@cosmo0920 is working on that so may have some comments around it.

[mabrarov]

If somebody is interested in alternative (providing more features and being more lightweight) for #11647, then I rebased source branch of this PR onto master branch, including removal of changes introduced by #11647. I also updated files referenced / linked in description of this PR and updated fluent/fluent-bit-docs#1143 (fluent/fluent-bit-docs#1143 (comment)) to match source branch of this PR.

It would be nice if I could reopen this PR (to keep the diff visible and to keep all links to this PR working and providing the latest state / diff) as draft (to not disturb Fluent Bit team which already made its choice in favor of #11647), but I failed to find an option to do it.