[DO NOT MERGE BEFORE #1606 and #1610] CSPL-3549 Splunk Operator Enhancement – Ingestion and Indexing Separation

[kasiakoziol]

Depends on

#1606

#1610

Description

This PR implements a separation between ingestion and indexing services within the Splunk Operator for Kubernetes. The goal is to enable the operator to independently manage the ingestion service while maintaining seamless integration with the indexing service.

Key Changes

• Introduction of new Custom Resource Definition for IngestorCluster and BusConfiguration
• Support for Remote Queue and Pipeline inputs in IngestorCluster and IndexerCluster via BusConfiguration reference

Testing and Verification

These changes were tested manually and via dedicated automated tests.

Helm and int tests run: https://github.com/splunk/splunk-operator/commits/CSPL-3551-ingestion-cr-int-helm-tests-run

Related Issues

Jira Epic: https://splunk.atlassian.net/browse/CSPL-3549

Notes

• Enhance documentation to reflect on the changes (Done - docs/IndexIngestionSeparation.md)
• Address TODOs (Done - only role update to splunk_ingestor left till new Docker image with this Ansible role gets released (CSPL-4002))
• Implement and enhance helm charts (CSPL-4003)
• Implement automated tests
  • Unit tests
  • Integration tests (CSPL-3558)
• Refactor
  • If message bus type changes, then there are two configs kept in the .conf files
  • .conf files get updated with all values, not only values that changed
• Separate PushBus and PipelineConfig into a separate resource (CSPL-4022)
• Do we need to support migration? As of now, only new customers are supported. Document it.

PR Checklist

• Code changes adhere to the project's coding standards.
• Relevant unit and integration tests are included.
• Documentation has been updated accordingly.
• All tests pass locally.
• The PR description follows the project's guidelines.

[coveralls]

Pull Request Test Coverage Report for Build 19165600558

Details

• 545 of 798 (68.3%) changed or added relevant lines in 12 files are covered.
• 5 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-1.0%) to 85.565%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/splunk/enterprise/types.go	6	8	75.0%
pkg/splunk/enterprise/util.go	18	26	69.23%
pkg/splunk/client/enterprise.go	45	54	83.33%
internal/controller/indexercluster_controller.go	14	25	56.0%
pkg/splunk/enterprise/busconfiguration.go	71	84	84.52%
internal/controller/ingestorcluster_controller.go	74	96	77.08%
internal/controller/busconfiguration_controller.go	28	52	53.85%
pkg/splunk/enterprise/indexercluster.go	90	143	62.94%
pkg/splunk/enterprise/ingestorcluster.go	186	297	62.63%

Files with Coverage Reduction	New Missed Lines	%
pkg/splunk/enterprise/cp.go	1	33.33%
pkg/splunk/enterprise/indexercluster.go	4	71.23%

Totals
Change from base Build 19137368717:	-1.0%
Covered Lines:	11262
Relevant Lines:	13162

---

💛 - Coveralls