splunk-forwarder-operator

This operator manages Splunk Universal Forwarder. It deploys a daemonset which deploys a pod on each node including the masters. It expects the service account for the namespace can deploy privileged pods. It also needs a secret that holds the forwarder auth.

If you are using Splunk Cloud, credentials can be obtained by downloading a credentials package from the specific Splunk application being used, such as the Universal Forwarder app. The credentials package is a tarball, so first extract the contents with tar xvf splunkclouduf.spl, then add the following fields in outputs.conf

sslCertPath = $SPLUNK_HOME/etc/apps/splunkauth/default/server.pem
sslRootCAPath = $SPLUNK_HOME/etc/apps/splunkauth/default/cacert.pem
sslPassword = <Your SSL Password>

Then create a secret named "splunk-auth" using the extracted spl files and modified outputs.conf:

oc create secret generic splunk-auth --dry-run=client -o yaml \
  --from-file=cacert.pem=/path/to/spl/cacert.pem \
  --from-file=limits.conf=/path/to/spl/limits.conf \
  --from-file=outputs.conf=/path/to/spl/outputs.conf \
  --from-file=server.pem=/path/to/spl/server.pem

The SplunkForwarder CRD explicitly points to the files you want to monitor (currently only supports monitor://).

apiVersion: splunkforwarder.managed.openshift.io/v1alpha1
kind: SplunkForwarder
metadata:
  name: example-splunkforwarder
spec:
  image: dockerimageurl
  imageDigest: sha256:ee36e2f76856b262fb19096c0bba04fcf7e4399c7819d914d9b8554f119ab42a
  splunkLicenseAccepted: true
  clusterID: optional-cluster-name
  splunkInputs:
  - path: /host/var/log/openshift-apiserver/audit.log
    index: openshift_managed_audit
    whitelist: \.log$
    sourcetype: _json
  - path: /host/var/log/containers/ip-*-*-*-*ec2internal-debug*.log
    index: openshift_managed_debug_node
    whitelist: \.log$
    sourcetype: linux_audit

The image and imageDigest are for the splunk-forwarder image. (The CRD supports imageTag, but this is deprecated in favor of imageDigest.)

To use the current version, 9.3.2-d8bb32809498-c43b7a7, specify the following:

• For splunk-forwarder:

  image: quay.io/app-sre/splunk-forwarder
  imageDigest: sha256:ee36e2f76856b262fb19096c0bba04fcf7e4399c7819d914d9b8554f119ab42a

Upgrading Splunk Universal Forwarder

Run make image-update to update to the current master branch commit of splunk-forwarder-images.

This process will update the Makefile with a new value for FORWARDER_IMAGE_TAG (from the forwarder version, forwarder hash and commit hash) and populate the OLM template with the by-digest URIs for that version.

To use a specific version, use make SFI_UPDATE=<commit/branch/etc> image-update or edit the Makefile by hand and run make image-digests to update the OLM template.

Commit and propose the changes as usual.

Building and Testing

app-sre pipeline

This repository is configured to support the testing strategy documented here.

Note that, in addition to creating personal repositories for the operator and OLM registry, you must also create them for splunk-forwarder.

Operator development requirements

golang

A recent Go distribution (>=1.17) with enabled Go modules.

$ go version
go version go1.17.11 linux/amd64

operator-sdk

The Operator is developed using the Operator SDK. Ensure this is installed and available in your $PATH.

v1.21.0 is the minimum-verified version required for splunk-forwarder-operator development.

OperatorSDK releases are avaiable here.

$ operator-sdk version
operator-sdk version: "v1.21.0", commit: "89d21a133750aee994476736fa9523656c793588", kubernetes version: "1.23", go version: "go1.17.10", GOOS: "linux", GOARCH: "amd64"

Local testing

To run the operator in a local environment (not via a pod running on-cluster), ensure that the following environment variables are set:

export OPERATOR_NAMESPACE=openshift-splunk-forwarder-operator
export WATCH_NAMESPACE=""
export OSDK_FORCE_RUN_MODE="local"