Adopting log aggregation is an important step for an organization to take in a world of microservices and APIs. Several of the companies we work with use Splunk Cloud to aggregate their logs. Below is an example of how we do this.

One organization we work with runs MuleSoft in a Kubernetes cluster on AKS, so we created a custom container for the MuleSoft Runtime for them to use. This custom container runs as part of a StatefulSet with persistent volume claims for persistent runtime data.

After using MuleSoft for a short time, they decided to adopt Splunk Cloud. We had to determine how we would send the logs to Splunk. We could do one of the following:

  1. Send directly via HTTP/S
  2. Send via the Splunk Universal Forwarder

We decided to use the Splunk Universal Forwarder because it handles error scenarios easier than sending directly via HTTP/S. For instance, what happens when there is a connectivity problem between the MuleSoft runtime and Splunk Cloud? We deployed the official container in the StatefulSet the MuleSoft container was running in and gave it access to the MuleSoft log files.

Configuring the Universal Forwarder Container

To configure the Splunk Universal Forwarder container (splunk/universalforwarder, you need to set up a few environment variables. Additionally, to use Splunk Cloud, download the Splunk Cloud credentials to install them in the forwarder. See below for the environment variable values:

  • SPLUNK_START_ARGS must include --accept-license to accept the Splunk license agreement.
  • SPLUNK_PASSWORD is the password to be used for this forwarder instance.
  • SPLUNK_CMD is a list of commands that will get executed upon container startup. Install the Splunk Cloud credentials to forward the logs. Use this command: install app /opt/splunk/uf/splunkclouduf.spl
  • SPLUNK_ADD is a comma-separated list of splunk add CLI commands to add monitors on files so their contents will be sent to Splunk.

How do the Splunk Cloud credentials get into the container? Add the splunkclouduf.spl file as a binary Secret and add it to the Splunk container as a volume mount.

Potential Problems

We did run into one issue while configuring the Universal Forwarder. A bug in the forwarder replaced a $_ in the password with /bin/python, so the password should have read something like super$_cretPassword, instead read super/bin/pythoncretPassword.

The easiest way to correct the problem is to change the password, so it doesn’t include $_ in it and download a new Splunk Cloud credentials file.

There’s an alternative method to correct the password if you’re unable to change it.  You can set the SPLUNK_ANSIBLE_POST_TASKS environment variable to call custom Ansible code to update the sslPassword in outputs.conf to the correct value. See Putting It All Together for a sample of the Ansible code to update the sslPassword.

Putting It All Together

Here is an example k8s YAML with some unrelated details left out:


apiVersion: apps/v1
kind: StatefulSet
# ...
spec:
  serviceName: mule
  # ...
  template:
    # ...
    spec:
      # ...
      containers:
      - name: mule
        image: customMuleImage
        # Container configuration
        volumeMounts:
        - name: mule
          mountPath: /mule
      - name: splunk
        image: splunk/universalforwarder:8.0
        env:
        - name: SPLUNK_START_ARGS
          value: --accept-license
        - name: SPLUNK_PASSWORD
          value: super_secret_password
        - name: SPLUNK_ADD
          value: "monitor /mule/logs/my-api.log, monitor /mule/logs/my-other-api.log"
        - name: SPLUNK_CMD
          value: "install app /opt/splunk/uf/splunkclouduf.spl"
        - name: SPLUNK_ANSIBLE_POST_TASKS
          value: "file:///opt/company/reset_splunk_cloud_password.yml"
        - name: CONFIG_PATH
          value: "/opt/splunk/uf/splunkclouduf.spl"
        - name: APP_NAME
          value: "100_company_splunkcloud"
        volumeMounts:
        - name: splunk-config
          mountPath: /opt/splunk/uf
        - name: mule
          mountPath: /mule
        - name: splunk-custom-ansible
          mountPath: /opt/company
        # Additional container configuration
      volumes:
      - name: splunk-config
        secret:
          secretName: splunk-config-spl
    volumeClaimTemplates:
    - metadata:
        name: mule
      spec:
        # volume claim details
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: splunk-custom-ansible
data:
  reset_splunk_cloud_password.yml: |-
    ---
    - name: Update sslCertPath to clientCert
      shell:
        cmd: "sed -i 's/sslCertPath/clientCert/' outputs.conf"
        chdir: "/etc/apps//default"
      become: yes
      become_user: ""
    - name: Get Plaintext Password
      shell:
        cmd: "tar xf  /local/outputs.conf && grep sslPassword /local/outputs.conf | cut -c15-"
        chdir: "/etc/apps/"
      become: yes
      become_user: ""
      register: plaintext_password
    - name: Encrypt Password
      shell:
        cmd: " show-encrypted --value ''"
        chdir: "/etc/apps"
      become: yes
      become_user: ""
      register: encrypt_password
    - name: Update Password
      lineinfile:
        path: "/etc/apps//local/outputs.conf"
        regexp: '^sslPassword'
        line: "sslPassword = "
      become: yes
      become_user: ""
      notify:
        - Restart the splunkd service