Adopting log aggregation is an important step for an organization to take in a world of microservices and APIs. Several of the companies we work with use Splunk Cloud to aggregate their logs. Below is an example of how we do this.
One organization we work with runs MuleSoft in a Kubernetes cluster on AKS, so we created a custom container for the MuleSoft Runtime for them to use. This custom container runs as part of a StatefulSet with persistent volume claims for persistent runtime data.
After using MuleSoft for a short time, they decided to adopt Splunk Cloud. We had to determine how we would send the logs to Splunk. We could do one of the following:
- Send directly via HTTP/S
- Send via the Splunk Universal Forwarder
We decided to use the Splunk Universal Forwarder because it handles error scenarios easier than sending directly via HTTP/S. For instance, what happens when there is a connectivity problem between the MuleSoft runtime and Splunk Cloud? We deployed the official container in the StatefulSet the MuleSoft container was running in and gave it access to the MuleSoft log files.
Configuring the Universal Forwarder Container
To configure the Splunk Universal Forwarder container (splunk/universalforwarder, you need to set up a few environment variables. Additionally, to use Splunk Cloud, download the Splunk Cloud credentials to install them in the forwarder. See below for the environment variable values:
- SPLUNK_START_ARGS must include --accept-license to accept the Splunk license agreement.
- SPLUNK_PASSWORD is the password to be used for this forwarder instance.
- SPLUNK_CMD is a list of commands that will get executed upon container startup. Install the Splunk Cloud credentials to forward the logs. Use this command: install app /opt/splunk/uf/splunkclouduf.spl
- SPLUNK_ADD is a comma-separated list of splunk add CLI commands to add monitors on files so their contents will be sent to Splunk.
How do the Splunk Cloud credentials get into the container? Add the splunkclouduf.spl file as a binary Secret and add it to the Splunk container as a volume mount.
We did run into one issue while configuring the Universal Forwarder. A bug in the forwarder replaced a $_ in the password with /bin/python, so the password should have read something like super$_cretPassword, instead read super/bin/pythoncretPassword.
The easiest way to correct the problem is to change the password, so it doesn’t include $_ in it and download a new Splunk Cloud credentials file.
There’s an alternative method to correct the password if you’re unable to change it. You can set the SPLUNK_ANSIBLE_POST_TASKS environment variable to call custom Ansible code to update the sslPassword in outputs.conf to the correct value. See Putting It All Together for a sample of the Ansible code to update the sslPassword.
Putting It All Together
Here is an example k8s YAML with some unrelated details left out:
apiVersion: apps/v1 kind: StatefulSet # ... spec: serviceName: mule # ... template: # ... spec: # ... containers: - name: mule image: customMuleImage # Container configuration volumeMounts: - name: mule mountPath: /mule - name: splunk image: splunk/universalforwarder:8.0 env: - name: SPLUNK_START_ARGS value: --accept-license - name: SPLUNK_PASSWORD value: super_secret_password - name: SPLUNK_ADD value: "monitor /mule/logs/my-api.log, monitor /mule/logs/my-other-api.log" - name: SPLUNK_CMD value: "install app /opt/splunk/uf/splunkclouduf.spl" - name: SPLUNK_ANSIBLE_POST_TASKS value: "file:///opt/company/reset_splunk_cloud_password.yml" - name: CONFIG_PATH value: "/opt/splunk/uf/splunkclouduf.spl" - name: APP_NAME value: "100_company_splunkcloud" volumeMounts: - name: splunk-config mountPath: /opt/splunk/uf - name: mule mountPath: /mule - name: splunk-custom-ansible mountPath: /opt/company # Additional container configuration volumes: - name: splunk-config secret: secretName: splunk-config-spl volumeClaimTemplates: - metadata: name: mule spec: # volume claim details --- apiVersion: v1 kind: ConfigMap metadata: name: splunk-custom-ansible data: reset_splunk_cloud_password.yml: |- --- - name: Update sslCertPath to clientCert shell: cmd: "sed -i 's/sslCertPath/clientCert/' outputs.conf" chdir: "/etc/apps//default" become: yes become_user: "" - name: Get Plaintext Password shell: cmd: "tar xf /local/outputs.conf && grep sslPassword /local/outputs.conf | cut -c15-" chdir: "/etc/apps/" become: yes become_user: "" register: plaintext_password - name: Encrypt Password shell: cmd: " show-encrypted --value ''" chdir: "/etc/apps" become: yes become_user: "" register: encrypt_password - name: Update Password lineinfile: path: "/etc/apps//local/outputs.conf" regexp: '^sslPassword' line: "sslPassword = " become: yes become_user: "" notify: - Restart the splunkd service