Adopting log aggregation is an important step for an organization to take in a world of microservices and APIs. Several of the companies we work with use Splunk Cloud to aggregate their logs. Below is an example of how we do this.
One organization we work with runs MuleSoft in a Kubernetes cluster on AKS, so we created a custom container for the MuleSoft Runtime for them to use. This custom container runs as part of a StatefulSet with persistent volume claims for persistent runtime data.
After using MuleSoft for a short time, they decided to adopt Splunk Cloud. We had to determine how we would send the logs to Splunk. We could do one of the following:
- Send directly via HTTP/S
- Send via the Splunk Universal Forwarder
We decided to use the Splunk Universal Forwarder because it handles error scenarios easier than sending directly via HTTP/S. For instance, what happens when there is a connectivity problem between the MuleSoft runtime and Splunk Cloud? We deployed the official container in the StatefulSet the MuleSoft container was running in and gave it access to the MuleSoft log files.
Configuring the Universal Forwarder Container
To configure the Splunk Universal Forwarder container (splunk/universalforwarder, you need to set up a few environment variables. Additionally, to use Splunk Cloud, download the Splunk Cloud credentials to install them in the forwarder. See below for the environment variable values:
- SPLUNK_START_ARGS must include --accept-license to accept the Splunk license agreement.
- SPLUNK_PASSWORD is the password to be used for this forwarder instance.
- SPLUNK_CMD is a list of commands that will get executed upon container startup. Install the Splunk Cloud credentials to forward the logs. Use this command: install app /opt/splunk/uf/splunkclouduf.spl
- SPLUNK_ADD is a comma-separated list of splunk add CLI commands to add monitors on files so their contents will be sent to Splunk.
How do the Splunk Cloud credentials get into the container? Add the splunkclouduf.spl file as a binary Secret and add it to the Splunk container as a volume mount.
Potential Problems
We did run into one issue while configuring the Universal Forwarder. A bug in the forwarder replaced a $_ in the password with /bin/python, so the password should have read something like super$_cretPassword, instead read super/bin/pythoncretPassword.
The easiest way to correct the problem is to change the password, so it doesn’t include $_ in it and download a new Splunk Cloud credentials file.
There’s an alternative method to correct the password if you’re unable to change it. You can set the SPLUNK_ANSIBLE_POST_TASKS environment variable to call custom Ansible code to update the sslPassword in outputs.conf to the correct value. See Putting It All Together for a sample of the Ansible code to update the sslPassword.
Putting It All Together
Here is an example k8s YAML with some unrelated details left out:
apiVersion: apps/v1
kind: StatefulSet
# ...
spec:
serviceName: mule
# ...
template:
# ...
spec:
# ...
containers:
- name: mule
image: customMuleImage
# Container configuration
volumeMounts:
- name: mule
mountPath: /mule
- name: splunk
image: splunk/universalforwarder:8.0
env:
- name: SPLUNK_START_ARGS
value: --accept-license
- name: SPLUNK_PASSWORD
value: super_secret_password
- name: SPLUNK_ADD
value: "monitor /mule/logs/my-api.log, monitor /mule/logs/my-other-api.log"
- name: SPLUNK_CMD
value: "install app /opt/splunk/uf/splunkclouduf.spl"
- name: SPLUNK_ANSIBLE_POST_TASKS
value: "file:///opt/company/reset_splunk_cloud_password.yml"
- name: CONFIG_PATH
value: "/opt/splunk/uf/splunkclouduf.spl"
- name: APP_NAME
value: "100_company_splunkcloud"
volumeMounts:
- name: splunk-config
mountPath: /opt/splunk/uf
- name: mule
mountPath: /mule
- name: splunk-custom-ansible
mountPath: /opt/company
# Additional container configuration
volumes:
- name: splunk-config
secret:
secretName: splunk-config-spl
volumeClaimTemplates:
- metadata:
name: mule
spec:
# volume claim details
---
apiVersion: v1
kind: ConfigMap
metadata:
name: splunk-custom-ansible
data:
reset_splunk_cloud_password.yml: |-
---
- name: Update sslCertPath to clientCert
shell:
cmd: "sed -i 's/sslCertPath/clientCert/' outputs.conf"
chdir: "/etc/apps//default"
become: yes
become_user: ""
- name: Get Plaintext Password
shell:
cmd: "tar xf /local/outputs.conf && grep sslPassword /local/outputs.conf | cut -c15-"
chdir: "/etc/apps/"
become: yes
become_user: ""
register: plaintext_password
- name: Encrypt Password
shell:
cmd: " show-encrypted --value ''"
chdir: "/etc/apps"
become: yes
become_user: ""
register: encrypt_password
- name: Update Password
lineinfile:
path: "/etc/apps//local/outputs.conf"
regexp: '^sslPassword'
line: "sslPassword = "
become: yes
become_user: ""
notify:
- Restart the splunkd service