NOTE: I consider this post a DRAFT — I have not validated these settings in my home lab, this article was written from memory of doing this for a customer. I intend to validate these steps later and remove this warning once I do so.
If you have a large number of deployment clients that have a .conf file stuck in /etc/system/local that you need to move out in order to manage it properly in an app – try this. I recently came across this at a customer who needed to update their deploymentclient.conf to point to their new deployment server, but all of their deployment clients had deploymentclient.conf in $SPLUNK_HOME/etc/system/local which couldn’t be easily updated remotely.
To solve this dilemma, we did the following:
First, we created and pushed our new deploymentclient.conf app pointing to our NEW deployment server to all the deployment clients. Of course we know that at this time, this app/conf file will be trumped by the deploymentclient.conf in /etc/system/local until that local file is removed or renamed.
Second, we created a custom app with a one-shot scripted input to delete the deploymentclient.conf file out of $SPLUNK_HOME/etc/system/local. This can be done with either a shell script on Linux or a Powershell script in Windows. Here are some examples:
Linux:
Customapp/bin/removeDeploymentClient.sh
#/bin/shrm -f $SPLUNK_HOME/etc/system/local/deploymentclient.conf
Windows:
customapp\bin\removeDeploymentClient.bat
del $SPLUNK_HOME\etc\system\local\deploymentclient.conf
Then add your inputs.conf — the key here is the setting of interval = -1 which means it will only run once on startup. Reference here: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
Customapp/local/inputs.conf
Linux:
[script://./bin/removeDeploymentClient.sh]
interval = -1
source = removeDeploymentClient
sourcetype = scriptedInput
index = _internal
disabled = 0
Windows:
[script://.\bin\removeDeploymentClient.bat]
interval = -1
source = removeDeploymentClient
sourcetype = scriptedInput
index = _internal
disabled = 0
That’s all! Set your custom app to restart Splunkd on your deployment server and push it to your clients.
NOTE: the Splunk service on your deployment clients may need to be restarted TWICE for the change to take effect. The first time Splunk is restarted, it will remove the $SPLUNK_HOME/etc/system/local/deploymentclient.conf AFTER Splunk starts, so it will already be loaded in memory. After you restart Splunk again, that file will no longer be present, so it will then load the deploymentclient.conf you pushed in your custom app. One way to force a restart of the Splunk service from the DS is to create a dummy app with “Restart splunkd” checked and push it.