Month: June 2021

NOTE: I consider this post a DRAFT — I have not validated these settings in my home lab, this article was written from memory of doing this for a customer. I intend to validate these steps later and remove this warning once I do so.

If you have a large number of deployment clients that have a .conf file stuck in /etc/system/local that you need to move out in order to manage it properly in an app – try this. I recently came across this at a customer who needed to update their deploymentclient.conf to point to their new deployment server, but all of their deployment clients had deploymentclient.conf in $SPLUNK_HOME/etc/system/local which couldn’t be easily updated remotely.

To solve this dilemma, we did the following:

First, we created and pushed our new deploymentclient.conf app pointing to our NEW deployment server to all the deployment clients. Of course we know that at this time, this app/conf file will be trumped by the deploymentclient.conf in /etc/system/local until that local file is removed or renamed.

Second, we created a custom app with a one-shot scripted input to delete the deploymentclient.conf file out of $SPLUNK_HOME/etc/system/local. This can be done with either a shell script on Linux or a Powershell script in Windows. Here are some examples:

Linux:

Customapp/bin/removeDeploymentClient.sh

#/bin/sh
rm -f $SPLUNK_HOME/etc/system/local/deploymentclient.conf

Windows:
customapp\bin\removeDeploymentClient.bat

del $SPLUNK_HOME\etc\system\local\deploymentclient.conf

Then add your inputs.conf — the key here is the setting of interval = -1 which means it will only run once on startup. Reference here: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

Customapp/local/inputs.conf

Linux:

[script://./bin/removeDeploymentClient.sh]
interval = -1
source = removeDeploymentClient
sourcetype = scriptedInput
index = _internal
disabled = 0

Windows:

[script://.\bin\removeDeploymentClient.bat]
interval = -1
source = removeDeploymentClient
sourcetype = scriptedInput
index = _internal
disabled = 0

That’s all! Set your custom app to restart Splunkd on your deployment server and push it to your clients.

NOTE: the Splunk service on your deployment clients may need to be restarted TWICE for the change to take effect. The first time Splunk is restarted, it will remove the $SPLUNK_HOME/etc/system/local/deploymentclient.conf AFTER Splunk starts, so it will already be loaded in memory. After you restart Splunk again, that file will no longer be present, so it will then load the deploymentclient.conf you pushed in your custom app. One way to force a restart of the Splunk service from the DS is to create a dummy app with “Restart splunkd” checked and push it.

Per the Operating system compatibility section of the follow doc https://docs.splunk.com/Documentation/Splunk/latest/Updating/Planadeployment Windows Deployment Servers cannot manage Linux deployment clients because Windows does not understand how to maintain Linux file permissions for scripts and executables. So when you push a TA that has an executable or script to a Linux deployment client, you will receive “Permission denied”.

The best solution is to switch your deployment server to Linux, but if you don’t have the ability to do that, read on.

Follow these steps to workaround the issue.

1. Download the Linux TA that you want to push and extract it on any Linux machine.
2. Make any required changes to the app: for example, add your <app>/local/inputs.conf file with whatever changes you want to implement.
3. Repackage the tgz file WITHOUT the parent directory. For example, if you are repackaging Splunk_TA_nix, you would navigate into Splunk_TA_nix and create your tgz file with your local, default, bin, etc. folders all at the top level of the tgz file.
   tar cvfz <newname.tgz> *
4. Move the repackaged app to your Windows deployment server and place it under $SPLUNK_HOME/etc/deployment-apps
5. Backup your serverclass.conf file on your Windows deployment server. If you’ve been managing your Forwarder Management through the web UI, it will typically be located in $SPLUNK_HOME/etc/system/local/serverclass.conf
6. Edit serverclass.conf to include the following setting under your (third level) app stanza:

appFile = <YourModifiedApp>.tgz

Reference: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Serverclassconf

Also verify that your (second level) serverClass stanza looks correct for whitelist/blacklist, etc.

7. Restart Splunk on your DS (might not be necessary)

That’s it! When the deployment client phones home, the Windows DS should provide the tgz file with all the Linux file permissions intact.

Verification steps:

• Search splunkd.log on your DS for the name of your custom tgz app that you created. It should indicate that the app loaded successfully.
• On your search head, run a search for something like this:
  index=_internal host=<linuxforwarder> Splunk_TA_nix

Where Splunk_TA_nix would be a keyword for whichever app you were having problems with. In my case, we were trying to get the rlog.sh script that comes with the Splunk_TA_nix app to work but it kept erroring out with Permission Denied. So I used “rlog” as my search keyword. I already verified that the Splunk Forwarder service on the Linux host was running as root, so after we did this workaround, it executed as expected.