Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Fix issues 230, 232, 235, and 249 #250

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 65 additions & 63 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,15 +42,17 @@ or apt to install these components if they're self-hosted.

### What splunk affects

* Installs the Splunk/Forwarder package and manages their config files. It does
not purge them by default.
* Installs the splunk or splunkforwarder package and manages their config
files. It does not purge them by default.
* The module will set up both Splunk Enterprise and Splunk Forwarder to run as
the 'root' user on POSIX platforms.
* By default, enables Splunk Enterprise and Splunk Forwarder boot-start, and
uses the vendor-generated service file to manage the splunk service.

### Setup Requirements

#### Module Installation

To begin using this module, use the Puppet Module Tool (PMT) from the command
line to install this module:

Expand All @@ -62,19 +64,26 @@ the --target-dir directive.
You can also use r10k or code-manager to deploy the module so ensure that you
have the correct entry in your Puppetfile.

Once the module is in place, there is just a little setup needed.
#### Package Dependencies

Once the module is in place, you will need to ensure the splunk package(s) are
available.

If your environment has the splunk package(s) available, and the supplied
`package_provider` supports it, it is not required for you to manage the splunk
packages.

First, you will need to place your downloaded splunk installers into the files
Otherwise, you will need to place your downloaded splunk installers into the files
directory, `<module_path>/splunk/files/`. If you're using r10k or code-manager
you'll need to override the `splunk::params::src_root` parameter to point at a
modulepath outside of the Splunk module because each deploy will overwrite the
files.
you'll need to override the `splunk::enterprise::src_root` or
`splunk::forwarder::src_root` parameter to point at a modulepath outside of the
Splunk module because each deploy will overwrite the files.

The files must be placed according to directory structure example given below.

The expected directory structure is:

$root_url/
$src_root/
└── products/
├── universalforwarder/
│ └── releases/
Expand All @@ -89,7 +98,7 @@ The expected directory structure is:

A semi-populated example files directory might then contain:

$root_url/
$src_root/
└── products/
├── universalforwarder/
│ └── releases/
Expand All @@ -109,43 +118,69 @@ A semi-populated example files directory might then contain:
├── splunk-7.0.0-c8a78efdd40f-linux-2.6-intel.deb
└── splunk-7.0.0-c8a78efdd40f-linux-2.6-x86_64.rpm

Second, you will need to supply the `splunk::params` class with three critical
pieces of information.

* The version of Splunk you are using
* The build of Splunk you are using
* The root URL to use to retrieve the packages

In the example given above, the version is 7.0.0, the build is c8a78efdd40f,
and the root URL is puppet:///modules/splunk. See the splunk::params class
documentation for more information.

### Beginning with splunk

Once the Splunk packages are hosted in the users repository or hosted by the
Puppet Server in the modulepath the module is ready to deploy.

## Usage

If a user is installing Splunk Enterprise with packages provided from their
modulepath, this is the most basic way of installing Splunk Server with default
settings:
### Splunk Enterprise

If splunk is already installed on the target node, the splunk `version` and
`build` will be determined by the `splunkenterprise` fact. You can simply
include the module on your node:

```puppet
include ::splunk::enterprise
include splunk::enterprise
```

This is the most basic way of installing the Splunk Universal Forwarder with
default settings:
Otherwise, if splunk is not installed, you will need to follow the instructions
in Setup Requirements. You may need to specify `src_root` if the defaults are
not suitable for your environment. You will also need to specify `release`,
in the format `version-build` as follows:

```yaml
---
splunk::enterprise::release: '7.0.0-c8a78efdd40f'
```

```puppet
class { '::splunk::params':
server => $my_splunk_server,
}
include splunk::enterprise
```

### Splunk Forwarder

If splunkforwarder is already installed on the target node, the splunk
`version` and `build` will be determined by the `splunkforwarder` fact. You
will only need to specify a `server`:

```yaml
---
splunk::forwarder::server: <your.servers.ip.addr>
```

```puppet
include ::splunk::forwarder
```

Otherwise, if splunkforwarder is not installed, you will need to follow the
instructions in Setup Requirements. You may need to specify `src_root` if the
defaults are not suitable for your environment. You will also need to specify
`release`, in the format `version-build` as follows:

```yaml
---
splunk::forwarder::release: '7.0.0-c8a78efdd40f'
splunk::forwarder::server: <your.servers.ip.addr>
```

```puppet
include ::splunk::forwarder
```

### Splunk Types

Once both Splunk Enterprise and Splunk Universal Forwarder have been deployed
on their respective nodes, the Forwarder is ready to start sending logs.

Expand Down Expand Up @@ -181,47 +216,14 @@ Alternatively the the `splunk::enterprise::password::seed` class can be used ind

### Upgrade splunk and splunkforwarder packages

This module has the ability to install *and* upgrade the splunk and splunkforwarder packages. All you have to do is declare `package_ensure => 'latest'` when calling the `::splunk` or `::splunk::forwarder` classes.

Upgrades from 7.0.X to >= 7.0.X are not tested.
Upgrades have not been tested with this module.

#### Upgrade Example

The following code will install the 6.6.8 version of the splunk forwarder. Then
comment out the 6.6.8 version and build values and uncomment the 7.1.2 version
and build values. Running puppet again will perform the following:

1. splunk forwarder package is upgraded
1. splunk service is stopped as part of the package upgrade process
1. new license agreement is automatically accepted
1. license agreement must be accepted or the splunk service will fail to start
1. splunk service is started

```puppet
# Tell the module to get packages directly from Splunk.
class { '::splunk::params':
version => '6.6.8',
build => '6c27a8439c1e',
#version => '7.1.2',
#build => 'a0c72a66db66',
src_root => 'https://download.splunk.com',
}

# Specifying package_ensure => 'latest' will ensure that the splunk and
# splunkforwarder packages will be upgraded when you specify newer values for
# version and build.
class { '::splunk::forwarder':
package_ensure => 'latest',
}
```
## Reference

See in file [REFERENCE.md](REFERENCE.md).

## Limitations

- Currently tested manually on Centos 7, but we will eventually add automated
testing and are targeting compatibility with other platforms.
- Tested with Puppet 5.x
- New installations of splunk up to version 7.2.X are supported, but upgrades
from 7.0.X to >= 7.0.X are not fully tested
Expand Down
57 changes: 57 additions & 0 deletions data/common.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
splunk::enterprise::manage_password: false
splunk::enterprise::seed_password: false
splunk::enterprise::reset_seeded_password: false
splunk::enterprise::secret: 'hhy9DOGqli4.aZWCuGvz8stcqT2/OSJUZuyWHKc4wnJtQ6IZu2bfjeElgYmGHN9RWIT3zs5hRJcX1wGerpMNObWhFue78jZMALs3c3Mzc6CzM98/yGYdfcvWMo1HRdKn82LVeBJI5dNznlZWfzg6xdywWbeUVQZcOZtODi10hdxSJ4I3wmCv0nmkSWMVOEKHxti6QLgjfuj/MOoh8.2pM0/CqF5u6ORAzqFZ8Qf3c27uVEahy7ShxSv2K4K41z'
splunk::enterprise::password_hash: '$6$pIE/xAyP9mvBaewv$4GYFxC0SqonT6/x8qGcZXVCRLUVKODj9drDjdu/JJQ/Iw0Gg.aTkFzCjNAbaK4zcCHbphFz1g1HK18Z2bI92M0'
splunk::enterprise::password_content: ":admin:${password_hash}::Administrator:admin:[email protected]::"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does embedding variables in this way work hiera the same as it did with manifest interpolation? Documentation seems to indicate, no.

https://puppet.com/docs/puppet/latest/hiera_merging.html#the-lookup-and-hiera-function

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corroborated by the fact that this was caught in other data files, e.g. Linux.yaml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this should certainly be a lookup

splunk::enterprise::manage_package_source: true
splunk::enterprise::src_root: 'https://download.splunk.com'
splunk::enterprise::service_ensure: 'running'
splunk::enterprise::splunkd_port: 8089
splunk::enterprise::logging_port: 9997
splunk::enterprise::web_httpport: 8000
splunk::enterprise::server: 'splunk'
splunk::enterprise::boot_start: true
splunk::enterprise::use_default_config: true
splunk::enterprise::input_default_host: "${facts.fqdn}"
splunk::enterprise::input_connection_host: 'dns'
splunk::enterprise::splunkd_listen: '127.0.0.1'
splunk::enterprise::purge_alert_actions: false
splunk::enterprise::purge_authentication: false
splunk::enterprise::purge_authorize: false
splunk::enterprise::purge_deploymentclient: false
splunk::enterprise::purge_distsearch: false
splunk::enterprise::purge_indexes: false
splunk::enterprise::purge_inputs: false
splunk::enterprise::purge_limits: false
splunk::enterprise::purge_outputs: false
splunk::enterprise::purge_props: false
splunk::enterprise::purge_server: false
splunk::enterprise::purge_serverclass: false
splunk::enterprise::purge_transforms: false
splunk::enterprise::purge_uiprefs: false
splunk::enterprise::purge_web: false

splunk::forwarder::server: 'splunk'
splunk::forwarder::service_ensure: 'running'
splunk::forwarder::manage_package_source: true
splunk::forwarder::src_root: 'https://download.splunk.com'
splunk::forwarder::boot_start: true
splunk::forwarder::use_default_config: true
splunk::forwarder::splunkd_listen: '127.0.0.1'
splunk::forwarder::splunkd_port: 8089
splunk::forwarder::logging_port: 9997
splunk::forwarder::purge_deploymentclient: false
splunk::forwarder::purge_outputs: false
splunk::forwarder::purge_inputs: false
splunk::forwarder::purge_props: false
splunk::forwarder::purge_transforms: false
splunk::forwarder::purge_web: false
splunk::forwarder::manage_password: false
splunk::forwarder::seed_password: false
splunk::forwarder::reset_seeded_password: false
splunk::forwarder::secret: 'hhy9DOGqli4.aZWCuGvz8stcqT2/OSJUZuyWHKc4wnJtQ6IZu2bfjeElgYmGHN9RWIT3zs5hRJcX1wGerpMNObWhFue78jZMALs3c3Mzc6CzM98/yGYdfcvWMo1HRdKn82LVeBJI5dNznlZWfzg6xdywWbeUVQZcOZtODi10hdxSJ4I3wmCv0nmkSWMVOEKHxti6QLgjfuj/MOoh8.2pM0/CqF5u6ORAzqFZ8Qf3c27uVEahy7ShxSv2K4K41z'
splunk::forwarder::password_hash: '$6$pIE/xAyP9mvBaewv$4GYFxC0SqonT6/x8qGcZXVCRLUVKODj9drDjdu/JJQ/Iw0Gg.aTkFzCjNAbaK4zcCHbphFz1g1HK18Z2bI92M0'
splunk::forwarder::password_content: ":admin:${password_hash}::Administrator:admin:[email protected]::"
splunk::forwarder::addons: {}
25 changes: 25 additions & 0 deletions data/kernel/Linux.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---

splunk::enterprise::package_name: 'splunk'
splunk::enterprise::splunk_user: 'root'
splunk::enterprise::staging_dir: '/opt/staging/splunk'
splunk::enterprise::homedir: '/opt/splunk'
splunk::enterprise::path_delimiter: '/'
splunk::enterprise::seed_config_file: "%{lookup('splunk::enterprise::homedir')}/etc/system/local/user-seed.conf"
splunk::enterprise::password_config_file: "%{lookup('splunk::enterprise::homedir')}/etc/passwd"
splunk::enterprise::secret_file: "%{lookup('splunk::enterprise::homedir')}/etc/splunk.secret"
splunk::enterprise::confdir: "%{lookup('splunk::enterprise::homedir')}/etc"
splunk::enterprise::src_subdir: 'linux'
splunk::enterprise::install_options: []

splunk::forwarder::package_name: 'splunkforwarder'
splunk::forwarder::staging_dir: '/opt/staging/splunk'
splunk::forwarder::path_delimiter: '/'
splunk::forwarder::homedir: '/opt/splunkforwarder'
splunk::forwarder::confdir: "%{lookup('splunk::forwarder::homedir')}/etc"
splunk::forwarder::src_subdir: 'linux'
splunk::forwarder::install_options: []
splunk::forwarder::splunk_user: 'root'
splunk::forwarder::password_config_file: "%{lookup('splunk::forwarder::homedir')}/etc/passwd"
splunk::forwarder::seed_config_file: "%{lookup('splunk::forwarder::homedir')}/etc/system/local/user-seed.conf"
splunk::forwarder::secret_file: "%{lookup('splunk::forwarder::homedir')}/etc/splunk.secret"
25 changes: 25 additions & 0 deletions data/kernel/SunOS.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---

splunk::enterprise::package_name: 'splunk'
splunk::enterprise::splunk_user: 'root'
splunk::enterprise::staging_dir: '/opt/staging/splunk'
splunk::enterprise::homedir: '/opt/splunk'
splunk::enterprise::path_delimiter: '/'
splunk::enterprise::seed_config_file: "%{lookup('splunk::enterprise::homedir')}/etc/system/local/user-seed.conf"
splunk::enterprise::password_config_file: "%{lookup('splunk::enterprise::homedir')}/etc/passwd"
splunk::enterprise::secret_file: "%{lookup('splunk::enterprise::homedir')}/etc/splunk.secret"
splunk::enterprise::confdir: "%{lookup('splunk::enterprise::homedir')}/etc"
splunk::enterprise::src_subdir: 'solaris'
splunk::enterprise::install_options: []

splunk::forwarder::package_name: 'splunkforwarder'
splunk::forwarder::staging_dir: '/opt/staging/splunk'
splunk::forwarder::path_delimiter: '/'
splunk::forwarder::homedir: '/opt/splunkforwarder'
splunk::forwarder::confdir: "%{lookup('splunk::forwarder::homedir')}/etc"
splunk::forwarder::src_subdir: 'solaris'
splunk::forwarder::install_options: []
splunk::forwarder::splunk_user: 'root'
splunk::forwarder::password_config_file: "%{lookup('splunk::forwarder::homedir')}/etc/passwd"
splunk::forwarder::seed_config_file: "%{lookup('splunk::forwarder::homedir')}/etc/system/local/user-seed.conf"
splunk::forwarder::secret_file: "%{lookup('splunk::forwarder::homedir')}/etc/splunk.secret"
37 changes: 37 additions & 0 deletions data/kernel/Windows.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
---

splunk::enterprise::package_name: 'Splunk Enterpirse'
splunk::enterprise::splunk_user: 'Administrator'
splunk::enterprise::staging_dir: "%{facts.archive_windir}\\splunk"
splunk::enterprise::homedir: 'C:\\Program Files\\Splunk'
splunk::enterprise::path_delimiter: '\\'
splunk::enterprise::seed_config_file: "%{lookup('splunk::enterprise::homedir')}\\etc\\system\\local\\user-seed.conf"
splunk::enterprise::password_config_file: "%{lookup('splunk::enterprise::homedir')}\\etc\\passwd"
splunk::enterprise::confdir: "%{lookup('splunk::enterprise::homedir')}\\etc"
splunk::enterprise::src_subdir: 'windows'
splunk::enterprise::install_options: [
{INSTALLDIR: "%{lookup('splunk::enterprise::homedir')}"},
{SPLUNKD_PORT: "%{lookup('splunk::enterprise::splunkd_port')}"},
'ARGEETOLICENSE=Yes',
'LAUNCHSPLUNK=0']

splunk::forwarder::splunk_user: 'Administrator'
splunk::forwarder::package_name: 'UniversalForwarder'
splunk::forwarder::staging_dir: "%{facts.archive_windir}\\splunk"
splunk::forwarder::homedir: 'C:\\Program Files\\SplunkUniversalForwarder'
splunk::forwarder::confdir: "%{lookup('splunk::forwarder::homedir')}\\etc"
splunk::forwarder::src_subdir: 'windows'
splunk::forwarder::path_delimiter: '\\'
splunk::forwarder::password_config_file: "%{lookup('splunk::forwarder::homedir')}\\etc\\passwd"
splunk::forwarder::seed_config_file: "%{lookup('splunk::forwarder::homedir')}\\etc\\system\\local\\user-seed.conf"
splunk::forwarder::install_options: [
{INSTALLDIR: "%{lookup('splunk::forwarder::homedir')}"},
'AGREETOLICENSE=Yes',
'LAUNCHSPLUNK=0',
'SERVICESTARTTYPE=auto',
'WINEVENTLOG_APP_ENABLE=1',
'WINEVENTLOG_SEC_ENABLE=1',
'WINEVENTLOG_SYS_ENABLE=1',
'WINEVENTLOG_FWD_ENABLE=1',
'WINEVENTLOG_SET_ENABLE=1',
'ENABLEADMON=1']
5 changes: 5 additions & 0 deletions data/os/Debian.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---

splunk::enterprise::package_provider: 'dpkg'

splunk::forwarder::package_provider: 'dpkg'
5 changes: 5 additions & 0 deletions data/os/RedHat.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---

splunk::enterprise::package_provider: 'rpm'

splunk::forwarder::package_provider: 'rpm'
5 changes: 5 additions & 0 deletions data/os/Solaris.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---

splunk::enterprise::package_provider: 'sun'

splunk::forwarder::package_provider: 'sun'
5 changes: 5 additions & 0 deletions data/os/windows.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---

splunk::enterprise::package_provider: 'windows'

splunk::forwarder::package_provider: 'windows'
16 changes: 16 additions & 0 deletions hiera.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
version: 5

defaults:
datadir: data
data_hash: yaml_data

hierarchy:
- name: "OS family"
path: "os/%{facts.os.family}.yaml"

- name: "Kernel"
path: "kernel/%{facts.kernel}.yaml"

- name: "common"
path: "common.yaml"
Loading