diff --git a/docs/bestpractices.md b/docs/bestpractices.md index 9552d1153..7692ba2e5 100644 --- a/docs/bestpractices.md +++ b/docs/bestpractices.md @@ -11,6 +11,38 @@ and replace with the pertinent IP Address. Installation of RedHat may be blocking ports required by microk8s. Installing microk8s on RedHat requires checking to see if the firewall is not blocking any of [required microk8s ports](https://microk8s.io/docs/ports). +### Accessing SC4SNMP logs + +SC4SNMP logs can be browsed in Splunk in `em_logs` index, provided that [sck-otel](gettingstarted/sck-installation.md) +is installed. Logs can be also accessed directly in kubernetes using terminal. + +#### Accessing logs via Splunk +If [sck-otel](gettingstarted/sck-installation.md) is installed, browse `em_logs` index. Logs can be further filtered +for example by the sourcetype field. Example search command to get logs from poller: +``` +index=em_logs sourcetype="kube:container:splunk-connect-for-snmp-worker-poller" +``` + +#### Accessing logs in kubernetes +To access logs directly in kubernetes, first run `microk8s kubectl -n sc4snmp get pods`. This will output all pods: +``` +NAME READY STATUS RESTARTS AGE +snmp-splunk-connect-for-snmp-worker-trap-99f49c557-j9jwx 1/1 Running 0 29m +snmp-splunk-connect-for-snmp-trap-56f75f9754-kmlgb 1/1 Running 0 29m +snmp-splunk-connect-for-snmp-scheduler-7bb8c79855-rgjkj 1/1 Running 0 29m +snmp-mibserver-784bd599fd-6xzfj 1/1 Running 0 29m +snmp-splunk-connect-for-snmp-worker-poller-78b46d668f-59mv4 1/1 Running 0 29m +snmp-splunk-connect-for-snmp-worker-sender-6f8496bfbf-cvt9l 1/1 Running 0 29m +snmp-mongodb-7579dc7867-mlnst 2/2 Running 0 29m +snmp-redis-master-0 1/1 Running 0 29m +``` + +Now select the desired pod and run `microk8s kubectl -n sc4snmp logs pod/` command. Example command to retrieve +logs from `splunk-connect-for-snmp-worker-poller`: +``` +microk8s kubectl -n sc4snmp logs pod/snmp-splunk-connect-for-snmp-worker-poller-78b46d668f-59mv4 +``` + ## Issues ### "Empty SNMP response message" problem @@ -103,4 +135,98 @@ If the changes are required to be applied immedietly, the previous inventory job ``` microk8s kubectl delete job/snmp-splunk-connect-for-snmp-inventory -n sc4snmp ``` -The upgrade command can be executed again. \ No newline at end of file +The upgrade command can be executed again. + + +### Identifying Traps issues + +#### Wrong IP or port +The first possible answer to why traps are not sent to Splunk is that SNMP agents send trap messages to the wrong IP +address or port. To check what is the correct address of traps server, run the following command: + +``` +microk8s kubectl -n sc4snmp get services +``` + +This command should output similar data: +``` +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +snmp-redis-headless ClusterIP None 6379/TCP 113s +snmp-mibserver ClusterIP 10.152.183.163 80/TCP 113s +snmp-mongodb ClusterIP 10.152.183.118 27017/TCP 113s +snmp-redis-master ClusterIP 10.152.183.61 6379/TCP 113s +snmp-mongodb-metrics ClusterIP 10.152.183.50 9216/TCP 113s +snmp-splunk-connect-for-snmp-trap LoadBalancer 10.152.183.190 114.241.233.134 162:32180/UDP 113s +``` + +Check the `EXTERNAL-IP` of `snmp-splunk-connect-for-snmp-trap` and the second port number for this service. In this case +the full `snmp-splunk-connect-for-snmp-trap` address will be `114.241.233.134:32180`. + + +In case agents send traps to the correct address, but there is still no data in the `netops` index, there might be some +issues with credentials. These errors can be seen in logs of the `snmp-splunk-connect-for-snmp-trap` pod. + +#### Unknown SNMP community name encountered +In case of using community string for authentication purposes, the following error should be expected if the arriving trap +has a community string not configured in SC4SNMP: +``` +2024-02-06 15:42:14,885 ERROR Security Model failure for device ('18.226.181.199', 42514): Unknown SNMP community name encountered +``` + +If this error occurs, check if the appropriate community is defined under `traps.communities` in `values.yaml`. See the +following example of a `public` community configuration: +```yaml +traps: + communities: + public: + communityIndex: + contextEngineId: + contextName: + tag: + securityName: +``` + +#### Unknown SNMP security name encountered + +While sending SNMP v3 traps in case of wrong username or engine id configuration, the following error should be expected: +``` +2024-02-06 15:42:14,091 ERROR Security Model failure for device ('18.226.181.199', 46066): Unknown SNMP security name encountered +``` + +If this error occurs, verify that the kubernetes secret with the correct username has been created ([SNMPv3 configuration](configuration/snmpv3-configuration.md)). +After creating the secret, add it under `traps.usernameSecrets` in `values.yaml`. Check that the correct snmp engine id +is configured under `traps.securityEngineId`. See the following example of a `values.yaml` with configured secret and engine id: +```yaml +traps: + usernameSecrets: + - my-secret-name + securityEngineId: + - "090807060504030201" +``` + +#### Authenticator mismatched + +While sending SNMP v3 traps in case of wrong authentication protocol or password configuration, the following error should be expected: +``` +2024-02-06 15:42:14,642 ERROR Security Model failure for device ('18.226.181.199', 54806): Authenticator mismatched +``` +If this error occurs, verify that the kubernetes secret with the correct authentication protocol and password has been created ([SNMPv3 configuration](configuration/snmpv3-configuration.md)). +After creating the secret, add it under `traps.usernameSecrets` in `values.yaml`. See the following example of a `values.yaml` with configured secret: +```yaml +traps: + usernameSecrets: + - my-secret-name +``` + +#### Ciphering services not available or ciphertext is broken +While sending SNMP v3 traps in case of wrong privacy protocol or password configuration, the following error should be expected: +``` +2024-02-06 15:42:14,780 ERROR Security Model failure for device ('18.226.181.199', 48249): Ciphering services not available or ciphertext is broken +``` +If this error occurs, verify that the kubernetes secret with the correct privacy protocol and password has been created ([SNMPv3 configuration](configuration/snmpv3-configuration.md)). +After creating the secret, add it under `traps.usernameSecrets` in `values.yaml`. See the following example of a `values.yaml` with configured secret: +```yaml +traps: + usernameSecrets: + - my-secret-name +``` diff --git a/splunk_connect_for_snmp/traps.py b/splunk_connect_for_snmp/traps.py index b7221ca6d..e1b043388 100644 --- a/splunk_connect_for_snmp/traps.py +++ b/splunk_connect_for_snmp/traps.py @@ -28,6 +28,7 @@ import asyncio import os +from typing import Any, Dict import yaml from celery import Celery, chain @@ -96,6 +97,14 @@ def cbFun(snmpEngine, stateReference, contextEngineId, contextName, varBinds, cb _ = my_chain.apply_async() +# Callback function for logging traps authentication errors +def authentication_observer_cb_fun(snmpEngine, execpoint, variables, contexts): + logging.error( + f"Security Model failure for device {variables.get('transportAddress', None)}: " + f"{variables.get('statusInformation', {}).get('errorIndication', None)}" + ) + + app.autodiscover_tasks( packages=[ "splunk_connect_for_snmp", @@ -111,9 +120,20 @@ def main(): # Get the event loop for this thread loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) + # Create SNMP engine with autogenernated engineID and pre-bound # to socket transport dispatcher snmpEngine = engine.SnmpEngine() + + # Register a callback function to log errors with traps authentication + observer_context: Dict[Any, Any] = {} + snmpEngine.observer.registerObserver( + authentication_observer_cb_fun, + "rfc2576.prepareDataElements:sm-failure", + "rfc3412.prepareDataElements:sm-failure", + cbCtx=observer_context, + ) + # UDP over IPv4, first listening interface/port config.addTransport( snmpEngine,