fix: add logs in traps server for situations when incoming traps have…

… wrong credentials (#953)

fix: delete unwanted files

fix: add logging of source device ip

fix: log only necessary info

fix: run pre-commit

fix: update the troubleshooting documentation

fix: add comment in code

fix: fixes from pr review

fix: fix typos in documentation

docs/bestpractices.md

@@ -11,6 +11,38 @@ and replace <IP_ADDRESS> with the pertinent IP Address.
 Installation of RedHat may be blocking ports required by microk8s. Installing microk8s on RedHat 
 requires checking to see if the firewall is not blocking any of [required microk8s ports](https://microk8s.io/docs/ports). 
 
+### Accessing SC4SNMP logs 
+
+SC4SNMP logs can be browsed in Splunk in `em_logs` index, provided that [sck-otel](gettingstarted/sck-installation.md)
+is installed. Logs can be also accessed directly in kubernetes using terminal.
+
+#### Accessing logs via Splunk
+If [sck-otel](gettingstarted/sck-installation.md) is installed, browse `em_logs` index. Logs can be further filtered 
+for example by the sourcetype field. Example search command to get logs from poller:
+```
+index=em_logs sourcetype="kube:container:splunk-connect-for-snmp-worker-poller"
+```
+
+#### Accessing logs in kubernetes
+To access logs directly in kubernetes, first run `microk8s kubectl -n sc4snmp get pods`. This will output all pods:
+```
+NAME                                                          READY   STATUS    RESTARTS   AGE
+snmp-splunk-connect-for-snmp-worker-trap-99f49c557-j9jwx      1/1     Running   0          29m
+snmp-splunk-connect-for-snmp-trap-56f75f9754-kmlgb            1/1     Running   0          29m
+snmp-splunk-connect-for-snmp-scheduler-7bb8c79855-rgjkj       1/1     Running   0          29m
+snmp-mibserver-784bd599fd-6xzfj                               1/1     Running   0          29m
+snmp-splunk-connect-for-snmp-worker-poller-78b46d668f-59mv4   1/1     Running   0          29m
+snmp-splunk-connect-for-snmp-worker-sender-6f8496bfbf-cvt9l   1/1     Running   0          29m
+snmp-mongodb-7579dc7867-mlnst                                 2/2     Running   0          29m
+snmp-redis-master-0                                           1/1     Running   0          29m
+```
+
+Now select the desired pod and run `microk8s kubectl -n sc4snmp logs pod/<pod-name>` command. Example command to retrieve
+logs from `splunk-connect-for-snmp-worker-poller`:
+```
+microk8s kubectl -n sc4snmp logs pod/snmp-splunk-connect-for-snmp-worker-poller-78b46d668f-59mv4
+```
+
 ## Issues
 
 ### "Empty SNMP response message" problem
@@ -103,4 +135,98 @@ If the changes are required to be applied immedietly, the previous inventory job
 ```
 microk8s kubectl delete job/snmp-splunk-connect-for-snmp-inventory -n sc4snmp
 ```
-The upgrade command can be executed again. 
+The upgrade command can be executed again. 
+
+
+### Identifying Traps issues
+
+#### Wrong IP or port
+The first possible answer to why traps are not sent to Splunk is that SNMP agents send trap messages to the wrong IP 
+address or port. To check what is the correct address of traps server, run the following command:
+
+```
+microk8s kubectl -n sc4snmp get services
+```
+
+This command should output similar data:
+```
+NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)         AGE
+snmp-redis-headless                 ClusterIP      None             <none>           6379/TCP        113s
+snmp-mibserver                      ClusterIP      10.152.183.163   <none>           80/TCP          113s
+snmp-mongodb                        ClusterIP      10.152.183.118   <none>           27017/TCP       113s
+snmp-redis-master                   ClusterIP      10.152.183.61    <none>           6379/TCP        113s
+snmp-mongodb-metrics                ClusterIP      10.152.183.50    <none>           9216/TCP        113s
+snmp-splunk-connect-for-snmp-trap   LoadBalancer   10.152.183.190   114.241.233.134   162:32180/UDP   113s
+```
+
+Check the `EXTERNAL-IP` of `snmp-splunk-connect-for-snmp-trap` and the second port number for this service. In this case 
+the full `snmp-splunk-connect-for-snmp-trap` address will be `114.241.233.134:32180`.
+
+
+In case agents send traps to the correct address, but there is still no data in the `netops` index, there might be some
+issues with credentials. These errors can be seen in logs of the `snmp-splunk-connect-for-snmp-trap` pod. 
+
+#### Unknown SNMP community name encountered
+In case of using community string for authentication purposes, the following error should be expected if the arriving trap 
+has a community string not configured in SC4SNMP:
+```
+2024-02-06 15:42:14,885 ERROR Security Model failure for device ('18.226.181.199', 42514): Unknown SNMP community name encountered
+```
+
+If this error occurs, check if the appropriate community is defined under `traps.communities` in `values.yaml`. See the 
+following example of a `public` community configuration:
+```yaml
+traps:
+  communities:
+    public:
+      communityIndex:
+      contextEngineId:
+      contextName:
+      tag:
+      securityName:
+```
+
+#### Unknown SNMP security name encountered
+
+While sending SNMP v3 traps in case of wrong username or engine id configuration, the following error should be expected: 
+```
+2024-02-06 15:42:14,091 ERROR Security Model failure for device ('18.226.181.199', 46066): Unknown SNMP security name encountered
+```
+
+If this error occurs, verify that the kubernetes secret with the correct username has been created ([SNMPv3 configuration](configuration/snmpv3-configuration.md)).
+After creating the secret, add it under `traps.usernameSecrets` in `values.yaml`. Check that the correct snmp engine id
+is configured under `traps.securityEngineId`. See the following example of a `values.yaml` with configured secret and engine id:
+```yaml
+traps:
+  usernameSecrets:
+    - my-secret-name
+  securityEngineId:
+    - "090807060504030201"
+```
+
+#### Authenticator mismatched
+
+While sending SNMP v3 traps in case of wrong authentication protocol or password configuration, the following error should be expected: 
+```
+2024-02-06 15:42:14,642 ERROR Security Model failure for device ('18.226.181.199', 54806): Authenticator mismatched
+```
+If this error occurs, verify that the kubernetes secret with the correct authentication protocol and password has been created ([SNMPv3 configuration](configuration/snmpv3-configuration.md)).
+After creating the secret, add it under `traps.usernameSecrets` in `values.yaml`. See the following example of a `values.yaml` with configured secret:
+```yaml
+traps:
+  usernameSecrets:
+    - my-secret-name
+```
+
+#### Ciphering services not available or ciphertext is broken
+While sending SNMP v3 traps in case of wrong privacy protocol or password configuration, the following error should be expected: 
+```
+2024-02-06 15:42:14,780 ERROR Security Model failure for device ('18.226.181.199', 48249): Ciphering services not available or ciphertext is broken
+```
+If this error occurs, verify that the kubernetes secret with the correct privacy protocol and password has been created ([SNMPv3 configuration](configuration/snmpv3-configuration.md)).
+After creating the secret, add it under `traps.usernameSecrets` in `values.yaml`. See the following example of a `values.yaml` with configured secret:
+```yaml
+traps:
+  usernameSecrets:
+    - my-secret-name
+```

splunk_connect_for_snmp/traps.py

@@ -28,6 +28,7 @@
 
 import asyncio
 import os
+from typing import Any, Dict
 
 import yaml
 from celery import Celery, chain
@@ -96,6 +97,14 @@ def cbFun(snmpEngine, stateReference, contextEngineId, contextName, varBinds, cb
     _ = my_chain.apply_async()
 
 
+# Callback function for logging traps authentication errors
+def authentication_observer_cb_fun(snmpEngine, execpoint, variables, contexts):
+    logging.error(
+        f"Security Model failure for device {variables.get('transportAddress', None)}: "
+        f"{variables.get('statusInformation', {}).get('errorIndication', None)}"
+    )
+
+
 app.autodiscover_tasks(
     packages=[
         "splunk_connect_for_snmp",
@@ -111,9 +120,20 @@ def main():
     # Get the event loop for this thread
     loop = asyncio.new_event_loop()
     asyncio.set_event_loop(loop)
+
     # Create SNMP engine with autogenernated engineID and pre-bound
     # to socket transport dispatcher
     snmpEngine = engine.SnmpEngine()
+
+    # Register a callback function to log errors with traps authentication
+    observer_context: Dict[Any, Any] = {}
+    snmpEngine.observer.registerObserver(
+        authentication_observer_cb_fun,
+        "rfc2576.prepareDataElements:sm-failure",
+        "rfc3412.prepareDataElements:sm-failure",
+        cbCtx=observer_context,
+    )
+
     # UDP over IPv4, first listening interface/port
     config.addTransport(
         snmpEngine,