Moloch overhaul + general bugfixes/updates

README.md

@@ -1,4 +1,4 @@
-# edcop-tools
+# edcop-tools - DEPRECATED: tools now have their own repos under sealingtech/EDCOP-$TOOL
 EDCOP-Tools are meant to be deployed on the EDCOP platform.  EDCOP-Tools is a compilation of DCO tools which are built on Docker and Kubernetes.  The goal of this project is to build an easily deployable and manageable DCO infrastructure which is fully integrated using DevOps practices.  This is still an early prototype at this phase.
 
  View the website

bro/README.md

@@ -6,7 +6,6 @@ Table of Contents
 * [Configuration Guide](#configuration-guide)
 	* [Image Repository](#image-repository)
 	* [Networks](#networks)
-	* [Persistent Storage](#persistent-storage)
 	* [Node Selector](#node-selector)
 	* [Bro Configuration](#bro-configuration)
 		* [Resource Limits](#resource-limits)
@@ -21,7 +20,7 @@ Please share any bugs or features requests via GitHub issues.
 
 ## Image Repository
 
-By default, images are pulled from *edcop-master:5000* which is presumed to be hosted on the master node. If you're changing these values, make sure you use the full repository name.
+By default, Bro is pulled from EDCOP's official repo and the rest of the applications use their respective official images hosted on Docker's hub. If you're changing these values, make sure you use the full repository name.
 
 ```
 images:
@@ -51,21 +50,6 @@ passive		1d
 inline-1	1d
 inline-2	1d
 ```
-
-## Persistent Storage
-
-These values tell Kubernetes where Bro's logs should be stored on the 
-host for persistent storage. The *spool* option is for Bro's current 
-logs and the *logs* option is for Bro's old logs. By default, these values are set to */var/EDCOP/data/logs/bro* but should be changed according to your logical volume setup. 
-
-```
-volumes:
-  logs:
-    spool: 
-      hostPath: /var/EDCOP/data/logs/bro/spool
-    logs: 
-      hostPath: /var/EDCOP/data/logs/bro/logs   
-```
 
 ## Node Selector
 
@@ -80,9 +64,9 @@ To find out what labels your hosts have, please use the following:
 ```
 # kubectl get nodes --show-labels
 NAME		STATUS		ROLES		AGE		VERSION		LABELS
-master 		Ready		master		1d		v1.9.1		...,nodetype=master
-minion-1	Ready		<none>		1d		v1.9.1		...,nodetype=minion
-minion-2	Ready		<none>		1d		v1.9.1		...,nodetype=minion
+master 		Ready		master		1d		v1.10.0		...,nodetype=master
+minion-1	Ready		<none>		1d		v1.10.0		...,nodetype=minion
+minion-2	Ready		<none>		1d		v1.10.0		...,nodetype=minion
 ```
 
 ## Bro Configuration
@@ -95,6 +79,9 @@ You can set limits on Bro to ensure it doesn't use more CPU/memory space than ne
 
 ```
 broConfig:
+  requests:
+    cpu: 100m
+    memory: 64Mi
   limits:
     cpu: 2
     memory: 4G
@@ -124,6 +111,9 @@ logstashConfig:
   maxJvmHeap: 4g
   pipelineOutputWorkers: 2 
   pipelineBatchSize: 150  
+  requests:
+    cpu: 100m
+    memory: 64Mi
   limits:
     cpu: 2
     memory: 8G
@@ -135,6 +125,9 @@ Redis is also included in the Daemonset for the same reasons Logstash is. Curren
 
 ```
 redisConfig:
+  requests:
+    cpu: 100m
+    memory: 64Mi
   limits:
     cpu: 2
     memory: 8G

bro/helm/templates/bro-daemonset.yaml

@@ -84,8 +84,6 @@ spec:
       - name: filebeat
         image: {{ .Values.images.filebeat }}
         imagePullPolicy: Always
-        securityContext:
-          privileged: true
         volumeMounts:
         - mountPath: /bro/
           name: bro-spool

elasticsearch/README.md

@@ -9,6 +9,7 @@ Table of Contents
 	* [Persistent Volume Storage](#persistent-volume-storage)
 	* [Node Selector](#node-selector)
 	* [Elasticsearch Configuration](#elasticsearch-configuration)
+		* [General](#general)
 		* [Environment](#environment)
 		* [Resource Limits](#resource-limits)
 	* [Curator Configuration](#curator-configuration)
@@ -23,12 +24,12 @@ Please share any bugs or features requests via GitHub issues.
 
 ## Image Repository
 
-By default, images are pulled from *edcop-master:5000* which is presumed to be hosted on the master node. If you're changing these values, make sure you include the full repository name.
+By default, Elasticsearch is pulled from Elastic's official repository and the Curator is pulled from a customized image hosted on Docker's hub. If you're changing these values, make sure you include the full repository name.
 
 ```
 images:
-  elasticsearch: edcop-master:5000/elasticsearch
-  curator: edcop-master:5000/curator
+  elasticsearch: docker.elastic.co/elasticsearch/elasticsearch:6.2.4
+  curator: bobrik/curator
 ```
 
 ## Networks
@@ -53,11 +54,11 @@ inline-2	1d
 
 ## Persistent Volume Storage
 
-These values tell Kubernetes where Elasticsearch's index data should be stored on the host for persistent storage. By default, this value is set to */var/EDCOP/data/esdata* but should be changed according to your logical volume setup. 
+These values tell Kubernetes where Elasticsearch's index data should be stored on the host for persistent storage. By default, this value is set to */EDCOP/bulk/esdata* but should be changed according to your logical volume setup. 
 
 ```
 volumes:
-  data: /var/EDCOP/data/esdata
+  data: /EDCOP/bulk/esdata
 ```
 
 ## Node Selector
@@ -74,14 +75,30 @@ To find out what labels your hosts have, please use the following:
 ```
 # kubectl get nodes --show-labels
 NAME		STATUS		ROLES		AGE		VERSION		LABELS
-master 		Ready		master		1d		v1.9.1		...,nodetype=master
-minion-1	Ready		<none>		1d		v1.9.1		...,nodetype=minion
-minion-2	Ready		<none>		1d		v1.9.1		...,nodetype=minion
+master 		Ready		master		1d		v1.10.0		...,nodetype=master
+minion-1	Ready		<none>		1d		v1.10.0		...,nodetype=minion
+minion-2	Ready		<none>		1d		v1.10.0		...,nodetype=minion
 ```
 
 ## Elasticsearch Configuration
 
-Elasticsearch is deployed as a daemonset spread across all of the worker nodes in a single cluster. These instances point to the master deployment that should be on your Kubernetes master node. 
+Elasticsearch is deployed as a statefulset spread across all of the worker nodes in a single cluster. These instances point to the master deployment that should be on your Kubernetes master node. 
+
+### General
+
+In order to prevent permission issues, elasticsearch is required to run as a different user and that user should own the volume directory you specified above. This user must e created beforehand and should only have access to this directory/subdirectories for security purposes. Enter the UID of this user in the space below:
+
+```
+elasticsearchConfig:
+  runAsUser: 2000
+```
+
+Since Elasticsearch's workers are run as statefulsets, you need to specify how many instances you want to maintain. By default, this value is 3, but should be scaled to include the number of worker nodes you have. Do not include the master as one instance because it is deployed in a seperate deployment that only runs on the master. 
+
+```
+elasticsearchConfig:
+  workerNodes: 3
+```
 
 ### Environment
 
@@ -96,11 +113,14 @@ elasticsearchConfig:
 
 ### Resource Limits
 
-The second part of Elasticsearch's configuration allows you to limit the CPU and memory usage. Elasticsearch recommends memory to be capped at a 32GB maximum per their instructions available [here](https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html).
+The second part of Elasticsearch's configuration allows you to limit the CPU and memory usage. The request values must be smaller than the limit values and are set low by default to accomodate VMs. Elasticsearch recommends memory to be capped at a 32GB maximum per their instructions available [here](https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html).
 
 ```
 elasticsearchConfig:
   ...
+  requests:
+    cpu: 100m
+    memory: 64Mi
   limits:
     cpu: 12
     memory: 32Gi

elasticsearch/helm/Chart.yaml

@@ -4,5 +4,5 @@ home: https://github.com/sealingtech/EDCOP
 version: 0.1.0
 description: EDCOP Elasticsearch Chart
 details:
-  This Chart provides an Elasticsearch daemonset deployed onto worker nodes within the EDCOP project. 
+  This Chart provides an Elasticsearch statefulset deployed onto worker nodes within the EDCOP project. 
 icon: http://www.elastic.co/assets/bltd74b9a6e0e479359/icon-white-circle-elasticsearch.png

elasticsearch/helm/templates/curator-config.yaml

@@ -1,3 +1,4 @@
+{{ if .Values.curatorConfig.enabled }}
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -56,8 +57,7 @@ data:
     # Remember, leave a key empty if there is no value.  None will be a string,
     # not a Python "NoneType"
     client: 
-      hosts:
-        - data-service
+      hosts: data-service
       port: 9200
       url_prefix:
       use_ssl: False
@@ -74,3 +74,4 @@ data:
       logfile:
       logformat: default
       blacklist: ['elasticsearch', 'urllib3']
+{{ end }}

elasticsearch/helm/templates/curator-cronjob.yaml

@@ -1,3 +1,4 @@
+{{ if .Values.curatorConfig.enabled }}
 apiVersion: batch/v1beta1
 kind: CronJob
 metadata:
@@ -7,7 +8,9 @@ metadata:
     heritage: {{ .Release.Service }}
     release: {{ .Release.Name }}
 spec:
-  schedule: {{ .Values.curatorConfig.cronjob_schedule }}
+  schedule: "{{ .Values.curatorConfig.cronjob_schedule }}"
+  successfulJobsHistoryLimit: {{ .Values.curatorConfig.successfulJobsLimit }}
+  failedJobsHistoryLimit: {{ .Values.curatorConfig.failedPodsLimit }}
   jobTemplate:
     spec:
       template:
@@ -18,16 +21,19 @@ spec:
           - name: curator
             image: {{ .Values.images.curator }}
             imagePullPolicy: Always
+            command: ["curator"]
             args:
-            - curator
             - --config
             - /etc/config/config.yml
             - /etc/config/actions.yml
             volumeMounts:
               - name: curator-config
                 mountPath: /etc/config
           volumes:
-            - name: curator-config
-              configMap:
-                name: {{ template "elasticsearch.fullname" . }}-curator-config
+          - name: curator-config
+            configMap:
+              name: {{ template "elasticsearch.fullname" . }}-curator-config
           restartPolicy: OnFailure 
+          nodeSelector:
+            nodetype: {{ .Values.nodeSelector.master }}
+{{ end }}

elasticsearch/helm/templates/elasticsearch-config.yaml

@@ -99,5 +99,8 @@ data:
     xpack.security.enabled: false
     xpack.watcher.enabled: false
     xpack.monitoring.enabled: true
+    # --------------------------------- Snapshots ----------------------------------
+    # Location of NFS storage for snapshots
+    path.repo: ["/var/EDCOP/snapshots"]
 
 

elasticsearch/helm/templates/es-client.yaml

@@ -1,5 +1,5 @@
-apiVersion: extensions/v1beta1
-kind: DaemonSet
+apiVersion: apps/v1
+kind: StatefulSet
 metadata:
   name: {{ template "elasticsearch.fullname" . }}
   labels:
@@ -9,6 +9,11 @@ metadata:
     component: {{ template "elasticsearch.name" . }}
     role: client
 spec:
+  serviceName: "data-service"
+  replicas: {{ .Values.elasticsearchConfig.workerNodes }}
+  selector:
+    matchLabels:
+      component: {{ template "elasticsearch.name" . }}
   template:
     metadata:
       name: {{ template "elasticsearch.name" . }}
@@ -23,6 +28,8 @@ spec:
       containers:
       - name: es-client
         image: {{ .Values.images.elasticsearch }}
+        securityContext:
+          runAsUser: {{ .Values.elasticsearchConfig.runAsUser }}
         env:
         - name: NAMESPACE
           valueFrom:
@@ -60,6 +67,8 @@ spec:
         - mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
           subPath: elasticsearch.yml
           name: elasticsearch-config
+        - mountPath: /var/EDCOP/snapshots
+          name: elasticsearch-snapshots
 
       volumes:
       - name: esdata
@@ -71,6 +80,9 @@ spec:
       - name: elasticsearch-config
         configMap:
           name: {{ template "elasticsearch.fullname" . }}-config
+      - name: elasticsearch-snapshots
+        persistentVolumeClaim:
+          claimName: {{ template "elasticsearch.name" . }}-snapshots
       restartPolicy: Always
       dnsPolicy: ClusterFirst
       nodeSelector:

elasticsearch/helm/templates/es-master.yaml

@@ -24,6 +24,8 @@ spec:
       containers:
       - name: es-master
         image: {{ .Values.images.elasticsearch }}
+        securityContext:
+          runAsUser: {{ .Values.elasticsearchConfig.runAsUser }}
         env:
         - name: NAMESPACE
           valueFrom:
@@ -58,6 +60,8 @@ spec:
         - mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
           subPath: elasticsearch.yml
           name: elasticsearch-config
+        - mountPath: /var/EDCOP/snapshots
+          name: elasticsearch-snapshots
 
       volumes:
       - name: esdata
@@ -69,6 +73,9 @@ spec:
       - name: elasticsearch-config
         configMap:
           name: {{ template "elasticsearch.fullname" . }}-config
+      - name: elasticsearch-snapshots
+        persistentVolumeClaim:
+          claimName: {{ template "elasticsearch.name" . }}-snapshots
       restartPolicy: Always
       dnsPolicy: ClusterFirst
       nodeSelector:

elasticsearch/helm/templates/snapshot-cronjob.yaml

@@ -0,0 +1,35 @@
+{{ if .Values.snapshotConfig.enabled }}
+apiVersion: batch/v1beta1
+kind: CronJob
+metadata:
+  name: {{ template "elasticsearch.fullname" . }}-snapshot
+  labels:
+    app: {{ template "elasticsearch.name" . }}-snapshot
+    heritage: {{ .Release.Service }}
+    release: {{ .Release.Name }}
+spec:
+  schedule: "{{ .Values.snapshotConfig.cronjob_schedule }}"
+  successfulJobsHistoryLimit: {{ .Values.snapshotConfig.successfulJobsLimit }}
+  failedJobsHistoryLimit: {{ .Values.snapshotConfig.failedPodsLimit }}
+  jobTemplate:
+    spec:
+      template:
+        metadata:
+          name: {{ template "elasticsearch.name" . }}-snapshot
+        spec:
+          containers:
+          - name: snapshot
+            image: centos
+            imagePullPolicy: Always
+            command: ["/snapshot.sh"]
+            volumeMounts:
+              - name: snapshot-config
+                mountPath: /snapshot.sh
+                subPath: snapshot.sh
+          volumes:
+          - name: snapshot-script
+            configMap:
+              name: {{ template "elasticsearch.fullname" . }}-snapshot-script
+              defaultMode: 0744
+          restartPolicy: OnFailure 
+{{ end }}