ipop-usagereport0.1 with MongoDB

README.md

@@ -64,69 +64,99 @@ module][nginx limit req].
 Running server (without installing as an debian package)
 --------------
 ```
+# install MongoDB and Pymongo client
+sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5
+echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list
 sudo apt-get update
-sudo apt-get install git python-pip python-flask python-sqlalchemy sqlite3 python-yaml
+sudo apt-get install -y mongodb-org
+python -m pip install pymongo
+
+sudo apt-get install git python-pip python-flask pymongo python-yaml
 git clone https://github.com/ipop-project/ipop-stats.git
-cd ipop-stats/ipopstat-0.12/DEBIAN
+cd ipop-stats/ipopusagereport-0.1/DEBIAN
 sudo bash preinst
-sudo passwd ipop
+sudo passwd ipopreport
 Enter new UNIX password: ipop
 Retype new UNIX password: ipop
 passwd: password updated successfully
-su ipop
+sudo service mongod start
+su ipopreport
 Password:ipop
-cd ../usr/share/ipop-stat
+cd ../usr/share/ipop-usagereport
 ./run.py
 ```
 
 now you can access this server through webbrowser. 
 
-http://ip_address:8080/api
+http://ip_address:8081/api
 
 controller reports status info 
 
-http://ip_address:8080/api/submit
+http://ip_address:8081/api/submit
 
-sqlite database file is located /var/lib/ipop-stat/ipop-stat.db
 
-Simple database query snippets below. 
+database location:
 ```
-$sqlite3 /var/lib/ipop-stat/ipop-stat.db
-sqlite>select * from user;
-sqlite>select * from ping;
+/var/lib/mongodb/
 ```
 
-Building Debian Packages
-------------------------
-
-
-Locate at the parent directory of ipop-stat then run below.
+you can query database using Mongo shell as:
+```
+$ mongo
+> use usage_report
+> db.user.find()          # to get the data in database
+> db.user.count()         # to get number of documents in collection
+```
 
+Reset Database:
 ```
-dpkg-deb --build ipopstat-0.14
+$ mongo
+> show dbs
+> use usage_report
+> db.dropDatabase()
 ```
 
+Building Debian Package
+------------------------------------------------
+locate at the parent directory of ipopusagereport-0.1
+```
+$ cd ipopusagereport-0.1
+$ dpkg-deb --build ipopusagereport-0.1
+```
 
+Installation and Runnnig ipopusagereport debian package
+------------------------------------------------
+```
+sudo dpkg -i ipopusagereport-0.1.deb
+sudo apt-get update
+sudo apt-get -f install
+```
 
-Install and Running ipopstat from Debian Package
+Start, Stop, Restart ipopusagereport service
 ------------------------------------------------
+Start service: (make sure mongodb server is running before starting ipopusagereport)
+```
+$ service mongod start 
+$ service ipop-usagereport start
 
 ```
-sudo dpkg -i ipopstat-0.14.deb
-sudo apt-get update
-sudo apt-get -f install 
+
+Stop service:
+```
+$ service ipop-usagereport stop
 ```
 
+Restart Service:
+```
+$ service ipop-usagereport restart
+```
 
+Check status:
+```
+$ service ipop-usagereport status
+```
 
-Note for next time
-------------------------------------------------
-I need to stop the ipop-stat before update. 
-Add the command in preinst or something like "service ipop-stat stop"
 
-Issues
-------------------------------------------------
-For some reason, "service ipop-stat start/stop" does not work. 
 
 
 

client.py

@@ -1,16 +1,18 @@
-#!/usr/bin/env python
+#!/usr/bin/env
 
 import datetime
 import json
 import hashlib
-import urllib2
+import urllib.request as urllib2
+import urllib.parse
 
-data = json.dumps({
-  "xmpp_host" : hashlib.sha1("127.0.0.1").hexdigest(),
-  "uid": hashlib.sha1("abcd").hexdigest(),
-  "xmpp_username":hashlib.sha1("username").hexdigest(), 
+report_data = {
+  "xmpp_host": hashlib.sha1("127.0.0.1".encode("utf-8")).hexdigest(),
+  "xmpp_username": hashlib.sha1("xmpp_username".encode("utf-8")).hexdigest(),
   "time": str(datetime.datetime.now()), "controller": "test_client", 
-  "version": 3})
+  "version": 3}
+
+data = json.dumps(report_data).encode('utf8')
 
 try:
   url="http://" + "127.0.0.1" + ":" +str(8080) + "/api/submit"
@@ -22,6 +24,6 @@
     res.read()))
   if res.getcode() != 200:
     raise
-except:
-  print("Status report failed.")
+except Exception as error:
+  print("Status report failed. Error is {0}".format(error))
 

ipopstat-0.15/DEBIAN/control

@@ -0,0 +1,9 @@
+Package: ipop-stat
+Source: linux
+Version: 0.14
+Architecture: amd64
+Maintainer: Kyuho Jeong <[email protected]>
+Installed-Size: 41150
+Pre-Depends: dpkg (>= 1.10.24)
+Depends: python-pip, python-flask, python-sqlalchemy, sqlite3, python-yaml, python-tornado
+Description: Humble stat server

ipopstat-0.15/DEBIAN/postinst

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+service ipop-stat start
+update-rc.d ipop-stat defaults

ipopstat-0.15/DEBIAN/preinst

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+mkdir -p /var/lib/ipop-stat
+touch /var/lib/ipop-stat/ipop-stat.db
+touch /var/log/ipop-stat.log
+useradd -d /var/lib/ipop-stat ipop 
+chown --quiet ipop:ipop /var/lib/ipop-stat
+chown --quiet ipop:ipop /var/lib/ipop-stat/ipop-stat.db
+chown --quiet ipop:ipop /var/log/ipop-stat.log

ipopstat-0.15/etc/init.d/ipop-stat

@@ -0,0 +1,134 @@
+#! /bin/sh
+### BEGIN INIT INFO
+# Provides:          ipop-stat service
+# Required-Start:    $remote_fs $syslog
+# Required-Stop:     $remote_fs $syslog
+# Default-Start:     2 3 4 5
+# Default-Stop:      0 1 6
+# Short-Description: ipop stat 
+# Description:       ipop stat 
+### END INIT INFO
+
+# Author: Kyuho Jeong <[email protected]>
+
+PATH=/sbin:/usr/sbin:/bin:/usr/bin
+DESC="ipop stat"
+NAME=ipop-stat 
+DAEMON=/usr/share/$NAME/run.py
+DAEMON_ARGS="-c /usr/share/$NAME/config/debug.yml"
+PIDFILE=/var/run/$NAME.pid
+SCRIPTNAME=/etc/init.d/$NAME
+
+. /lib/lsb/init-functions
+
+do_start()
+{
+	# Return
+	#   0 if daemon has been started
+	#   1 if daemon was already running
+	#   2 if daemon could not be started
+	start-stop-daemon --start --quiet --pidfile --chuid ipop $PIDFILE --exec $DAEMON --test > /dev/null \
+		|| return 1
+        start-stop-daemon --start --quiet --chuid ipop --make-pidfile --pidfile $PIDFILE --background       \
+                          --startas /bin/bash -- -c "exec $DAEMON $DAEMON_ARGS > /dev/null" \
+		|| return 2
+}
+
+#
+# Function that stops the daemon/service
+#
+do_stop()
+{
+	# Return
+	#   0 if daemon has been stopped
+	#   1 if daemon was already stopped
+	#   2 if daemon could not be stopped
+	#   other if a failure occurred
+	start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
+	RETVAL="$?"
+	[ "$RETVAL" = 2 ] && return 2
+	# Wait for children to finish too if this is a daemon that forks
+	# and if the daemon is only ever run from this initscript.
+	# If the above conditions are not satisfied then add some other code
+	# that waits for the process to drop all resources that could be
+	# needed by services started subsequently.  A last resort is to
+	# sleep for some time.
+	start-stop-daemon --stop --quiet --chuid ipop --oknodo --retry=0/30/KILL/5 --exec $DAEMON
+	[ "$?" = 2 ] && return 2
+	# Many daemons don't delete their pidfiles when they exit.
+	rm -f $PIDFILE
+	return "$RETVAL"
+}
+
+#
+# Function that sends a SIGHUP to the daemon/service
+#
+do_reload() {
+	#
+	# If the daemon can reload its configuration without
+	# restarting (for example, when it is sent a SIGHUP),
+	# then implement that here.
+	#
+	start-stop-daemon --stop --signal 1 --quiet --chuid ipop --pidfile $PIDFILE --name $NAME
+	return 0
+}
+
+case "$1" in
+  start)
+	[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
+	do_start
+	case "$?" in
+		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
+		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
+	esac
+	;;
+  stop)
+	[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
+	do_stop
+	case "$?" in
+		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
+		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
+	esac
+	;;
+  status)
+	status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
+	;;
+  #reload|force-reload)
+	#
+	# If do_reload() is not implemented then leave this commented out
+	# and leave 'force-reload' as an alias for 'restart'.
+	#
+	#log_daemon_msg "Reloading $DESC" "$NAME"
+	#do_reload
+	#log_end_msg $?
+	#;;
+  restart|force-reload)
+	#
+	# If the "reload" option is implemented then remove the
+	# 'force-reload' alias
+	#
+	log_daemon_msg "Restarting $DESC" "$NAME"
+	do_stop
+	case "$?" in
+	  0|1)
+		do_start
+		case "$?" in
+			0) log_end_msg 0 ;;
+			1) log_end_msg 1 ;; # Old process is still running
+			*) log_end_msg 1 ;; # Failed to start
+		esac
+		;;
+	  *)
+		# Failed to stop
+		log_end_msg 1
+		;;
+	esac
+	;;
+  *)
+	#echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
+	echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2
+	exit 3
+	;;
+esac
+
+:

ipopstat-0.15/usr/share/ipop-stat/MANIFEST.in

@@ -0,0 +1,2 @@
+recursive-include ipop_stats/templates *
+recursive-include ipop_stats/static *

ipopstat-0.15/usr/share/ipop-stat/config/config.yml

@@ -0,0 +1,32 @@
+# http://flask.pocoo.org/docs/config/
+database_name: "usage_report"
+database_uri: "mongodb://localhost:27017"
+new_database: False
+flask:
+    debug: true
+    testing: true
+# http://docs.python.org/2/library/logging.config.html#logging-config-dictschema
+logging:
+    version: 1
+    root:
+        level: DEBUG
+    handlers:
+        console:
+            class: logging.RotatingFileHandler
+            stream: ext://sys.stdout
+            formatter: default
+        file:
+            class : logging.handlers.RotatingFileHandler
+            formatter: default
+            filename: /var/log/ipop-stat.log
+            maxBytes: 99999999
+            backupCount: 9
+    formatters:
+        default:
+            format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
+            datefmt: '%Y-%m-%d %H:%M:%S'
+ratelimit:
+    burst: 10 # allow a burst "pool"
+    uuid: 60 # seconds
+    ipv4: 60 # seconds
+    ipv6: 60 # seconds

ipopstat-0.15/usr/share/ipop-stat/config/debug.yml

@@ -0,0 +1,36 @@
+# http://flask.pocoo.org/docs/config/
+database_name: "usage_report"
+database_uri: "mongodb://localhost:27017"
+new_database: False
+flask:
+    debug: true
+    testing: true
+# http://docs.python.org/2/library/logging.config.html#logging-config-dictschema
+logging:
+    version: 1
+    root:
+        level: DEBUG
+    handlers:
+        console:
+            class: logging.StreamHandler
+            stream: ext://sys.stdout
+            formatter: default
+        file:
+             class : logging.handlers.RotatingFileHandler
+             formatter: default
+             filename: /var/log/ipop-stat.log
+             maxBytes: 9999999
+             backupCount: 9
+    formatters:
+        default:
+            format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
+            datefmt: '%Y-%m-%d %H:%M:%S'
+    loggers: 
+        all: 
+            level: DEBUG
+            handlers: [console, file]
+ratelimit:
+    burst: 10 # allow a burst "pool"
+    uuid: 60 # seconds
+    ipv4: 60 # seconds
+    ipv6: 60 # seconds