AEGIS Integrated Services
The AEGIS platform is built upon Hopsworks Integrated services. For deployment and debugging refer to this document.
When installing the Hopsworks services, the karamel tool, developed in-house is used. There are two ways of using karamel:
- Use karamel binaries you can get here together with a cluster definition file to deploy Hopsworks on a bare metal machine or a cloud instance (gce, aws)
- Use the run.sh script within karamel-chef together with a cluster definition file and a vagrant file in order to deploy Hopsworks within a vm.
Note: Use karamel v0.5 together with aegis/hopsworks v0.9 in order to deploy the latest version of the platform.
- Software required on the machine where the vm will be created.
- chef-dk, version >0.5+
- git
- vagrant
- vagrant omnibus plugin
- virtualbox
- Cluster definition
- Vagrant file
- Clone karamel-chef where both the above cluster definition and vagrant file are located and execute the run.sh script:
./run.sh ubuntu 1 aegis
Note: For more information look here
The AEGIS platform Testbed is accessible from AEGIS
A user can create an account by providing his information to the platform using the following registration page.
Once a user is registered, he/she can login to the platform. The main page after logging in is as follows. The user can create a new project by clicking on the NewProject button.
Once the project is created the user can navigate to the project by clicking on the project. The project is pre-populated with a set of default datasets such as Logs where log files reside, Resources, etc.. On the left pane of the project main page exist the project settings and members controls to control the project details and add/view/delete members to/from the project. AEGIS provides a set of integrated notebooks "AEGIS tools" that includes the Query Builder, Visualizer, and Algorithm Execution Container. Moreover, the AEGIS platform offers support for data parallel processing frameworks using the Jobs service, and real-time analytics using Kafka, and deep learning model serving using Tensorflow.
The AEGIS platform provides support for data parallel processing frameworks such as MapReduce, Spark, and Flink through the use of the Jobs service. To run a job, the users need to first upload their job’ files (.jar, .py, .ipynb) to a dataset. Then, they can add a new job by clicking on the New job button as shown in the following figure.
To create a new job, the user provides the job name, type, configuration, and other external libraries as shown in the figure below.
Once the job is created, the user can run the job, and then look at the execution logs listed at the end of the page. Also, the user can rerun, edit, delete, and export the existing job or add new jobs.
The AEGIS platform provides real time analytics using the Kafka service. The users can create topics and their respective schemas using the user interface provided by AEGIS, as shown in the following figure.
After clicking on the new topic button, the user is required to fill the topic details such as topic name, number of partitions, number of replicas, and the topic schema. The user can select a schema from the existing schema in the platform or create his own schema first by navigating to the schemas tab.
The user can create his/her own schema by clicking on the new avro schema button and then provide the schema’s details.
The AEGIS platform uses Tensorflow for developing deep learning models. It provides a model serving service to deploy pre trained model using the Tensorflow serving. The user can create a new serving by providing a model name, version, and a path.
The REST API documentation for the core components of AEGIS can be found in Swagger
- First, you will need to login to hopsworks using auth service.
- The response will contain the jwt taken under the header "authorization"
- For any subsequent requests, you will need to set the Authorization header with the token value
The token expiration interval is 30 minutes and will be renewed automatically on any successful REST call. Tokens will be automatically renewed if the token’s renewable field is set to true. This is done when a successful request (response 2xx HTTP status codes.) is made to hopsworks with an expired token within the renewal period (i.e. expiration+expLeeway). expLeeway is set to 15 min by default.
If the token expired for a period longer than the renewal period (expiration+expLeeway, 45 minutes by default), then the application will require to re-login to get a new token.