Demo

[gyachdav]

Hello there #PokemonGo,

It is time to see a live demo of your apps. You can either:

• Stage your demo on a hosting service

or

• Give a detailed explanation in your README how to run your apps.

Either way, we need to see that status of your project.

[sockeqwe]

@gyachdav I'm a little bit confused about the general setup / deployment of the project.
Our project is not really a stand alone project. I thought it will be integrated in Project A i.e. we write into Project A's database (direct database access, we don't communicate over a REST api with project A). Also, should we provide a public API? I thought this will be part of project A, isn't it? See #8

Since we are just generating data (no GUI), what kind of demo would you like to see?

[gyachdav]

@sockeqwe the output of the apps you build is indeed being integrated into project A. However it would be beneficial for us to be able to run even a command line version of your app and see what that output is. Furthermore it would be interesting to see what params and calls we need to make to get to this output. We also want to test your apps under different use cases. No public API is needed but some form of interaction with your app is required.

Keep in mind that you're suppose to create an NPM packages for each app. Project E is suppose to assist you with that. The NPM pack should allow us to deploy your app and the docs should give us a clear idea on how to run your app and what output to expect.

Hope this clarifies it a bit. Feel free to ping me some more in case I did not answer your questions or if you have additional questions or concerns.

[phdowling]

Hi @gyachdav. We currently have an index.html file which I use to debug the websockets output generated by e.g. the pokemob detection module. We could add some more functionality there (live sentiment tweets logging and showing the output of our sentiment trend module) and have this available as a standalone demo. All you would need to do to run is run node main.js to start the server, and open the index.html file somewhere and look at the console output.

Would a setup like that be okay to test our applications stand-alone?

Regarding the other points:

• I was thinking we can point project E to our main.js, which shows what modules one needs to import and how to start the different processes. The actual modules should then be imported through npm though (any good resources on how to easily package and publish modules there would be appreciated by the way!).
• We import stuff from project A as far as I see, so if project E requires both project A and our project, will things work out? for instance, would we actually end up referencing the same objects if they are global?
• who should import our code in the end? currently project A is somehow hosted, but from what I understand, in the end project E will import every other module and set up all the server logic centrally, right? this would be important to clarify.

[gyachdav]

@phdowling yup hat setup will be perfect.

I was thinking we can point project E to our main.js, which shows what modules one needs to import and how to start the different processes. The actual modules should then be imported through npm though (any good resources on how to easily package and publish modules there would be appreciated by the way!).

Keep in mind that project E is suppose to help you pack your apps. That's why they have 8 people there. summoning @PokemonGoers/catch-em-all .

Also keep in mind that project E is not going to import your apps. You will be running your listeners as a node process. the output from that process is stored on the project A database. @sacdallago can help you set up the process on the dev server. You will need to consult with @PokemonGoers/pokedata on how to store your output in their db.

We import stuff from project A as far as I see, so if project E requires both project A and our project, will things work out? for instance, would we actually end up referencing the same objects if they are global?

I don't see project E require project A, instead they will be making API calls to fetch data for the wiki portion of their webapp. Most of the API calls to project A's data will however be initiated by the map groups who will visualize sightings, predictions and pokemobs indications.

who should import our code in the end? currently project A is somehow hosted, but from what I understand, in the end project E will import every other module and set up all the server logic centrally, right? this would be important to clarify.

The only case in which I see your apps being directly integrated into E is the sentiment visualization component. This one will need to be provided as an exported module that can be required by E.

Let me know if this makes sense.

[sockeqwe]

Also keep in mind that project E is not going to import your apps. You will be running your listeners as a node process. the output from that process is stored on the project A database. You will need to consult with @PokemonGoers/pokedata on how to store your output in their db.

@gyachdav so we provide a npm library to project A (so project A is requiring our code) so that they can include our code?

concrete example:
We periodically query Twitter for tweets with tags to run sentiment analysis. This will run as a node process. So we provide a function to start this periodically sentiment analysis by packaging that as a npm library. Then Project A (as they are hosting the backend as far as I know) will require our function to periodically run sentiment analysis and Project A will start that function as a node process.
I thought about something like this:

// This function runs sentiment analysis periodically.
// this function is provided by our npm library, but will invoked by Project A in a own node process.
function runSentimentAnalysis(twitterCredentials, databaseConfig) { 
   // 1. Query Twitter REST API for tweets
   // 2. Sentiment Analysis on each tweet
   // 3. Save sentiments into database
   ... 
} 

Do we agree on that?

if yes, then, we also have to provide a REST api for the maps team where I would take the same road.
Provide some functions like this as part of the npm module:

function getSentimentFor(pokemonId, lat, lng){
    // Execute some Database query and return sentiment data over time
   ...
}

Then Project A will add a route to their express router:

router.get('/sentiment', function(req, resp){
   // Execute our function
   getSentimentFor(req.query.pokemonId, req.query.lat, req.query.lng);
   ...
}

Have I understood that correctly? If not, can you please give me a concrete example of what the architecture / communication should look like in your opinion?

I would also do the same for mobdetection etc. just provide a exportet npm module so that Project A can require our code.

[gyachdav]

Yup that sounds about right although you may elect to just run the node process separately from A. The rest of the architecture makes sense.

[phdowling]

@gyachdav Sounds good, I think we will probably go for the separate node process in the end - two issues remain with this though:

1. Will/can we be hosted on the same server as project A? If so, we could still access their mongoDB instance without actually running in the same process.
2. One of our Apps has a REST endpoint, while the other two only provide websocket endpoints. The REST API would have to be on a different port if we run a separate process. Is this acceptable?

Cheers

[johartl]

One of our Apps has a REST endpoint, while the other two only provide websocket endpoints. The REST API would have to be on a different port if we run a separate process. Is this acceptable?

At project E we have a little node server that intercepts requests to /api and proxies them to pokedata.c4e3f8c7.svc.dockerapp.io:65014. We would then do the same for requests to /sentiment and send them to the port where your application is running at. So the different ports shouldn't be a problem at all.

In terms of deployment I see two different approaches.

1. Both server applications can be packaged into the same Docker container which runs the two processes alongside each other and also exposes both ports. One team would need to write an entry script that launches the two server instances (and shuts them down gracefully upon receiving a stop signal). This would work just fine but is somehow inconsistent with Docker's philosophy of 'One application per container'
2. The HashPokemonGo team build their own Docker container. This is a lot cleaner but also requires a bit more work at first. The mongoDB instance can still be accessed even if it is running on a different host. I think in the current setup the PokeData Docker container and the MongoDB server are also running on different hosts. This might change in the final production setup though? @sacdallago can probably say a few words here.

[sacdallago]

@phdowling @johartl
I would strongly advise to keep the REST APIs to one (as in: when you are done building that part, simply coordinate with project A to integrate it directly in their API in some route). At the end of the day, you can write the entire logic to the controller and package it in an NPM package, then the only thing to do will be for A to create the routes and use your packaged functions. Even easier is if you write an express middle layer, then they would just need to add the express.use('/some/rute', myMiddleware) directive.

As far as mongodb: it's always best to separate logic from data, so yes: currently using a cloud mongodb instance, but I'm thinking of migrating to something in RostLab because of space (we only get 500mb for free).

[sockeqwe]

@gyachdav @sacdallago
Demo is available now:

Setup / Running standalone demo

Ensure that mongodb server is running locally on your machine and that database with name /test is accessible (without authentication / password etc.)

Simply clone the repo (develop branch) and run npm install. To run the server, use (Twitter credentials)

CONSUMER_KEY=<your_consumer_key> CONSUMER_SECRET=<your_consumer_secret> ACCESS_TOKEN=<your_access_token> ACCESS_TOKEN_SECRET=<your_access_token_secret> npm start

A demo webapp is also provided: once the server is running, open localhost:8080 in your browser for live sentiment analysis and cluster/mob detection, and provide an interface to query historic tweet sentiment for different pokemon.

Keep in mind that for the historic tweet sentiment analysis the server will query for tweets periodically to generate some data. So you may want to run the server for some minutes so that some data has been mined that can be displayed.

Feedback is very welcome!

[johartl]

Any news with regards to integrating the mob detection websocket into project A's backend?
The PR in PokemonGoers/Catch-em-all#65 will integrate the map of the PokeMap-2 team and it would be nice if the map could already connect to the websocket. Right now we just get lots of errors because the connection to the websocket fails.

[sockeqwe]

Our code is stable and production ready. we can integrate it with project A ... We only have to check what's the best way to integrate it. maybe we will provide our code through a npm package to project A. I will take a look at npm packaging ...

[sacdallago]

@sockeqwe yes, I'd say the NPM solution is the cleanest.