tokenize articles

…ic_modelling

…r yet)

2. A path helper to assit import
3. modified token_pool to make it compatible with LDA model

… into topic_modelling

…ic_modelling

1. Made every variable and method priavte if possible
2. Reformatted code with Pycharm shortcut
3. Added tests for TokenPool (works well) and ModelGensim (does work due to 'no module named XXX' problem when model_gensim is calling its abstract parent)
4. Decoupled token_pool and model_*
5. Used if __name__ == '__main__' to give a simple demonstration on how to use each mehtod

Model_*
1. Renamed mode_lda.py and model_lda2.py to model_gensim.py (which uses the Gensim package) and model_lda.py (which uses the LDA package)
2. Added a abstract parent class TopicModel.py
3. Moved some code from summarise() to add_stories() (a. better structure of code; b. improved performance)
4. Changed some constants to function arguments (e.g. total_topic_num, iteration_num, etc.)

TokenPool
1. Added mc_root_path() when locating the stopwords file
2. Modified query in token pool:
	1. added "INNER JOIN stories WHERE language='en'" to guarantee all stories are in English
	2. added "LIMIT" and corresponding "SELECT DISTINCT ... ORDER BY..." to guarantee only fetch the required number of stroies (thus improves performance)
	3. added "OFFSET"
3. Restructured token_pool.py, so that the stories are traversed only once (thus improves performance)
4. Decoupled DB from token_pool.py
5. Replace regex tokenization with nltk.tokenizer
6. Added nltk.stem.WordNetLemmatizer to lemmatize (which gives a better result than stemming) tokens

…ic_modelling

…model_lda

The result of this algorithm is similar but slightly different from the LDA model
+
It allows multiple topics for each story

…ic_modelling

… into topic_modelling

2. renamed a few methods/variables due to the change of functionalities

…view, leave the alternative query and related code as comments

2. Allowing TokenPool to take either a DBHandler or a TextIOWrapper

they are now taking the stories in the sample file as input

…files

1) Faster (Travis doesn't have all day)
2) We only use WordNet at the moment

This reverts commit db1c584.

…xisting files"

This reverts commit a361b01.

This reverts commit d995cb8.

This reverts commit 0e04ff1.

This reverts commit 9c68669.

1) Faster (Travis doesn't have all day)
2) We only use WordNet at the moment

2. fix newly occurred pycharm warnings (expect iterator get list)

…ic_modelling

Before running unit tests, Travis installs all Perl and Python
dependency modules which takes up a lot of time and doesn't always leave
enough time (of the available 50 minutes) to complete all the unit
tests.

After a successful unit test run, Travis caches all the installed
dependencies so that it doesn't have to install anymore and can get to
running unit tests themselves faster.

So, we temporarily disable the unit tests (replace them with a simple
"echo" statement) for Travis to be able to install the dependencies and
cache them. Subsequent Travis runs (with actual unit tests reenabled)
will then be able to use the pre-cached dependencies.

This reverts commit 36817b9.

Caching fails because Travis is unable to find /usr/share/nltk_data for
whatever reason:

https://travis-ci.org/berkmancenter/mediacloud#L3361

...and so nothing gets cached (including Perl dependencies which take a
long time to install), and so builds time out.

This reverts commit 8861d9e.

…nt accuracy

…abstract class

added method named evaluate as in the abstract class

…ic_modelling

… into topic_modelling

…ic_modelling

…d based on a few points

…ihood comparisons

…ter efficiency and performance

I will combine these two later

…en combined with tune_with_polynomial

…ic_modelling

… and cache dependencies

This allows more flexibility in Travis (i.e. use larger samples if we can run tests longer in Travis)

2. improve performance based on empirical results