Inappropriate usage of gensim models in gensim_models.py

[jonaschn]

The code for support of gensim models looks pretty old.
I am not sure if gensim (at the time of writing this code) didn't support better means to achieve the goals this code tries to achieve.

Example:

pyLDAvis/pyLDAvis/gensim_models.py

Lines 24 to 29 in 8e534a6

	# TODO: add the hyperparam to smooth it out? no beta in online LDA impl.. hmm..
	# for now, I'll just make sure we don't ever get zeros...
	beta = 0.01
	fnames_argsort = np.asarray(list(dictionary.token2id.values()), dtype=np.int_)
	term_freqs = corpus_csc.sum(axis=1).A.ravel()[fnames_argsort]
	term_freqs[term_freqs == 0] = beta

The LDA model does not offer the beta parameter because it is called eta in gensim.
Furthermore, the gensim's Dictionary offers the term frequency (across the collection) as model.id2word.cfs and document frequency model.id2word.dfs (in how many documents the term occurs).

[msusol]

Need to take a look at this after 3.4.0 release

https://radimrehurek.com/gensim/models/ldamodel.html
https://stackoverflow.com/questions/66111075/tuning-lda-topic-models
https://www.analyticsvidhya.com/blog/2021/06/part-3-topic-modeling-and-latent-dirichlet-allocation-lda-using-gensim-and-sklearn/