Conditional mutual information in pyagrum.

[kenneth-lee-ch]

Suppose I have a pandas dataframe that is a discrete dataset, how can I go about using the function pyAgrum.InformationTheory.mutualInformationXYgivenZ() WITHOUT having a Bayesian network? Any pointers or examples will be greatly appreciated!

[phwuil]

Hi @kenneth-lee-ch ,
when dealing with data, a good idea is to look at BNLearner :

https://pyagrum.readthedocs.io/en/1.13.1/BNLearning.html#pyAgrum.BNLearner.mutualInformation

PS- oups, the description of "returns" is not correct ...

[phwuil]

Fixed ! c55b7f7

[kenneth-lee-ch]

But how come it is not returning the actual mutual information instead of G statistic (I am guessing its some sorts of CI tests)?

…

On Sat, May 4, 2024, 5:03 AM Pierre-Henri Wuillemin < ***@***.***> wrote: Hi @kenneth-lee-ch <https://github.com/kenneth-lee-ch> , when dealing with data, a good idead is to look at BNLearner : https://pyagrum.readthedocs.io/en/1.13.1/BNLearning.html#pyAgrum.BNLearner.mutualInformation PS- oups, the description of "returns" is not correct ... — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF3Z3CUOS647PRSNCV6NWETZASP5RAVCNFSM6AAAAABHF7ZMB6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGMJSGU2DM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[phwuil]

It does return the actual (log2) mutual information. This is just the documentation that was wrong.

(G2 and chi2 both return the value of the statitistics and the p-value)

[kenneth-lee-ch]

@phwuil is there a way I can directly compute mutual information from the bn.cpt without samples? How can I use these functions? https://pyagrum.readthedocs.io/en/1.9.0/BNToolsInformationTheory.html

[phwuil]

Hi @kenneth-lee-ch , here also, the doc must be improved ... :-(

[phwuil]

(in the next release)

[kenneth-lee-ch]

@phwuil thanks for all the help. I wonder if you know of any scalable way of computing conditional mutual information? The kind that can scale with 1000+ variables if exists..? When I use bn.learner, it basically kills the program when I condition on around 30 variables.

[phwuil]

Hi @kenneth-lee-ch, you have to take into account the inherent complexity of this computation :

IF Z consists in 30 variables, the sum over Z represents (at least) 2^30 iterations. Do not try this at home if Z contains 1000 variables :-)

The way pyAgrum organizes the computation of the conditional mutual information is to first compute p(x,y,z) in a unique inference because all the probabilities in the above formula are somehow marginalizations of this joint probability. Of course if Z contains >30 variables, this is not a correct solution since you need at least 2^30 double numbers in memory to store this joint probability.

With an inference in each iteration (rather than an inference before iterations), you could get rid of this memory problem but at the cost of a significant increase in computing time of each iteration ... And you still have at least 2^30 iterations to perform ...

[kenneth-lee-ch]

@phwuil I tried to directly derive mutual information based on a randomly generated bn via the following code below

import pyAgrum as gum
import numpy as np
bn = gum.randomBN(n=10) # 
ie = gum.LazyPropagation(bn)
names = list(bn.names())
n = len(names)
mi_matrix = np.zeros((n, n))

for i in range(n):
    for j in range(i+1, n):
        it = gum.InformationTheory(ie, names[i], names[j])
        mi = it.mutualInformationXY()
        mi_matrix[i, j] = mi

print("Pairwise Mutual Information Matrix:")
print(mi_matrix)

But I run into this error, shouldn't I use it.mutualInformationXY() for computing mutual information given any pair?

UndefinedElement: [pyAgrum] Undefined element: The variable X03(2) does not belong to this optimized inference.

[phwuil]

Hi, you found a problem : it seems that (a part of) the state of the inference is kept from an InformationTheory instance to another. For now, a (not so expensive) workaround is to create the LazyPropagation object in the loop :

import pyAgrum as gum
import numpy as np

bn = gum.randomBN(n=10) # 
mi_matrix = np.zeros((bn.size(), bn.size()))

for i,ni in bn:
    for j,nj in bn:
        if i<=j:
            continue
        mi_matrix[i, j] =  gum.InformationTheory(gum.LazyPropagation(bn), 
                                                 ni,nj).mutualInformationXY()

print("Pairwise Mutual Information Matrix:")
print(mi_matrix)

(Note the loop in bn that iterates over all the pairs (node_id,node_name).

[kenneth-lee-ch]

I see that some entries are negative values in the matrix. Is this an expected behavior?

[phwuil]

This is just floating point errors :

[phwuil]

Note also that this code has a small problem : it assumes that the nodeids are integers from 1 to n. This is not always the case :

However, if you do not use the method bn.erase(n), this assumption is valid.