Bayesian Minimum Error Rate Classifier

Running the model

Create a new directory Data and place the csv files containing the data of the two classes (separately) in it.

Add the relevant column names to the list features in the binClassifier.py. Assign the split values to split1 and split2 in binClassifier.py.

On running binClassifier.py, the dataset is shuffled and sampled 100 times. The mean, minimum and maximum accuracies are printed(classwise and overall).

Mathematical Background

Likelihood

The probability density function used is the multivariate normal distribution. the likelihood p(x|w_i) is given by

x is the d-dimensional feature vector, μ is the mean vector, Σ is the covariance matrix,|Σ| is the determinant of the covariance matrix, Σ⁻¹ is the inverse of the covariance matrix and (x − μ)^t is the transpose of the (x - μ) vector. p(x) is calculated using the covariance matrix of the data of a class w_i.

p(x) for each of the classes is computed given the equation for multivariate normal distribution. This would be p(x|w_i ) for i = 1, 2 (being a binary classifier).

Apriori Probabilities

the apriori probabilities P(w₁) and P(w₂) are calculated using

P(w_i) = (numberof data points in w_i) / (total number of datapoints)

Evidence

the evidence for each data point(in the test set) is calculated using the equation

p(x) = P (w₁) ∗ p(x|w₁) + P (w₂) ∗ p(x|w₂) (being a two category case)

Posterior Probability

Using Bayes rule, the posterior probability (Conditional probability) is found for each of the two classes.

posterior probability = apriori probability ∗ likelihood / evidence

Now with the conditional probabilities computed for each of the two classes, we can make a prediction based off of the values of P(w₁|x) and P (w₂|x).

Prediction

And being a minimum error rate classifier, we define the discriminant function g_i(x) as P (w_i|x). if P(w₁|x) ≥ P (w₂|x) (i.e., g₁(x) ≥ g₂(x)) then we predict the class to be w₁ and predict w₂ otherwise.