ENH: calculating d-prime from confusion matrices and samples

[hahong]

Two functions were added: dprime_from_confusion_ova() and dprime_from_samp()
Note: unittests should be added!

[npinto]

Thanks, this looks good! It needs some refactoring + tests but it should be ready to go. Comments to follow.

[npinto]

not sure about the _ova part, it could be made more generic if one provide the binary output codes used to compute the confusion matrix -- what do you think?

[npinto]

By default it could be interpreted as OvR (only if it is square), but any binary-like output code could work -- we would just need a convention for it (e.g. -1 = False, +1 = True, 0 = ignore), at this stage we should just assume it's OvR and let the user know in the docstring, but possibly open up the possibility of other output codes (e.g. OvO).

Thoughts?

[hahong]

That is a definite possibility. I like that convention --- will work on that.

Meanwhile, the current version of dprime_from_confusion_ova() is meant to be used when there's no access to internal representation / decision making (like human data). This function computes n OVA d's from the given n-by-n confusion matrix --- but I admit that there's no clear analytical/mathematical connection between d' computed from a n-way confusion matrix and a 2-way binary classifier.

[npinto]

Do you have real-world tests that could be reduced to regression tests?

[hahong]

Which part are you mentioning? (Didn't you put this comment by mistake?)

On Wed, Jul 18, 2012 at 7:48 PM, Nicolas Pinto <
[email protected]

wrote:

Do you have real-world tests that could be reduced to regression tests?

---

Reply to this email directly or view it on GitHub:
#8 (comment)

[hahong]

Methinks this would be closer to what you've suggested. Take a look and let me know if you have some comments!
PS: passed flake8

[npinto]

Hu? Why is accuracy changing here?

[hahong]

I added a support for confusion matrices in accuracy() as in dprime(). There are some changes/rearrangements, so might be good to take a look at the whole code.

[npinto]

@hahong, your experimental features on dprime could go in wildwest if you'd like.

[hahong]

(Gosh.. time goes fast.) You mean the "wildwest" branch? ;-)
How could I make these new features included in a more major branch?

[npinto]

You mean the "wildwest" branch? ;-)

Actually the wildwest submodule. What do you think?

[hahong]

Aha, I see. That sounds good to me.
Btw, can I ask the criteria for wildwest and non-wildwest? I was just curious, in particular, whether you have some preferred coding styles, patterns, etc.

[npinto]

You can basically consider wildwest as a sandbox, not as strict w.r.t API, docs, tests, etc. wild west ;-)