High baseline value in searchlight analysis

[mz973]

I'm doing some searchlight analyses on beta maps using crossnobis for between-condition distance and cosine similarity to compare RDMs, and I think this combination is recommended in the RSAtoolbox documentation for a meaningful zero point. But the results have very high baseline value which cause second-level outcome to be significant throughout the whole brain.

My data was collected over 12 runs, but because the conditions were not balanced entirely I cannot use a leave-one-run-out cross-validation. Instead each time it was a leave-two-run-out. The noise was estimated from measurement. I'm not sure if this could be the culprit. When I switched to "spearman" for RDM comparison the result looks normal.

I wonder if some of you have encountered this issue before and can point me to the right direction? Thank you all!

[JasperVanDenBosch]

Hi Mengya!
I've moved this to the Q&A discussion section as it seems to be less about the software and more a methods question (although I could be wrong). If you find the answer could you report it here for future visitors?

When you mention the high baseline value, I'm assuming you mean on the data (not the RDM)? And what sort of second level analysis are you applying? It may help to post some screenshots here.

[JasperVanDenBosch]

Can I ask what your size is for the searchlight sphere?
If I understand correctly the statistic here is the second-level stat across participants, where in each participant you are using the fit of a single model against the data RDMs? It is typical for cosine to give a higher value in general. Have you considered comparing the model against others?

[mz973]

The radius is 3 voxels.
Yes I was referring to the second-level stat across subjects, specifically in each voxel is the t-score of a group comparison against 0 (in the screenshot it is not tscore though, just averaged correlation values across people).

I can certainly have competitive models, but I also want to understand why using cosine similarity gives high values everywhere. Shouldn't it generate meaningful zero points like the documentation has explained? And on the subject of having multiple models, I have not tried it before with searchlight class - is it as simple as combining ModelSelect with evaluate_model_searchlight?

Much appreciate your swift replies btw!

[JasperVanDenBosch]

Afaik, the meaningful zero point applies to the dissimilarity values in the RDM, not necessarily to the model comparison.
If you were to use ModelSelect in this context, you'd be interested in the thetas (weights) on the Result objects returned, not the model fit result (as this would be the either/or fit).
To be honest, we're not currently actively maintaining the searchlight code since we are planning to do a major rework of it in the near future. One reason for this is that model comparison isn't well integrated for the searchlight right now..
Happy to help

[mz973]

Ah ok, I was misunderstanding it. Thanks for clarifying!

My experience with the toolbox has been only with ModelFixed. I will give Modelselect a try as there is certainly a use for it too. Will report back if for some reason it doesn't run and I couldn't fix it. Cheers again.

[HeikoSchuett]

Hi @mz973 ,

just as a short note: the high baseline value you observe is entirely expected for cosine similarity. This similarity measure behaves differently from correlations. Due to all true distances being positive, all true cosine similarities between RDMs are positive and typical similarities are substantially different from 0. This makes tests whether this is significantly different from 0 meaningless. If you need such a test, either compare against a null model (like a constant RDM) or use a correlation as your similarity.

best
Heiko