Changing a column name in dataset returns different results. Should I raise an issue?

[ivivalle]

Hi everyone!

This is Ivan. I am a newbie here but working with Robyn a few months (newbie too I guess).

I would like to share with you something that happened while running several tests with my dataset, in order to find out whether it might be an issue or not.

I have four channels with costs named in the dataset as: Employment, Events, TP_Events, and Customers.
Running Robyn with these names returns result X (let's call it like that): sorting alphabetically desc (internally) is displayed like below in the one-pagers:
TP_Events, Events, Employment, Customers.

Well, it turns out that if I change the last column name to Sale_Costs, for example, the results are Y, not the same anymore despite the data hasn't been touched, remains the same. Only that column name was changed. The sorting this time in the one-pagers is like:
TP_Events, Sales_Costs, Events, Employment.

Both results pulled out by Robyn are different (R2, model, one-pagers plots, etc.) by only changing one column name (data in dataset and Robyn code remains the same), which modifies the internal alphabetically sorting of Robyn and based on this, it goes towards different outputs.

Might it be an issue? Do you recommend me to raise one?

Many thanks in advance for your help.
Best regards.

[gufengzhou]

Robyn has a evolutionary process (see Nevergrad) that includes certain randomness. Different dataset (incl. different names) might result in different seeds. Although, Robyn outputs multiple results. I believe you should be able to find similar results between these runs.

[ivivalle]

Hi!
Thank you for your response.

I do understand that the algorithm uses certain randomness but it is quite surprisingly that running twice the same dataset both results are exactly the same. However, by just changing a column name only (data still the same) which alters the alphabetical order of the channels only, Robyn's output is quite different (even though I've tested this again with a fixed "seed = 123"). It is kind of confusing, isn't it?

Then, I guess that within the randomness Nevergrad uses is the alphabetical order of the columns names in the dataset, depending on how they are sorted, the results will vary. Do you know if the difference lies in the models Robyn chooses and in fact the same results should be found in the results? If so, how does Robyn choose the Pareto outputs?

Thanks!
Best regards.

[gufengzhou]

as you know, robyn outputs many model candidates. by changing column names, it definitely triggers different random process. but it'd surprise me if you can't find similar results among all the outputs? Also you could set higher min_candidates in robyn_outputs to get more candidates.