seed assignment based on target name has undesirable side effects

[sworland-thyme]

There are situations where it is desirable for a given target to return different output each time the target is built. e.g., stochasticity in a monte carlo simulation. This is currently not possible because a seed is assigned based on the name of the target. Considering that the seed value is available in tar_meta and can be retrieved for reproducibility, why not allow the seed to change each run? The example below will always return the same value unless the target name is changed.

tar_target(x, sample(1:100,1), cue = tar_cue_force(TRUE))

We have found some weird workarounds like including this in the function that requires a random component,

set.seed(as.numeric(Sys.time()))

which is why I didn't list it as a bug and opted to start a discussion around it.

[wlandau]

There are situations where it is desirable for a given target to return different output each time the target is built.

In those situations, as you say, you can call set.seed() in the target itself. But in most cases, committing to a seed is critical for reproducibility, and reproducibility is the highest priority of targets. The current behavior is a deliberate safeguard.

[sworland-thyme]

You can set the seed in the target itself but if you want the output to change each run you still have to find a way to dynamically change the seed value within the function like we did the Sys.time(). e.g., this will not work: set.seed(sample(1:100,1)) because the output of sample will remain the same each run, and hence the seed, and hence the output. Thank you for the quick reply!

[joelnitta]

@wlandau I take it from the principle that reproducibility is the highest priority (which I agree with) that you may be unlikely to implement this idea, but would you consider an option to turn off the automatic seed-setting by target? I have encountered behavior where I run set.seed() within a function to enforce my own seed, but I find the results are the same for some targets and different for others (sorry for the lack of a reprex, haven't worked it down to that point yet). But if there was an option to turn automatic seed-setting off, I could eliminate the source of the variation as coming from targets.

[wlandau]

I have thought about this, but for the reproducibility reasons you describe, I would prefer not to implement a way to disable this feature. For your use case, it sounds like it would be useful for debugging purposes. Another approach would be to force the same seed within the command of every target by applying a call to `tar_hook_before() to the entire pipeline.

[joelnitta]

Thanks! Just to make sure I understand... with the tar_hook_before() approach, is the idea that the automatic seed assignment still happens, but happens first and then the hook is run so that my custom seed will be set and override the automatic seed?

[wlandau]

Yes, targets still assigns a seed automatically, but then the hook resets it to the seed of your choice.

[wlandau]

ropensci/tarchetypes#111 and ropensci/tarchetypes#113 are already making the targets seed policy a little more complicated (for good reasons) so I am more open to this. And there's a reproducibility argument that any one seed is not magic and there should be a way to test the sensitivity of the results to the chosen seed. PR soon.

[wlandau]

see #959