Usability considerations and execution options

[HeroicKatora]

Hi! First of all, siderophile is a great tool. We've tried it as part of identifying MIRI/fuzz targets and I wanted to inform you on our experiences with the tool and what we had wished to be possible.

No false positives as far as I can see!

It would be good if certain crates or paths can be excluded from the badness computation. There are a number of crates that extend the ability of the standard library carefully, execute their own MIRI test, run fuzzers or even mathematically proof check their implemenation (efforts ongoing on that front). Consider std itself is not part of the badness considerations. (This would require the std-src component to be feasible in the first place.) It makes a limited amount of sense to run those tools from the downstream consumer crates if only safe interface are accessed. The benefit is small because sound interfaces—like the whole of std—mustn't have any UB and the high level causes escalating computational effort to be expended on reaching the unsafe leafs in the first place. As an example, a large fraction of the badness in image is caused by bytemuck and byteorder. But contrary to other ad-hoc unsafety, these are carefully reviewed. This has the effect of hiding the ad-hoc cases that are more likely to be error prone.

Secondly, it would be incredible if siderophile could determine which functions with badness are not executed in a set of tests or fuzzers, and how many tests cover them! In terms of the call graph: Given the the test methods as roots find which bad methods are reachable from them. And conversely the program could answer which tests are purely functional tests and do not execute any unsafe blocks. This would allow a CI approach where MIRI only runs on tests determined relevant by the badness metric and guide the process of creating new test suites—which again would conserve computational efforts.

[woodruffw]

Thanks for the kind words, and sorry for the late response here!

I agree that both of these features would be great to have. Hopefully I (or someone else at ToB) will have time to work on them in the near future. That being said, PRs are always welcome as well 🙂