Feature: Improve dependabot detection thru PRs or parsing config file

[arieltorti]

Describe the bug
When running Scorecard against this repo https://github.com/tngan/samlify I get an score 0 on the Dependency Update Tool check, however the repo does have dependabot.

Reproduction steps
Steps to reproduce the behavior:

1. Run docker run -e GITHUB_AUTH_TOKEN=<token> gcr.io/openssf/scorecard:stable --show-details --repo=https://github.com/tngan/samlify
2. Check the score of the Dependency Update Tool.

Expected behavior
Dependency Update Check has a non-zero score because the repo has dependabot.

[laurentsimon]

do you have dependabot enabled as a GitHub setting rather than a config file under .github/?

[arieltorti]

I'm not the owner of the samlify repository so I can't fully confirm.

But after some testing on a private repo I own I noticed that dependabot is not detected if enabled through Github settings, I would suspect same thing is happening with samlify.

[laurentsimon]

yep that makes sense. We encourage owners to make their config public.

That said, we have plans to improve the check to look for dependabot PRs (accepted/rejected) to make the check more robust. Thanks for reminding us, I realized there was no tracking issue for it.

If you're interested in helping, PRs are welcome!

[laurentsimon]

Additional note: we should also check for reverted dependabot PRs by maintainers

[arieltorti]

That said, we have plans to improve the check to look for dependabot PRs (accepted/rejected) to make the check more robust.

Any ideas on how to implement this ? I've been looking at the Github API and haven't found a way to know if the repository has dependabot (or similar) enabled.
Another way would be to check for commits authored by the dependabot in the last N days, what do you think about it ?

[laurentsimon]

That said, we have plans to improve the check to look for dependabot PRs (accepted/rejected) to make the check more robust.

Any ideas on how to implement this ? I've been looking at the Github API and haven't found a way to know if the repository has dependabot (or similar) enabled.
Another way would be to check for commits authored by the dependabot in the last N days, what do you think about it ?

I think that is a good start and aligned with what we had in mind.
Do you know if this API https://docs.github.com/en/rest/reference/pulls#check-if-a-pull-request-has-been-merged can list PRs that have been rejected/dismissed as well?

@jeffmendoza please let us know if you have better ideas/insights for now.

FYI @inferno-chromium @naveensrinivasan @oliverchang @azeemsgoogle how many days shall we look back for merged PRs?

[laurentsimon]

see also a discussion step-security/agent#35 (comment), between dependabot security and dependabot dep. We should check whether the config file can differentiate between the two types of dependabot.

FYI @varunsh-coder

[laurentsimon]

(renaming the title of this issue to Feature: Improve dependabot detection thru PRs or parsing config file)

[azeemshaikh38]

Looking at PRs to get a hint about dependabot usage sounds like a good idea to me. I would say a good starting point would be to use the existing ListMergedPRs API. We can then consider expanding the support to non-merged or reverted PRs too.