Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make fuzz vary by rating and over the season #344

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

nicidob
Copy link
Contributor

@nicidob nicidob commented Jan 16, 2021

still has some issues

  1. Doesn't disable itself for draft prospects
  2. is a binary flag for playoffs. would love for it to be a smooth curve but I couldn't figure out how to do a fast 'games played' query without async/await.
  3. can reverse engineer it. Check in playoffs for prior season (no fuzz) -> look at player in new season for that same prior season.
  4. Not sure if it's well balanced as is. RAPM is std dev of 2 in +/- terms. Could use that as guideline.

return ovrC;
};

const fuzzRating = (rating: number, fuzz: number, isDraft = false): number => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would need to accept season as a parameter too, right? Otherwise fuzz for past season ratings would be constantly oscillating.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could default to g.get("season") since in many places it's only the current season, probably.

Copy link
Contributor Author

@nicidob nicidob Jan 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But you don't store fuzz per season, do you? That'd need to be added? At which point, better remove the storage for fuzz entirely and make it something you'd want to do deterministically on the fly then (hash pid, season and a per-league salt as the seed to an RNG).

pf.ratings.ovr,
Math.round(p.ratings[1].ovr + p.ratings[1].fuzz),
);
// fuzz is not this simple anymore
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should just be deleted rather than commented out

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking was I should add tests. Check for decent bounds. Check that the +/- functions behave correctly. I don't think losing tests make sense, especially when the code just got more complicated. I should just write some additional replacement test.

@dumbmatter
Copy link
Member

  1. Doesn't disable itself for draft prospects

I can do that.

  1. is a binary flag for playoffs. would love for it to be a smooth curve but I couldn't figure out how to do a fast 'games played' query without async/await.

Assuming 82 games is enough to get as accurate as we're going to get is fine, I think.

  1. can reverse engineer it. Check in playoffs for prior season (no fuzz) -> look at player in new season for that same prior season.

What about this...

Fuzz here is a random number. So if your 35 year old player goes up, you'd assume that 99% of the time it's just fuzz.

What if instead fuzz was a random number picked from a distribution centered around the expected change in ratings (given a player's age)? Factoring in the actual change in ratings. So like if a 35 year old is expected to decline by 5 points, but actually declined by 3, his fuzz would be drawn from a distribution centered around -2.

Another complication - ideally that would be per rating, since different ratings decline differently.

Is this too complicated? It sounds kinda cool though...

@dumbmatter
Copy link
Member

Can you sell me on fuzzing on the +/- scale rather than the ovr scale? Is it worth adding that complexity, and having to do something else for football?

@nicidob
Copy link
Contributor Author

nicidob commented Jan 17, 2021

Can you sell me on fuzzing on the +/- scale rather than the ovr scale? Is it worth adding that complexity, and having to do something else for football?

So there's a handful of reasons

  1. I think we're more confident in higher quality players, in general. Fuzz should be lower for higher overall players. Giannis and LeBron and Curry are probably more 'certain' in their ratings than Jamal Murray & DeRozen, who are more certain than some bench/role players like Jordan Clarkson or Austin Rivers. I think this is true for ratings as a whole -- it's easier to be confident in a 'great' 3P shooter than knowing exactly how middling someone is. A kind of crappy analogy: it's easier to grade tests that are nearly correct; it's a lot harder to grade the tests from the middling students because of partial credit and the like.
  2. The non-linear sigmoid does a lot of this work for you. With some noise -- Great players stay great and players in the middle move around. One way to see the sigmoid is really just a transformation from standard deviations to rank order (logistic is roughly gaussian). So you're taking Curry and saying he's a +3σ player, and then fuzzing that to +2.75σ or +3.25σ) -- which won't change that he's in the top 5% (94% to 96.3%). While taking the mean player (Otto Porter?) and fuzzing him from -0.25σ to +0.25σ will change his result more (43.8% to 56.2%). The fact that I got this sigmoid from fitting a +/- result doesn't really matter for this effect. You can store a sigma and a mean and you'll get very similar results.
  3. This +/- sigmoid is actually linear near 70 ovr and nearly flat below 40 overall since its mean is 73ovr. And it came from trying to compute team MOV as a function of the top 10 overalls, and it seems coherent with NBA +/- ideas (minimum is -2. high 50s is 0, below that is negative impact. 70 ovr is around a +5, an 80 would be +8).
  4. the fitting results of RAPM do obtain a standard deviation of around 2. Likewise, box score +/- also has a variance in +/- terms. So any estimates of +/- that we see about NBA players have a fuzz/uncertainty in that same scale. We're as sure that LeBron is +5 to +7 as Willie Cauley-Stein is +1 to -1 or Dragan Bender is -1 to -3.
  5. You could in theory pick a different sigmoid (like mean = 50, std_dev = 10). Or even a linear function where f(100) = 0 and f(0) = 1 and you scale the fuzz linearly. Using +/- just seemed in line with major stats, but honestly any of those curves would probably work.

@nicidob
Copy link
Contributor Author

nicidob commented Jan 17, 2021

What if instead fuzz was a random number picked from a distribution centered around the expected change in ratings (given a player's age)? Factoring in the actual change in ratings. So like if a 35 year old is expected to decline by 5 points, but actually declined by 3, his fuzz would be drawn from a distribution centered around -2.

That's also be pretty interesting. I like the idea that the uncertainty is drawn from progs distribution. That handles the age factor as well. And it has the property that "known" guys (like Harden) are going to have less uncertainty than unknown guys, although there the variance would be coming from age, not overall quality, it'd have similar gameplay effect of non-uniform fuzz.

@dumbmatter
Copy link
Member

After thinking more about it, I think it would ultimately be too confusing to have ratings change over time, but only due to fuzz and nothing else. People would think the actual underlying ratings are changing.

Do you think this PR would still be worthwhile without that part?

@nicidob
Copy link
Contributor Author

nicidob commented Jan 20, 2021

Yeah the idea to have fuzz vary during the season was actually your proposal in discord. I've always been more interested in non-uniform fuzz (or at least uniform in something like "on court impact", where I used +/- as a surrogate)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants