How BRADY Works

What Is BRADY?

BRADY (Bayesian Regressed Adjusted Down Yield) is Held Out's in-house QB stat, named for the undisputed GOAT of the NFL (spoiler: yes, the numbers back that up, at least on career value). It's meant to predict how good a QB is at a given time.

The goal of any QB metric is to isolate the quarterback's play from the players around him. This is pretty difficult in the NFL, because (unlike sports like basketball) the players around a given QB don't fluctuate enough to directly regress on what happens when those players change, and football is noisy, with a lot of variance game to game.

BRADY stats solve for this using a number of statistical corrections and mini-models. The system is smart and adaptive; it understands how good different defenses are on different types of plays, how much to credit a quarterback for recent play vs. historical play, how experience factors into a QB's development, and a lot more. As far as I can tell, it's on par or better compared to every other QB metric out there (and significantly better than any other free one).

The BRADY Family

Held Out publishes 3 variants of BRADY: - BRADY (a.k.a. Career BRADY, Held Out's headline career grade and best estimate of a QB's current ability) - All information from a player's career - Cross-season adjustments for opposing defense and receiver credit - Season BRADY (how well a QB played, based on only a given season) - Only information from the season - Within-season adjustments for opposing defenses and receiver credit - Game BRADY (best estimate of how good a QB is, based on only a given game) - Only information from the game - No opponent or receiver adjustments

(Career) BRADY

A QB's BRADY rating is their career rating, an all-data metric that uses all information about the QB across their career to rate the QB.

I go into more detail on comparisons and this metric's approach below, but this is broadly the best predictor we could make of how good a QB currently is: opponent adjustments, per-skill ability estimators, a 'ladder' of history lengths, a prior based on how much the QB has played, and a number of related optimizations.

Season BRADY

The latter two are meant to map to stats that are more familiar to most people: QBR and Passer Rating, in particular.

Season BRADY is very similar to Career BRADY with only this season's information. Unlike Career BRADY, it isn't predictively optimized: every QB starts the season at the league average and every game counts the same, so it tells you how well a quarterback has played in games so far this season, rather than how well he's expected to play in the next game.

You can find Season BRADY using the "Season Only" option on various pages. It's on its own grade scale (the median qualifying season is a 75 and the best ever is a 99), so a Season BRADY of 85 isn't the same thing as a career 85. Both ratings use the same kinds of adjustments, but Season BRADY only reads this season's games for them.

Game BRADY

Game BRADY is a simple average over the valuation function and plays which we use to grade games for Season and Career BRADY. If you're familiar with EPA (expected points added), it's conceptually similar to EPA/play over plays the QB is involved in, rescaled to our human-friendly ~50-100 grade scale.

Game BRADY isn't perfect (no defense adjustments or particular predictive power), but it's a very good "grade" of how a player played in a particular game.

Why BRADY, Instead of ___?

Obviously, this is the key. Why another QB metric?

1. Cost & Thoroughness

BRADY is, in my humble opinion, the best predictive QB statistic that's fully free and broadly available. Held Out doesn't charge anything for any non-commercial usage (including resharing with credit), and this site publishes detailed historical ratings: per-week progressions and per-year QB rankings dating back to 1977. If you want to know how Steve Young's 1994 season went, this is the place for you.

Held Out also publishes a Season-Only variant of BRADY (Season-BRADY), so you can see just how a single season went, instead of our estimation of the QB's actual ability. It's also available back to 1977. I'm not aware of any other statistic available at this grain and quality for free.

2. Interpretability

BRADY is designed to be very easy to understand. The average starter per-year has an ability grade around 75, the best QBs have grades 87+, and benching candidates have grades below 69. If you've played Madden or taken a class with an A-F grading scale, the grades should feel familiar right away, and they also should make sense after thinking for a second. As far as analytics go, they pass the "eye test". You can take a look at "Approach" below for more information on how the grades are scaled.

3. Performance

Finally, the nitty gritty. The cost and simplicity are perhaps deceptive: BRADY is a frontier-level quarterback metric. I have a lot of respect for the creators of 'competitive' metrics, writing from whom directly contributed to my designs and experiments for this metric: QBR from ESPN, QBERT by Nate Silver, QBI by The Athletic's Austin Mock, and more.

For cases where historical data on these other metrics exists, I ran comparison benchmarks, and BRADY grades out well. Note that these aren't the only benchmarks I optimized BRADY on, and other benchmarks may have slight deviations depending on the data/inputs/output you use. I think two evaluation styles are most useful:

Next Game

My primary objective in designing BRADY was "next-game" predictive performance: given weeks 1 through N, how can we predict a QB's performance in week N+1? Specifically, I wanted to predict this in a way that tells us more than just an "offensive team grade" could.

To best measure this next-game incremental predictive performance, we can check how well different metrics perform when we incorporate them into RICE, Held Out's team rating system.

RICE has a QB adjustment. We can refit RICE with each metric as this adjustment; RICE blends QB grades with team-level components, so this does a good job at testing isolation (value on top of just knowing the team's offensive performance).

Here, I compare RICE's performance if you swap its QB component with a rolling version of ESPN's QBR (8-game EWMA) and Projected-QBERT (from Nate Silver's site, Silver Bulletin). When measured against BRADY, QBR is negative and QBERT is neutral to slightly negative:

BRADY holds up well! To be clear, QBERT is a well-optimized metric and I think the folks at Silver Bulletin are doing a great job with it. BRADY, from what I can tell, has some architectural differences, but both metrics are on the information frontier if you want to know the best QB today. QBERT may be better tuned in their team rating harness, ELWAY.

The RICE test measures marginal impact outside of team effects, but team effects do absorb some amount of QB play. We can also evaluate predictiveness using forward correlation with output metrics (how well we predict next-game PIV or QBR), which will be less isolated but also more completely reflective of QB play.

BRADY grades out well here too. Compared to career QBR or career QBERT, a QB's career BRADY grade in a given week is equally or more correlated with next-game QB performance. This test isn't paired, so I can't put a significance level on these gaps, but it also puts BRADY among the best. Interestingly, BRADY is more predictive of future QBR than QBR itself (despite not training on or optimizing for QBR in any way).

Next Year

The other set of benchmarks I use is prediction of the next year's results. The most important property of QB metrics is isolation: evaluating QB talent invariant of the surrounding team. One indirect way to check this is to see how strongly a QB's grade at the end of one season predicts their performance in the next; a strong estimator of QB ability should maximize this if it truly judges ability across contexts (with changing teammates and coaches).

The same RICE QB rating swap, with end-of-season ratings from the prior season:

Similar to the in-season ratings, BRADY does significantly better than QBR, and similar to slightly better than QBERT. We can also look at the QB performance metrics:

Again, we see the same trends here as we do with the in-season predictions. The forward validity of the approach, with all trends holding solid between all 6 benchmarks, makes me confident that BRADY is discriminatory on a quarterback's current talent, and is one of the best ways to estimate said talent. It's also pretty notable that QBERT and BRADY both are better predictors of future QBR than average QBR itself; this lends credence to the idea that a predictive-optimized metric does better at the goal of measuring talent broadly.

Season and Game BRADY

Notes

• All data is fitted using an expanding, out-of-sample approach for BRADY (i.e. model and grade for a given week is only based on information from before that week). I wasn't able to find information on the fitting for historical QBR and QBERT, but I'd assume the same of those metrics.
• Historical QBERT data is non-public and I sourced it through my subscription to Silver Bulletin. I'd strongly encourage a subscription to his site; perhaps, right after a donation to Held Out ;)
• I applied a non-linear transformation to QBR to make it comparable to QBERT and BRADY (it uses a logit, so we undo that transformation to get the raw player grade). This transformation helps it in the analysis, so I thought it fairest to apply it even if it mutates the original metric.
• I tested a few different (naive) ways to aggregate QBR into a "career" style grade for this analysis. 8-game exponentially weighted moving average generally tested best (on the same seasons as the comparison, which if anything flatters QBR), so that's what I used; season-to-date QBR, other history lengths, and other weighting methods were slightly worse.

BRADY's Modeling Approach

To get around the noise and isolation issues I described earlier, all NFL QB metrics usually take play-level data and figure out how much positive or negative credit to assign the quarterback for each play (as opposed to the opposing defense or receivers).

One common approach to this is averaging an offensive impact metric on some set of 'QB plays': passing attempts, sacks, QB rushes, and DPI flags are what I found to be the ideal set. Basically, you estimate "how many points of value has the offense added when the QB controls the play". This 'QB offensive impact/play' is a pretty good metric and captures most of the information about how good a given QB currently is when you 'average' it correctly.

BRADY is an extension of that approach. It's built on QB-play PIV (Play Incremental Value), Held Out's in-house play-impact metric that's similar to EPA; see our PIV page for more details on EPA, PIV, and our approach to play value.

Both raw PIV and raw EPA have major flaws, though. Roughly in order of solvability:

1. Interpretability

This is non-technical, but I'd argue it's the biggest one. Raw impact numbers don't mean anything to the average sports fan. While framing it in terms of points/play is interesting, it's not intuitively interpretable: you need multiple decimal points to distinguish the ratings, which makes them very hard to parse at a glance. The average offensive QB EPA/PIV has also drifted upward over time as the league gets more pass-friendly, making the mapping from per-play-impact to actual skill dicey as a historical comparison.

To make the stats easier to understand, BRADY uses the same scaling as our team-rating system, RICE. Each QB gets a numerical ability grade from ~50-100, on one fixed scale for every season, pinned so the median starter is about a 75 (PIV already measures each play against that season's league scoring level, which keeps eras comparable). Our grades work similarly to Madden ratings and classroom grades:

• ~93 and up: 'Surefire MVP' quarterbacks. Most players with an end-of-season BRADY ability grade of 93 or higher either won MVP that year or lost MVP to another 93+ player (23 of 39); the rest are seasons like Marino 1986, Brady 2012, Brees 2018, and Mahomes 2019 and 2021.
• ~87 to 93: All-Pro caliber quarterbacks. Players in this range are either at an All-Pro caliber or just off it; they significantly increase their team's odds of winning, and are usually at least top-3 in the league. Of 99 end-of-season BRADY ability ratings in this range, there were 11 MVPs and 30 All-Pros.
• ~81 to 87: Good quarterbacks in their primes and HOFers having good years. Most of these players are still considered elite, around top-5, with some All-Pros and MVPs scattered among years graded out here.
• ~77 to 81: Ratings compress here as they move toward average. Most teams would be happy with a guy graded 77 and up; they're usually ~top-10 and efficient, but uneven or flawed in some way as a passer.
• ~73 to 77: Guys who are above average for a starter that season, ~top 15, still mostly guys who get big contracts and are desirable for QB-needy teams.
• ~69 to 73: Below average but very serviceable QBs, typically either declining older players, developing younger ones, or journeymen veterans. Usually clearly better than the best backups in the league.
• Below ~69: bad players that won't keep starting for long if they continue playing this way.

This grade scale makes sense even if you don't read about it! 69 and below is typically where teams will search for a new QB, much like a D is typically considered a failing grade. It's a little distorted at the top end, since the very best QBs of all time are MUCH better than average, but "Rodgers had a 95.9 BRADY in 2011" is simpler than alternatives like "Rodgers had a WAR/start of 0.339".

2. Not Tuned for Predictiveness

Play value is pretty noisy. Interceptions and touchdowns are obviously extremely valuable plays, but they're also subject to a lot of random luck. Nate Silver, working on his excellent QB metric QBERT, found that to measure a QB's true interception-avoidance skill, you need multiple years of data. If a QB improves/regresses over that span, you'd need even more. Even more fluky are QB fumbles: my analyses found they carry basically no predictive value and are mostly fluky, but are some of the most negative-value plays.

If you just took the average quarterback PIV/play, you'd find that the ratings are incredibly noisy for several years before evening out. Even then, average PIV/play will _weight_ some types of plays too highly and be biased in a few ways.

BRADY is a predictive statistic, meant to explain how good QBs are today in terms of their next-game contribution to winning. It learns to correct for the noise on a per-play-type basis using a rolling approach. Rather than directly updating based on play values, it corrects play values by estimating how predictive that result is of future performances (after adjusting for noise from the play type, defensive ability against that play type, etc.).

3. History Weighting

If you want to tell how good a quarterback is today, how many games into the past should you consider?

It's actually not too easy a question, and the answer is more complicated than you might think. BRADY uses a dynamic weighting curve that knows how to weight older games lower over time (this is known as "decaying" those observations), without ever discarding them entirely.

To add another layer, BRADY ratings are weighted over several decayed "memory lengths", from four games to thirty-two. Shorter memories react to last month. A longer memory is closer to a career.

Quarterbacks who are young, in their prime, and older all change at different rates, so rather than betting on one memory, the rating averages all four (averaging over the spread tested better than any single tuned length). How hard each memory is pulled toward the prior depends on how many plays the quarterback has and how fast his career is moving, so a player with a thin record is pulled toward similar players, but players getting better quickly aren't penalized.

4. Resilience to 'Bad Games'

Means can be very noisy. My early BRADY experiments dropped ratings quite a lot when a quarterback had a really bad game, as you might expect; popular perception changes a TON when a QB has a few bad games. Drake Maye early this year is a good example, with his ugly performances early in the year spawning a ton of 'meatball sub' memes.

But bad games aren't necessarily predictive. Underneath those rough games were stronger fundamentals: he was a young quarterback with a mediocre supporting cast, great past performances, and faced off against a great defense (Seattle, 3 INT) and a good one (PIT, 1 INT) in that span. BRADY was basically neutral on Maye after his 1 TD, 6 INT span, modifying his rating from 81.2 to 81.0.

Though that might've seemed absurd at the time, it's proven prescient. With great performances in weeks 4, 5, 6, and 7, Maye vaulted himself into the MVP conversation. BRADY doesn't overreact; it moves just the right amount to be a good predictor of the next game, because it's learned the correct "update rate".

5. QB Isolation

This is the toughest aspect to handle by far, and nobody has fully solved it. BRADY does everything it can to correct for context out of the QB's control, splitting credit for receptions between receivers and the quarterback, maintaining per-play-type estimators of every defense's ability, and a host of smaller tweaks and adjustments. Ultimately, though, it's nearly impossible to "control" away certain parts of a quarterback's play. The toughest one, in my modeling, was coaching/scheme; it's pretty hard to disentangle the effect of good/bad coaching on quarterback play, and the attempts I made to do so didn't end up actually improving the model.

I also made an explicit effort to maintain a related property to isolation: cross-year consistency. A QB's rating this season shouldn't just predict their performance in the next game; it should also predict their performance next year (even if they're on a different team). I'm pretty proud of BRADY's design for this; it's very good at holding validity cross-season (see the Why BRADY, Instead of ___? section above).