Posted by:
Prove It

For in depth details of the items below, also see:

Benchmark Conference Comparison Part 1 - SEC

Benchmark Conference Comparison Part 2 - Big East

Benchmark Conference Comparison Part 3 - PAC 10

Benchmark Conference Comparison Part 4 - ACC

Benchmark Conference Comparison Part 5 - Big 12

Benchmark Conference Comparison Part 6 - Big 10

The idea for the series started with talk of which of the major 6 conference was best, generally supported with garbage arguments irrelevant to the topic at hand.

So I devised the benchmark system to give an apples to apples comparison.

1. Use the standard tie breakers to settle how the teams ranked (finished) inside their conferences.  The results are listed in Part 1

2. Equate the rankings between conferences of different sizes.  For example, 3rd and 4th in the Big 10 was equated to 2nd in a conference with divisions.  This was also detailed in Part 1 (I had no problem in high school or college with the guy next to me copying off my test)

3. List out the results of the inter-conference games including the respective ranks of the teams in each.  These were scattered across the bottom of parts 1 thru 6 because of size.

4. Establish a comparison (benchmark) record based on how the teams were matched.  The benchmark record is the record if the conference won every game where they were the higher ranked team, lost every game where they were the lower, and went .500 when they were evenly matched.

5. While the percentage in each category is a better indication of how the conferences played (even, higher, or the lower team), a benchmark score was used to draw attention to large deviations from the average.  The benchmark score is the actual number of wins minus the benchmark (expected) wins.  The higher the score, the better.

6. When there was a substantial deviation from the Benchmark score a detailed analysis following procedures similar to those used to determine to root cause of failures in process controls was performed.   The goal of the analysis was to determine the most likely reason for the deviation when viewed against other available data (performance in other years, against other teams, against other conferences, etc.)

7. Finally, summarize the results and verify them against the record.  The conference by conference analysis and summary caqn be found in other blogs linked below.  This blog is dedicated to the summary and accuracy verification.

Special Thanks to the creators of Excel (I sure as heck won't type and calculate all of these by hand)

The Benchmark for all Conferences in Percentage

From 2002 to 2007

Even - 50%

Higher - 68.9%

Lower - 31.1%

Higher + Lower % - 100%

The higher and lower percentage is the average for all 6 conferences

Not Your Big Brother's Big East

I expected the Big East to be the most difficult to analyze because of the low sampling.  3 teams have only been in the Major 6 conferences for 3 years, a 4th only 4 years.  Results prior to 2005 are irrelevant.

I was expecting to find a conference showing improvement since 2004.  What I discovered was even more surprising.

Even     4-4                   50%

Higher  14-4                 77.8% (avg 68.9%)

Lower  4-9                   30.8% (avg 31.1%)

Higher + Lower            108.6% (avg 100%)

The results may be exaggerated by the few games available and because there were very few higher and lower games where the teams were close, but they do show the Big East as a conference is already back.

The analysis showed the conference members were already in the process of upgrading their facilities and improving their coaches before the conference shake up leading to a substantial improvement in performance and recruiting.

PAC 10 - forget 2002

The PAC 10 had a miserable 2002 year for reasons I didn't care enough to investigate (it was 2002 for crying out loud).  For 2003 to 2007:

Even     6-1                   85.7% (avg 50%)

Higher  27-10               73% (avg. 68.9%)

Lower  7-18                 28% (avg 31.1%)

Higher+Lower  101% (avg 100%)

The record when even is almost all USC (5-1 when even) who keeps finding ways to miss the title game with losses to teams like Stanford (it could be worse - you could be on a down streak against your arch rival).

The analysis did show the PAC 10 was playing below the average against the Big 12.  No reason could be found other than the Big 12 is just outplaying them in heads up competition.  I explore this in more detail below.

I did not fully investigate the Stanford factor.  They typically average near the bottom of the conference but have played the other major 6 conferences only once in 6 years.  This could lead to the lower wins percentage being artificially high.

The ACC's 2 Records

While the overall record the last 6 years is average, the ACC is a conference with 2 distinct records.  From 2002 to 2005 they were indeed the best conference in the nation:

Even     11-9                 55.0% (avg 50%)

Higher  23-5                 82.1%  (avg 68.9%)

Lower  16-25               39.0% (avg 31.1%)

Higher+Lower              131.1% (avg 100%)

The last 2 years they have been the worse:

Even     2-6                   25.0% (avg 50%)

Higher  9-8                   52.9% (avg 68.9%)

Lower  6-15                 28.6% (avg 31.1%)

Higher+Lower              81.5% (avg 100%)

Cumulative, this has left the ACC about average with the other major 6 conferences:

Even     13-15               46.4% (avg 50%)

Higher  32-13               71.1% (avg 68.9%)

Lower  22-40               35.5% (avg 31.1%)

Higher+Lower              106.6% (avg 100%)

The reason was traced to the downfall of Miami and FSU.  With dismal performances the last 2 years by the ACC division champions no other teams was ready to fill the void.

This is devastating in this analysis.  When the top teams falter without a replacement available to fill the gap, they get hit hard in the conference rankings.  Even games are transformed into higher matches, lower games become even matches, and the benchmark percentages tumble.  The reality is likely they are the same conference with the same teams performing at the same level minus the top 2 teams.

The Big 12 Up and Down

The Big 12 revealed the downfall of looking at the benchmark score (actual wins minus benchmark wins) instead of the performance in terms of percentage in each category.  The benchmark wins can be skewed if a conference plays a disproportional number of games as the higher or lower team.  While the Big 12 has a benchmark score of +9, their percentages are more even:

Even     9-5                   64.3% (avg 50%)

Higher  24-13               64.9% (avg 68.9%)

Lower  20-37               35.1% (avg 31.1%)

Higher + Lower            100%   (avg 100%)

The analysis showed 2 major anomalies in the Big 12's inter-conference record.

The 1st was the Big 12 vs. the PAC 10.  This encompassed most of the teams and most years.  Even if you ignore the PAC 10s 2002 record, from 2002 to 2007 the Big 12 has an impressive record against the PAC 10:

Even     1-2                   33.3% (avg 50%)

Higher  5-1                   83.3% (avg 68.9%)

Lower  5-9                   35.7% (avg 31.1%)

Higher + Lower            129% (avg 100%)

The 2nd was the Big 12 vs. the SEC West.  As impressive the Big 12 has been against the PAC 10, they have been equally unimpressive against this division.  Again, this encompassed most teams and most years.  Surprisingly, this did not extend to the SEC East (in no small part because of the low sampling):

Even     no games

Higher  8-6                   57.1% (avg 68.9%)

Lower  2-5                   28.6% (avg 31.1%)

Higher + Lower            85.7% (avg 100%)

Unable to find any other pattern that fit with the performance of the respective conferences against the other conferences, the conclusion was the Big 12 is outplaying the PAC 10 by about as much as the SEC West is outplaying the Big 12.

I did not fully investigate the Baylor factor.  They typically finish last in the conference but have played the other major 6 conferences only twice in 6 years.  This could lead to the lower wins percentage being artificially high.

SEC - Are We Making too Much of 2 Games?

In 2 words, YES and NO.  Looking at the results, the nation is wearing blinders with an extremely narrow field of vision.

The SEC West against the Big 12 is noted above.

The SEC's inter-conference rivalries against ACC teams that finished above Miami and FSU placed them in position to capitalize on the downfall of the ACC the past 2 years.  Prior to that the conferences were playing dead even.  In the past 2 years the SEC has not lost to the ACC when the teams have been evenly ranked or the SEC team was higher, and the SEC is 50% when they finished lower in their conference than their opponent finished in the ACC.

Against the rest of the conferences the SEC has played average.  If you negate the last 2 years against the ACC, the record is:

Even     7-7                   50.0% (avg 50%)

Higher  28-12               70.0% (avg 68.9%)

Lower  13-35               27.1% (avg 31.1%)

Higher + Lower            97.1% (avg 100%)

As well as the SEC West has played against the Big 12, and the SEC has played against the ACC, they have not fared well against the Big 10 (despite public opinion).  Over the last 6 years:

Even     5-3                   62.5% (avg 50%)

Higher  5-5                   50.0% (avg 68.9%)

Lower  1-3                   25.0% (avg 31.1%)

Higher + Lower            75.0% (avg 100%)

The cumulative record (placed here just to show how far out of whack your opinions is with the results of the field of play):

2007    2-1

06-07   3-4

05-07   4-6

04-07   6-8

03-07   9-9 (finally, .500 again)

02-07   11-11

Let me summarize in a way which likely explains why you opinion is so different from the results of the field of play:

Against OSU the SEC has looked awesome.   Against the rest of the Big 10, the SEC is 50% when the teams are evenly matched, 50% when the SEC team finished higher than the Big 10 team, and you have to go all the way back to 2002 to find when a team that finished lower in the SEC defeated a team that finished higher in the Big 10.

(Don't start crying now - if you liked the results of the Big 12 and ACC analysis, live with the results of this analysis or provide a better answer - all my research is listed in this series of blogs, so most of the hard work is done.)

Big 10 Hoping to Forget 2007

2007 was a lousy year for the Big 10.  Problems plagued every team.

Over the past 6 years the Big 10 has done well, negating any need to overlook a down year (as was necessary with the PAC 10):

Even     7-11                 38.9% (avg 58.9%)

Higher  38-16               70.4% (avg 68.9%)

Lower  15-27               35.7% (avg 31.1%)

Higher + Lower            106.1% (avg 100%)

If you negate the 2007 season, the numbers are Even-43.8%, Higher-70.5%, Lower-39.5% (admit it - 40% when lower is impressive), Higher+Lower 114.3%

The Big 10 performances against the other conferences individually are detailed above.

Other Findings

Other anomalies were traced to:

-A large imbalance between the number of higher and even games (which is why I went to percentages in this summary).

-An unusually large difference in the rankings between the conferences in higher or lower matches (for example, in the Big 10 vs. PAC 10, the PAC 10 was found to be on average over 3.3 conference standings higher in their higher wins, negating much chance the Big 10 could accumulate a substantial lower winning percentage.

-A small sampling (for example, the PAC 10 played the ACC only 6 times in 6 years).

-A lower place team(s) is played the other major 6 conferences only once or twice in 6 years, distorting the lower winning percentage.

Otherwise the conferences were found to be playing about even.

Which Conference is Best?

Even - with the outcome of 2 or fewer games reversed, every conference would be 50% in even games.  Even if you ignore this, conferences with an advantage in this category have lower winning percentages when higher, lower, or both.  If you think about it this makes sense -the majority of the games played were between conferences of different sizes.  This places teams that were slightly higher or lower as even in comparison.  Use this to show dominance and I can show there are better answers (not to mention this encompasses substantially less than 1/3 of the games).

Higher, Lower, and Higher+Lower - Teams that finished well when higher typically finished a little worse when lower.  Negate the 2007 ACC from the SEC record (I will still give them 2006), look at the Big East just the last 3 years, and negate the 2002 season for the PAC 10 and the following is the Higher+Lower winning percentages normalized to 100% as the average:

Big 10              101.8%

Big East            104.2%

ACC                102.3%

PAC 10           94.5%

Big 12              96%

SEC                 98.8%

+/-4.2% might as well be called even.  The measurable differences are limited to the following:

1. ACC has been lousy the last 2 years to the benefit of the SEC

2. PAC 10 was lousy in 2002 (who cares?)

3. Big 12 has looked awesome against the PAC 10

4. SEC West has looked awesome against the Big 12

5. Big 10 as a conference has looked awesome against the SEC

6. PAC 10 has been difficult for the Big 10 to beat when they are the higher team, which while we traced to a difference in conference rankings, it leads to the ultimate conclusion:

When you look at the performance of the conference as a whole, the SEC has outplayed the Big 12 who has outplayed the PAC 10 who has outplayed the Big 10 who has outplayed the SEC - the major 6 conferences equivalent to a round robin tie.  Meanwhile the Big East been playing everyone about even since their shake-up, and the ACC after dominating for 4 years has hit the bottom the last 2.

When you compare the games in an apples to apples fashion, that's how it shakes out.


Honestly, this wasn't the result I was expecting.  I didn't compare 670 games and rank 66 teams in their conference finish over 6 years, look at the results, them change the basis of comparison to make everyone look even (if I was going to do that, I would have gone the homer route to put my home conference on top).

I really wanted to end the series saying "Behold, I, Prove It, have established the definitive ranking of the conferences and the basis for comparing from this day forth" but that wasn't to be the case - instead I am stuck saying "Eh, as close as any analysis can show, they've played about even as a conference."

A quick look at the results over just 4 or 5 years rather than 6 showed the conferences even closer in their results (remember - I dropped the 2002 season for the PAC 10 and only looked at the last 3 years for the Big East).

For any stat or anomaly you can find to show a conference as a whole is better or worse over a decent time span and with an acceptable sampling, I can find and prove your analysis is missing the real reason for the anomaly (or in most cases is wholly irrelevant to the question at hand).


Discrepency with Common Opinion

While not as notable here as in Part 1 - SEC, the results of this apples to apples comparison frequently differs from common opinion.  Below are the primary reason.

1. They are Sports Writers and Sportscasters, NOT Sports Journalists Their job isn't to research and investigate.  If they are in a newspaper, their job is to write columns to attract readers to the newspaper.  If they are on a radio TV commentary show, their job is to build an audience, not perform in depth analysis.  If they are part of a network that broadcasts games, their job is to promote the teams their network shows.

They aren't likely to tell you that since the membership was shaken up, the Big East is about .500 against the other major conferences and even against the benchmark - they are going to tell you the Big East is an inferior conference to yours because that is what you want to hear.

ESPN isn't going to tell you that over the last 6 years, 7 of the 12 teams in the SEC have a loosing record against the other major conferences any more than they are going to tell you Mississippi, Mississippi State, and Vanderbilt are a combined 4-14 against the other conferences.  They have spent a lot of money buying the rights to televise the games and are going to tell you the SEC is a tough conference, top to bottom.

Even if they don't have a particular interest, they aren't going to do an in depth study (ie-work), instead trying to land a high profile interview.  Think about it next time you read a newspaper sports article - does it really read like something someone spent an 8 hour day drafting?

2. Lack of Knowledge There are 65 teams in the major 6 conferences spread out across the country plus all the lesser conferences.  In the vacuum of reliable information from the media, there is no way you will get the details without some investigation on your own.

You may know the Big 10 went 2-6 in the bowls in 2007, but you won't know these included lop sided match ups such as the 10th place team against the MAC West Champion, the 9th place team (helped along by the easiest schedule in the Big 10) against a team which tied for 3rd in their division in the Big 12, or the 8th place team against the 2nd place team from the ACC.

3. Lack of Games The teams from the major 6 conferences average about 1.6 games per year against the other major conferences (and many of those are bowl games).  They don't play enough games to make a good comparison until the bowls, after which your interests are onto other things.

4. History Bias You've spent years forming an opinion of the different teams and conferences.  When the tide shifts you will likely be behind the times.  How many people were aware the Big East had done so well the last 3 years?

5. Conference Bias You will naturally have a higher opinion of the teams from your conference and the conferences you frequently play than you will of teams and conferences you rarely see.  We are all inherently homers.

6.  Your Judgement is Based on a Few Games Even if you know the SEC has spent most of the last 6 years with a .500 or less record against the Big 10, you probably didn't know that most of that record was compiled by Big 10 teams against teams that finished higher in the SEC.  Instead, your judgement is based on a couple of National Championship games.


Calculations and Games

Details of the calculation of conference ranking are at the bottom of Part 1, the individual games, rankings, and results for the Big 10 can be found listed at the bottom of parts 2-6.

If you find a mistake, please let me know.

If you have a better reason than the analysis, put it forth - if it is a better explanation, I will herald your name and place it in these blogs.


Friendly Reminder

This is a Prove It blog, not a bastion of free speech.  As a courtesy to my 6 readers, leave an idiot comment and I will zap it or trash you with facts as the mood hits - either way, at best you're wasting your time, at worse you are stepping on an uneven field.


Thanks for reading.  Really.  I have been swamped at work with this as about the only fun I can have in the 45 minutes between dinner and bed.  As I slow down on the blogging for a few weeks to catch up on work around the house, etc. best wishes for your favorite team in 2008.



Remember to keep your posts clean. Profanity will get filtered, and offensive comments will be removed.

Start Your Own Blog

Start Now

Truth & Rumors


  1. 1
    'Melo wants to be wooed (like, Howard wooed)
  2. 2
    Farrell defends Fenway's 'sleep room'
  3. 3
    Youthful Red Wings are rattled
  4. 4
    Bishop's severe injury
  5. 5
    Texans may take Clowney ... for someone else

SI Photos