Evaluating the Amateur Scouting Department

Alright, quick note here. I've been away from writing most of the season, for which I apologize. I can offer many reasons, but honestly, I hate making excuses. I'm happy to be providing something for people to discuss again and I hope you enjoy it...

The impetus for this post came from a twitter conversation with our own Ryan Batty and The Oilers' Rig's Romulus (@RomulusNotNuma). Rom does great work over at the Rig and in this post he outlines a lot of what he wants to see over the rest of the year. We agree in many ways. When it comes to amateur scouting, he contends that it will take time to assess the value of each person's contributions before you can feel comfortable taking steps to weed out those who have contributed to the team's poor draft record in favour of some new blood and a new approach.

Rom's belief as I understand it (he'll correct me if I'm wrong I'm sure) is that influence from senior levels of management likely impacted many of the drafting decisions in recent years and it's important to figure out which guys were part of the problem, and which guys had the right idea, but lost the battle at the draft table. I think this is very true, but I think the successes have been so minimal for this organization outside of the first round that it's going to take some convincing for me to believe there were guys that have been with the franchise for years and knew better all along if only the team had listened to them.

Of course "Some" time is required in this evaluation, and I'm sure Rom will respond to clarify how much he feels is necessary, but if you've got a couple of analysts who know as much as your average hockey blogger, I can't imagine this takes more than a month or so to get some actionable data on. I'm pretty sure the "forensic audit" has already been going on that long.

It might not be pretty, but here's one guy's thoughts on how you go about conducting this assessment:

First things first, I'd be going back to the 2007 draft and ignoring everything that came before. So that gives a 7 year window of selections to evaluate, though obviously the jury is still out on many of the more recent selections. Still, the process can be evaluated for picks made as recently as last summer.

Second point. What is the criteria for keeping your job? It would be nice to be able set a clear quantifiable line in the sand for that, but HR decisions are rarely that black and white. Lots of shades of gray here. Still, here's what I'm looking for:

1) A history of identifying the some of the best prospects available within the draft range of the team's selections, with emphasis on the first 3 rounds.

2) A proven ability to make your voice heard among a group of people who all have their own unique opinions.

3) An ability to both execute on the team's objectives AND find potential hidden gems that might be good fits whether they align with the team's goals or not. Execute on your assignments, but don't just have tunnel vision.

4) Don't be a "Yes Man".

5) Have no history of bias or preference to players from certain geographic regions, hockey programs, countries or players to whom the scout or the team have historical connections. Basically...have a history of objectivity.

There's likely more than that, but that's a decent start and if someone can satisfy those five criteria, I'm okay with letting them see another day as a part of the organization.

So, how do we get our data?

a) Assess each individual scout's personal draft board for each year since 2007. Who has the best track record of identifying the players that succeed?

b) Assess each selection made.

1. Was it the best selection given the information available at the time?
2. What was the criteria used to select that player vs. others who were available? (Meet organizational need? Best long-term potential? etc.)
3. What attributes were the things that encouraged the team to select that player? Were they the right attributes to target?
4. If the right choice was made, which scouts got it right?
5. If not, which ones advocated for a different player?
6. Which scouts had the ear of the GM with regard to that pick?
7. Who made the ultimate decision? Was it the scouting staff or did the GM or another party intervene.
8. If the pick has not worked out, is this because of poor drafting (Cam Abney, Troy Hesketh) or a failure of the developmental staff (Anton Lander, Curtis Hamilton, etc.)

It should be noted that some of the players drafted are no longer with the organization. That should not be a factor. Somebody made a good choice in targeting Tobias Rieder a few years ago and they deserve credit for it whether he is succeeding in Arizona or Edmonton.

Some of this data will be readily available in terms of notes and even on video if you get the raw footage of "Oil Change".

For other information, you'll have to have someone interview each of the scouts, which is exactly what should happen. Give them an opportunity to explain their rationale, but be sure to reconcile their narrative with their personal draft boards. Are they saying what you want to hear given years of hindsight, or are they giving a reasonable account of what likely took place? Looking at their draft boards, are they continuously over-valuing anything? (Size, hockey program/junior team, etc.) Had the team made selections based on their recommendations and their draft lists, how would they be better/worse off?)

Lastly, I'd be asking for mid-season draft recommendations from each scout to see what attributes they are placing the most value on for this year's eligible prospects.

You're Fired!

Then comes the tough part...Honestly, if you have even the slightest doubt as to whether a scout can be a valuable contributor to a revamped scouting department, they have to go. During my conversation on twitter I jokingly suggested that everyone should be fired. I didn't mean to be flippant or to reduce the impact of someone losing their livelihood. It's a big deal to terminate people's employment. That said, the performance of this group has been so abysmal that I'd begin the process EXPECTING somewhere in the neighbourhood of a 75-80% turnover. Honestly, every single person would have to convince me NOT to fire them. That may be guilt by association, but every single member of the amateur staff has had a hand in these failures and unless they can demonstrate that they had a hand in many of the small successes or that they advocated a different course and were overruled, then I'm not sure how I can find a rationale for keeping them around.

I wrote this quickly and, given the time, could likely have doubled the length of this post to further flesh out my arguments, which I'm happy to do as part of an ongoing discussion around this topic, but hopefully this gives everyone some idea as to how I'd go about evaluating the Oilers' amateur scouts.

Honestly, This process can be done in a matter of weeks if you assign a few people to it. We're only talking about 61 selections here. I fully suspect that interviews with these scouts would quickly make the decisions pretty clear.  My personal belief is that the house needs to be cleaned by Valentine's Day so that if you need to bring in some additional support, you've got the time to do so and allow them the proper timeframe to help you evaluate this year's draft class.

It cannot be said enough. This group of employees SHOULD NOT BE PERMITTED ANOTHER OPPORTUNITY to mess with the future of this franchise.