As an Ultimate player, I’ve been involved in tryouts and assessments for years, both as a tryout and as a leader. The Huddle had a great issue on the topic two years ago, with many knowledgeable players weighing in on tryouts from the candidate’s perspective.

However, I felt that the authors did not devote enough attention to the work that goes into building a good tryout system. As a graduate student, my passion is decision research; in fact my master’s thesis is on how people can improve decision accuracy. Therefore, I hope to offer some unique insight to the captains and leaders that are responsible for choosing who gets cut and who makes their Ultimate team.

I can think of at least three reasons why it is important to use a good selection system. First, a bad tryout experience discourages players from coming back. Talented players that get cut will only try out next year if they believe that the tryouts were a fair judge of their ability. Psychologists like to talk about ‘procedural justice,’ but all this means is that your selection system is so transparent and fair that no reasonable person can complain about it. By contrast, if they think the system is flawed or rigged, they will not try out in the future, and may even form a rival team that saps your area’s talent pool.

Second, a system that is well-built will gain goodwill and support from your local Ultimate community. An unfair and arbitrary tryout process will alienate you from people that love and care about Ultimate. If you want your program to be a fixture in your Ultimate community, build something that people want to invest in! This is equally true for college and club teams, national contenders, and first-year programs. Leagues and schools will rally behind programs that act professionally, and dissociate themselves from teams that choose their teammates haphazardly or secretively.

Finally, you are reading this because you want to make the right choices and avoid the wrong ones. If I’ve learned one thing in graduate school, it’s this: people are better at assessing talent in some situations than others. For example, experts do a pretty good job of evaluating performance on different tasks, but are lousy at combining all of these evaluations to make a final decision. Therefore, it makes sense to build a system that caters to our strengths and minimizes our weaknesses.

Trust me; a structured, rigorous tryout process will lead to more correct personnel decisions and fewer mistakes. It’s costly to invest time and resources in a player that never ends up contributing to your team. Likewise it’s just as expensive to miss out on a player that could have the game-changing talent you need.

The next few paragraphs will explain how build your tryout process systematically, from the ground up. Many of these steps may be considered common sense. However, if you take the time to walk through these steps with the other captains/leaders, you will build a better tryout process. I’m not building it for you (although if you are interested in that, shoot me an email and we’ll talk), but I’m giving you a framework so that you can build it yourself.

The first step is to determine what individual attributes will help your team win Ultimate games. Based on my own experience (and some informal research), most of these attributes would fit into one or more of the categories listed here. However, this taxonomy is far from complete, so you may come up with a number of attributes that are important to your team but not on this list:

1. Physical abilities (speed, quickness, vertical leap, reaction time, coordination).
2. Fundamentals/Mechanics (throwing, receiving, layouts, defensive positioning, footwork, reading discs, boxing out, marking).
3. Knowledge (working knowledge of different offensive and defensive strategies).
4. Decision making/field sense (with the disc, without the disc, on defense).
5. Personality/individual differences (work ethic, attendance, aggressiveness, commitment, leadership, charisma, communication skills, person-team fit).
6. ‘Teachability’ (learning aptitude, attentiveness, etc.).
7. History/Experience (team membership, leadership positions, awards, sports background, etc.).
8. Demographics (age, height, weight, geographic location).

Second, you must decide (as a committee) which knowledge, skills and abilities (KSAs) you are willing to train, and which ones you want to select for. Elite club teams can demand that their tryouts are exemplary across the board. College programs may be more willing to consider gifted ‘raw’ recruits, and lower level programs may have to plan on training nearly everything.

You want to select for the attributes that are most important, are hardest to train, and fill an immediate need. If you’re going to spend a lot of time teaching junk defense later this season, it may not be all that important to assess whether tryouts are familiar with the 1-3-3, clam, and 2-3-2. It is much more important to measure and select based on relatively stable traits and abilities. Because we are humans (with limited mental resources), we have a really hard time thinking about more than 3-5 attributes at a time.

Ok, now that you’ve decided what attributes are important for Ultimate, and which ones you are specifically interested in assessing, it’s time to settle on a timeframe for tryouts. Personally, I love the multiple hurdle approach that many teams have in place. Tryouts are assessed over multiple sessions, with the easiest cuts made after the first weekend. A certain number of tryouts are invited back for extra practices/sessions, followed by an X/Y tryout tournament.

Your first round tryout is a great opportunity to collect individual level data (information that is not really affected by other people). For example, you can collect biographical information such as where tryouts have played in the past, and their role on those teams. You can also start to form an opinion about personality characteristics. Is the tryout giving 100% in the drills? Were they the first one at the field? Do they seem to mesh well with the returners? Can they take critique and direction?

You can evaluate throwing range, accuracy, and/or consistency for each person. This can be done with an ‘objective’ rating (tape measure approach) or through ‘expert’ ratings (a guy with a clipboard rating each player from 1 to 10 on each dimension). For building block skills like this, I think expert ratings are usually the best way to go IF you define what performance deserves each rating. If you don’t ‘anchor’ the rating scale, Jon may rate tryout X as a 4 while Eric may give him a 2.

This is also the best time to measure physical fitness such as speed, quickness, and vertical leaping ability. When measuring these sorts of things, it’s important to remember that there is always measurement error- it’s not unusual for stopwatch times to vary by 2 or 3 tenths of a second. We can reduce the impact of this error with (1) repeated measures, or (2) longer fitness tests. I no longer advocate the shuttle drill at Ultimate combines because the drill only lasts about 5 seconds, which leaves a lot of variation due to error. Times on an 8×25 yard sprint or 4×40 are much more likely to yield meaningful differences. If you do want to keep the shuttle drill, I recommend using multiple timers and/or 4 or 5 trials per person, and averaging all of the times together.

Other dimensions require another person. 1 v 1 cutting drills, jump ball discs, and mini games all may give you a chance to evaluate important components. It is hard to evaluate complex constructs such as decision making skills outside of a game situation, so I recommend including some structured scrimmages as well.

As I’ve mentioned before, it is hard for raters to evaluate a hundred things at once. Therefore, for each drill that you are going to incorporate into your tryout weekend, list any important Ultimate knowledge, skills or abilities that it demonstrates. Then, give players a score for each drill rather than trying to split it into ratings of each separate component.

For this first round of tryouts, I advocate using a ‘cut score’ for all important dimensions. For example, at Clemson, I require all tryouts to be able to throw a flat 40 yard flick and backhand without travelling in order to be considered past the first round of tryouts. Why 40 yards? In my experience, players that can’t throw 40 yards are liabilities within our offense. The team has to change the way we cut if the player can’t throw a swing or complete a half-field leading pass. We tell our rookies this well in advance of tryouts, and I have never received a complaint from someone that was cut because of this.

If you decide that a tryout needs to complete an 8×25 in 40 seconds or less to have any chance at performing at your level, you stick with it. If they run slower than that, they are removed from consideration regardless of their other skills. If you don’t have a specific time, you could make another decision rule (e.g. “We will only consider players that perform in the top 50% at tryouts on this dimension,” or “we will only consider players that run as fast or faster than the slowest veteran.”).

After this first round of tryouts, it becomes increasingly important to determine how a player would fit within your team. Individual performance metrics become somewhat less important, and it becomes critical that you evaluate how each tryout performs within the team. You’ve narrowed yourself down to only the guys that you are seriously considering, and they all have passed the minimum threshold for the ‘building block’ skills. At this point, you will need to begin assessing much more complex events. My recommendation is that you have returners/veterans make performance assessments of each player during ‘job previews’ such as scrimmages or tryout tournaments. As always, it is important that you anchor the ratings so that people know what values to give. For example:

To recap, research has shown that people are actually pretty good at evaluating others if they are given a rating scale that has behavioral anchors for each rating level. We are also pretty good at evaluating how important each of these skills is to be successful at Ultimate. Unfortunately, people are lousy at integrating all of these evaluations into a global judgment. We tend to experience information overload and make decisions based on our first experience or our most recent experience. Even ‘expert’ hiring managers do worse than a mechanical model at combining scores for these individual dimensions. However, we can work around this problem by splitting our decision into pieces.

After we’ve decided what we want to assess, we decide how much each ‘predictor’ (the drill, scrimmage, or activity you are rating) should be worth. This decision will vary from team to team, but the specific drills (and their importance) should sum to 100%. If you can’t decide how important each one is, it’s perfectly appropriate to weight them all equally. Second, we rate the tryout on each predictor using one of our behaviorally anchored scales (usually a 5 or 10 point scale). If we can get more than one individual to rate the tryout on each drill, we simply average their ratings together (e.g. 4 + 5 /2 = 4.5). Finally, we multiply each rating by the importance of that dimension, and sum all of the scores up. This will give you a final score for each individual.

At this point, it is up to you to decide exactly how you will make your final decision. One approach you can take is top-down selection, simply starting with the highest score and continuing down until you’ve filled your empty roster space. Another approach you can take is to use these scores but subsequently select based on position (selecting the top two defensive handlers, the highest-rated deep receiver, etc.). Alternatively, you could institute a banding approach (e.g. if three spots are remaining, select amongst the top five rated candidates). This has the advantage of incorporating a structured tryout process while still allowing for some decision latitude within the selection committee. Each of these final decision strategies have some pros and cons, but they are all grounded in the structured tryout process you have implemented.

I know this whole structured tryout process seems like a lot of work. This is overkill for somebody that wants to get some buddies together to drink beer while attending a pickup tournament. However, a lot of captains are willing to put hours in at the track and the gym preparing for the season. They are willing to plan out their season from June to the Series, and draw up strategies and practices that can give them an edge come tournament time. They’ll give up weekends to travel and compete and win. Personally, I think those captains would be crazy to do all that if they aren’t willing to bring that same kind of effort to preseason selections.

Structured tryouts will help you identify talent that you would otherwise miss. They improve both procedural justice and decision accuracy; a rare win-win situation. They are also self-sustaining if done correctly: An excellent tryout process will lead to a stronger tryout class the following season.

As I’ve mentioned before, selection systems are one of my major passions. I love talking about this sort of thing, and I love doing it. If you have questions or ideas, I’d like to talk about them. If you want to implement a structured tryout but are feeling overwhelmed, I can help you get started. Feel free to email me at bslade86@yahoo.com.