Tag Archives: capability

How to Evaluate Capability in a Candidate

1 Reply

From the Ask Tom mailbag –

Question:

How can I test to see if a person has Stratum II or Stratum III capability?

Response:

If you are looking for a paper and pencil test, there is none.  There is no test with a set of answers that you shove into a computer that divines a person’s capability.  Elliott chuckled when this question was posed.  Most psychometric instruments, he observed, have, at best, a .66 correlation with reality.  Most are based on personality, or behavior, or behavior connected to temperament.  While those tests, or profiles have statistical significance for repeatability and in most cases, a stunningly accurate description of a person’s tendencies or behaviors, their evidence of predictability, a specific profile for a specific role has significance barely above flipping a coin (.5 correlation).

Elliott conjectured, if there were a paper and pencil test for capability, its likelihood to stand the same test would likely yield no more than the same .66 correlation with reality.

But your question is still valid and there is a method to satisfy the high curiosity we have about a person’s capability related to the level of work.  There is no trick, no special technique, no psychological requirement that we climb inside the head of our candidate and play amateur psychologist.

Moreover, the validity of this method reveals between .89 and .97 inter-rater reliability.

It’s all about the work.  Focus on the work.  As you define the role, its task and activities, goals and objectives, what is the level of work?  Does the role contain Stratum II level of work or Stratum III level of work?  Examine the decisions that have to be made and the problems that have to be solved.  Examine the time-span of the goals and objectives in the role.  What is the longest time-span task in the role?

The biggest mistake most companies make is underestimating the level of work required in the role.  A defect in the definition of the level of work in the role will most assuredly result in hiring the wrong person.

Examine your role description.  What are the tasks and activities?  What are the decisions that have to be made?  What are the problems that have to be solved?  What is the time-span of the longest task assignment in the role?

Based on that definition of the role, does the candidate provide evidence of effective task completion?  It’s all about the work.

When we spend the time to accurately define the work, and accurately calibrate the level of work in the role, the questions become very simple.  Does this person work as effectively as someone in the top half of the role or the bottom half of the role?  And, in that half, does this person operate as effectively as someone in the top, middle or bottom.

When you ask the team member to do a self-assessment, ask the manager and ask the manager-once-removed (MOR) about effectiveness, the inter-rater agreement approaches .97 (.89-.97).  With this practical evaluation system, why would you want to resort to other methods that might only have a .66 correlation with reality?

It’s all about the work.

How to See Evidence of Potential in an Interview

Leave a reply

“If you are not going to let me hope,” Monica protested, “then explain to me how I got this job? When I was promoted to manager, I had never been a manager before. If the interview had only centered around my prior role as a supervisor, then how did the interviewer make the judgment that I had the potential to be a manager?”

“Do you think the interviewer only had hope for you in this manager role?” I asked. “Monica, I watched you, in your role as a supervisor for three years. I sat in on the debriefing after you were interviewed for your current role as a manager. Do you think that decision was made based on hope?”

“Not if you were in the room,” Monica admitted. “But, then how did you know I had the potential to be a manager if I had never been a manager before?”

“Okay, let’s step through some questions. As a supervisor, do you think you were operating as effectively as someone in the top half of a supervisor’s role or the bottom half?”

Monica smiled politely, nodding, “Top.”

“And in the top half, were you operating as effectively as someone in the top third, middle third or bottom third?”

Monica continued to shake her head. “Top,” she repeated.

“What is the evidence for that?” I pressed.

“You always want evidence,” Monica replied. “My projects always came in on time, within the specs from the customer and always within budget.”

“And why did your projects always come in on time? Did you always get the easy projects or were there problems?”

“There are always problems, but you know, 90 percent of the obstacles are predictable. For example, permits are always a problem. And permits are outside my control, it’s a government agency that processes the permits. But I took the time to get to know the inspectors down at the building department. I know it is not part of my job description and sometimes they are not the easiest people to get acquainted with, but I also know it’s important.”

“So, you took the time to go beyond prescribed duties in your role as a supervisor. You anticipated obstacles that might get in the way and created alternate paths, to solve problems that might occur,” I recounted.

“Well, you know, if you don’t have a relationship with the building inspectors, then you don’t know what criteria they are using to get your project approved. And if you don’t know what they are looking for, your project can get stuck. It’s easy to blame it on the building department, but if your project is 18 months in scope, thirty days might mean the difference between an on-time finish or having to pay liquidated damages for coming in late. There is a lot of risk.”

“So, when we decided that you had the potential to be a manager, it is because we could see evidence of that potential beyond your role as a supervisor.”

How to Interview for Potential

1 Reply

“I want to hire this person. Of all the candidates I have talked with, they seem to show the most promise,” Monica explained.

“So, you haven’t made up your mind?” I asked.

“No, I said I want to hire this person,” she clarified.

“Are you basing your decision on evidence? You sound uncertain.”

“You are right. The level of work in their previous job is short of the level of work we need in this position. But it might be that she was just underemployed,” Monica thought out loud.

“So, far, you are basing your decision on a promise and a maybe,” I clarified.

“Yeah, but how do you know? How do you know whether or not she has the potential?”

“I asked you if you were basing your decision on evidence. Is there evidence of potential? Look, you spent a great deal of time properly writing the role description. You carefully organized the tasks into Key Result Areas. In each Key Result Area, you defined the level of work. In your interview, you either establish evidence in the level of work or you don’t.”

“You mean I can’t hope?”

Don’t Go to Hope Island

Leave a reply

From the Ask Tom mailbag –

Question:
Yesterday, you talked about looking for evidence of potential. I followed the explanation, but I am still looking for some sort of instrument to help measure a candidate’s capability or more importantly, a candidate’s potential for higher capability. I get that I can measure the level of work in a candidate’s current or former role and match (or not match) that to the level of work in our open role. I can assume that if a person was effective in that level of work somewhere else, they are likely to be effective in that level of work with us. But I still want to know if they might have the potential for higher levels of work now, or sometime in the future.

Response:
There is a temptation in this discussion for managers to visit a place called Hope Island. Hope Island is a wonderful resort destination with luxury accommodations at the Assumption Hotel. The five-star Assumption Hotel has a water theme park sporting a replica of Egypt’s Denial River and a water flume ride where participants ride down the steep Pitch of Performance into a pool of reflective water.

As long as we are not going to Hope Island, we can continue this discussion.

Kevin Earnest replied to yesterday’s post to discuss some specific tools that may be useful to the manager and the manager-once-removed. “As we know from Human Capability, when an employee, manager and MoR “calibrate” to a person’s CPC (current potential capability), they are placing a dot on the Talent Pool Maturation Data Sheet and making a hypothesis about the person’s potential development. If you follow the trajectory on the Data Sheet, we can foresee their anticipated long term potential development. CEOs and managers must continue testing the hypothesis with additional tasks or special projects (real talent pool development) to determine if, in fact, the person is maturing as anticipated.”

So, there is an instrument, a chart that can be useful to the manager and manager-once-removed. This chart is documented most thoroughly in a book first published in 1994 called Human Capability, by Elliott Jaques and Kathryn Cason. This chart is a visual representation of how a person’s natural capability matures through time. It shows different trajectories as each person, in their own life, is on their own path.

Talent Pool Maturation Data Sheet

Talent Pool Maturation Data Sheet

But here is the critical part of Kevin’s response. While the chart may be helpful, it must be tested continually with projects and tasks. Stay off of Hope Island. The only measure of performance is performance. Lee Thayer said that.

Capability in the Team

Leave a reply

I was talking with Claude, a supervisor, about his team. “Those two over there, are the new guys, one has been here a month, the other just got out of orientation last week. They are learning, but it will take them a while to catch on to how we do things around here.”

“How often do you have to check up on them?” I asked.

“In the morning, we go over the work orders from the production schedules. A little huddle meeting. I check back in about 15 minutes to make sure they are moving in the right direction. Then, they’re good for a couple of hours. Right now, I am not as worried about their production output as much as doing the work correctly.”

“And the rest of your team?”

“The rest of the crew has been here at least a year, some, four or five years. They know what to do. For them, our morning huddle is as much social as it is to look at production for the day. I walk the floor a couple of times, morning and afternoon, just to see if they have questions, admire some of their handiwork.”

“When they run into a problem, how do they solve it?” I pressed.

“There are some things they can try, but if they can’t figure it out pretty quickly, they either come to Tony, or me?” Claude replied.

“Tony?”

“Tony is the team leader. Sharp kid. Only been here two years, great technician, twenty-eight years old.”

“So, how does Tony solve problems?” I was curious.

“Same as the other guys, but he is quick. If one solution doesn’t work, he has something else to try. If that doesn’t work, he tries something else. Boom, boom, boom, problem is usually solved. When I have to be out of the office, or on vacation, Tony is my assistant. I can leave him in charge, and not worry. But Tony won’t be with us much longer.”

“Why’s that?”

“I was talking with my manager. She has had her eye on Tony since the beginning, thinks he ready for supervisor training?”
_____
Orientation for our online program Hiring Talent kicks off tomorrow. Registration is now open. Follow this link for more information. Hiring Talent – 2013.

Not a Matter of Training

2 Replies

“And that’s where he stops. He can keep one or two machines busy, but we have fifteen machines and plenty of work for all of them.”

“Who was the supervisor before Ryan got hired?”

“Oh, he was a good guy, kept the place humming. Got promoted to our other plant in Michigan,” Drew explained.

“And there was no one else on the production crew that could take over?”

“No, a good technician doesn’t necessarily make for a good supervisor. It’s one thing to push out today’s work. Totally different to make sure all the machines are scheduled for each shift for the next three weeks. Lots of moving parts.”

“Can’t you train someone?” I probed.

“It’s not a matter of training,” Drew shook his head. “Some people have it and some people don’t.”

“So, what is it, that some people have and others don’t?” I wanted to know.
_____
Orientation kicks off this Friday. Registration is now open for Hiring Talent – 2013. This is the only program that blends Elliot Jaques’ Levels of Work with the Behavioral Interview. This 6-week online program is practical, hands-on, coached by Tom Foster. Follow this link for more information and registration.

Why Can’t Ryan Handle It?

Leave a reply

Drew was beside himself. “I don’t know why Ryan can’t handle this job. We asked him all the questions in the interview. We were quite thorough. He knows the name of each piece of equipment. He can tell you exactly what it is used for, how it is used. We even have trained technicians for him to manage to run the equipment.”

“What’s the problem?” I asked.

“All he has to do is keep the equipment busy. We have sales orders that come out of the front office. All he has to do is look at the sales orders, translate those into work orders, make sure we have the right materials in stock and schedule the work on each machine.”

“And?” I pressed.

“And that’s where he stops. He can keep one or two machines busy, but we have fifteen machines and plenty of work for all of them.”
_____
Registration is now open for Hiring Talent – 2013. This is the only program that blends Elliot Jaques’ Levels of Work with the Behavioral Interview. This 6-week online program is practical, hands-on, coached by Tom Foster. Follow this link for more information and pre-registration.

A Decision Based on Hope?

Leave a reply

Sylvia was perplexed. Difficulty trusting her judgment. “I have this gut feeling that Porter would make a good supervisor. But, he is our best technician. If I promote him and it doesn’t work out, I might lose my best technician.”

“Why do you feel Porter has the potential to be supervisor?” I asked.

“Intuition,” Sylvia replied. “The only thing I am concerned about is his people skills. As a technician, he is a good producer, and whenever anyone has a question, he is the lead guy. Whenever anyone has a problem, they talk to Porter. When anyone has a decision to make, Porter gets consulted. He has a knack for knowing what needs to get done next. I can see his planning skills, always looking ahead. He knows when materials are supposed to arrive, when we need to order, even for the longer lead time stuff.”

“Then what is your hesitation?”

“Sometimes, his people skills are a little rough,” she explained. “I don’t want to promote him and then find out he is a dictator.”

“Rather than assume, or guess, or hope that Porter has the potential to be a supervisor, how could you find out? How could you find out before you promote him? How could you confirm that he is not a dictator?”

“I guess I could talk to him,” Sylvia searched.

“And, so, he tells you he is not a dictator. Is that enough? Is that enough evidence to make a firm decision to promote him?” I pressed.

“Well, no.”

“Then how? How can we create tangible evidence that he has the potential to work effectively with other people?”

“I guess I could give him something to do where he has to work with other people in the capacity of a leader?” Sylvia tested.

“Not a permanent role assignment, but project work. Give Porter a project where he is the project leader for a specific task that requires him to use the resources of other people on a project team. If he fails, you have a broken project, big deal, you can manage that risk. If he is successful, you will have tangible evidence on which to base your decision. Not a hope, a wing and a prayer, but tangible evidence.”

Discovering Potential Capability in an Interview

1 Reply

Quick excerpt from a candidate interview –

“How are you given work assignments?” I asked.

“Well, I meet with the PMs on a weekly basis, just to catch up on progress completed the prior week, update them on logistics for this week. I have to coordinate with our manufacturing shop to make sure the manufactured cabinets and installation components are all coming out to staging at the right time to be installed. So, I really have to figure things out based on piecing together all these moving parts,” the candidate replied.

“How often are you given work assignments?” I pressed.

“Well, even though I recast everything on a weekly basis, I am really trying to run, believe it or not, one year ahead of schedule. In my role as project scheduler, I use a project management software to book out the jobs based on various schedules and the contracts. It’s not really my job to dissect everything, but I do it anyway, just to double-check, make sure no one is asking for the impossible. It’s only when I plan out a year, especially for some of our big jobs, that I can schedule in all the smaller jobs. Things get very fluid at times. It’s easy to get in the weeds.”

This candidate was currently in a Stratum II role. It was his job to publish details in a 60-day look forward production schedule. To do that, he had to accumulate data from several sources and coordinate people, materials and equipment. From one week to the next, there were significant changes to that schedule that required constant coordination and re-coordination. To be effective required solid S-II (Cumulative) processing.

The question on the table is potential. What is this candidate’s potential? Is it possible that the candidate has greater potential capability than is required by his current role?

I always examine the difference between prescribed duties and discretionary duties. Prescribed duties in this role required a 60-day look forward, a published schedule. As long as that 60-day schedule was published, no one had complaints.

But it’s in the world of discretionary judgment that effectiveness lives. “It’s only when I plan out a year, especially for some of our big jobs, that I can schedule in all the smaller jobs.”

“Oh, really. Tell me more,” I wondered.

“You can’t just schedule projects one after the other. Project schedules have their ups and downs. We have a committed crew on a large project, but we might get a project delay waiting for another trade to finish a segment. If that happens, I have a crew that I can temporarily shift to a smaller project. If I can do that, sometimes I can accelerate the schedule of the smaller project, knock it out and get ahead.”

As I listen to this description and ask more drill-down questions, it appears this candidate may be moving in transition from S-II (Cumulative) to S-III (Serial) applied capability. He is planning “what-if” scenarios, alternate paths to the goal, and truly working a 12-month schedule. I don’t make this judgment based on a hope and a prayer. I make this judgment based on real facts and behavior.

Potential does not live in the land of hypothetical. Potential lives in the land of discretion. How does this person make decisions? What is the Time Span of those decisions?

Interviewing for Potential

1 Reply

From the Ask Tom mailbag –

Question:
Many proponents of Requisite Organization claim that a person needs to have work commensurate with their potential capability to be engaged and fulfilled at work. They claim that being required to work at a level of work that is below one’s potential capability can lead to high levels of stress and negatively affect a person’s health. Assuming this is true, how do you assess a person’s potential capability in an interview? If you ask questions about their past experience to assess the level of work they have done before this may not reflect their potential capability (because they may not have had the opportunity to do work commensurate with their potential capability before). Doesn’t this approach entail the risk of hiring someone who will be frustrated, stressed or bored by the level of work in the position?

Response:
You make it sound like working below one’s potential capability is devastating. Everyone works on Time Span task assignments all over the place. What is necessary is that a significant portion of one’s work be fully challenging. And understand that this is ALWAYS a moving target. People constantly grow and mature, we are constantly changing, our Time Span capability constantly increasing. Matching the Level of Work with capability is, as Elliott puts it, always a “work in progress.” So, we do the best we can. As managers, we do the best we can to make this match.

Conducting a candidate interview is likely the most difficult assessment challenge we face, as managers. In most managerial situations, we can observe behavior and output, we can have managerial conversations with our team members, we can ask very direct questions about problems that have to be solved and decisions that have to be made. It’s a walk in the park compared to the candidate interview.

Hiring Talent always carries risk. Making the wrong hire is expensive. It costs dollars, time, energy, morale. I will only make hiring decisions based on evidence of work output based on past experience. I will not speculate, I will not hope, I will not assume. I will only hire on evidence. This means I will restrict my questions to real situations that can be observed and verified.

Does that mean I might miss potential? Perhaps. But I don’t use the interview to assess potential capability. I use the interview to assess applied capability. I am not a clinical psychologist, I am not a soothsayer, I am not a fortune teller. I am a practitioner.

And, as a practitioner, here is one method to get an accurate picture of the prospective candidate.

I take the resume and work it chronologically. This means, I start at the back and work forward, because resumes are typically presented in reverse chronology. I have difficulty seeing patterns and progress in reverse, so I start young and work forward. This simple chronological method reveals natural progress of increasing capability as someone moves through their career. Gets me really close to their highest level of current applied capability.

I have some other thoughts about interviewing for potential capability, so let’s pick that up tomorrow.