To Measure Learning Impact for Competencies, First Define Specific Desired Behaviors
Previously, I covered the importance of identifying the set of observable behaviors that would indicate successful application of a leadership development learning program. I also pointed out that these behaviors typically fall squarely between the traditional concerns of the Learning and Development organization (which focuses on the accomplishment of appropriate learning objectives) and the realm of Performance Management, which looks at sets of behaviors (often referred to as competencies). I concluded by saying that, unless specific desired behaviors resulting from participation in leadership development programs are clearly defined it is virtually impossible to measure the behavior and impact of these high visibility and typically expensive programs!
To address this common deficiency I suggested that Learning and Development professionals should initiate discussions with Performance Management and senior organizational leaders to identify and define these behaviors as a component of each leadership development course or program.
To illustrate this concept, I will explain this process of identifying and measuring some specific, intermediary, observable (Level 3) behaviors using an example from the Executive Core Qualifications (ECQs) used by the U.S. government. (For more on the ECQs visit http://www.opm.gov/ses/recruitment/ecq.asp)
Example: The Entrepreneurship Competency in the U.S. Government ECQs
The ECQs consist of five essential qualifications that the U. S. Office of Personnel Management has defined, based on current research of effective leaders in successful organizations. The five ECQs are 1) Leading Change; 2) Leading People; 3) Results Driven; 4) Business Acumen; and 5) Building Coalitions. Each of the ECQs in turn consists of a set of competencies. Unfortunately for those seeking to measure the impact of learning programs, these competencies are typically complex and intertwined behavior sets, and they can be difficult to observe and measure changes over a short period of time. To arrive at the clearly observable individual behaviors we wish to measure, we have to go further.
To illustrate this process of defining specific observable behaviors in order to measure the impact of learning programs, let’s look at the competency called “Entepreneurship,” which is included in ECQ #3, “Results Driven,” as quoted from the web site above:
ECQ #3: Results Driven
Definition: This core qualification involves the ability to meet organizational goals and customer expectations. Inherent to this ECQ is the ability to make decisions that produce high-quality results by applying technical knowledge, analyzing problems, and calculating risks.
ECQ #3 Competencies
1) Accountability; 2) Customer Service; 3) Decisiveness; 4) Entrepreneurship; 5) Problem Solving; 6) Technical Credibility
Entrepreneurship Definition:
Positions the organization for future success by identifying new opportunities; builds the organization by developing or improving products or services. Takes calculated risks to accomplish organizational objectives.
This definition of the entrepreneurship competency includes behaviors that may be difficult for an organization to observe and measure for each of its leaders. In response to this challenge, we suggest that an organization that wishes to use learning programs to develop the “entrepreneurship” competency within its leadership group should begin by defining specific observable behaviors that demonstrate the application of learning to the entrepreneurship competency. Unfortunately, in our experience working with clients in federal agencies, this is the critical step that is often omitted.
And herein lies the problem: If both the Learning and Development organization and the Performance Management organization fail to connect the dots between learning and performance, it’s very likely that the learners will as well!
Defining Measurable Behaviors to Demonstrate Application of Learning in Entrepreneurship
The challenge then is for the organization to clearly define the connection between the learning program and specific on-the-job behaviors that can be monitored and measured during the appropriate follow-up period for the learning program. For this example, I’ll use two common learning objectives for entrepreneurship programs: 1) openness to new ideas (an attitude); and 2) conceptual problem solving (a skill).
A Leader who accomplishes both objectives should be able to 1) readily embrace new ideas suggested by customers and colleagues and 2) develop conceptual solutions to problems that are not constrained by existing processes, technologies or paradigms. But these valuable, intuitive behavioral outcomes would be difficult to monitor and measure in most organizations. For our example, let’s assume that these behaviors are not typically monitored or measured.
In this case, it becomes necessary to define specific follow-up activities that are introduced in the learning program and utilized to formally monitor and support the implementation of new skills. Here is an example:
Learning Objective #1 (attitude): “Openness to New Ideas”
Tie in to Entrepreneurship Competency: “Positions the organization for future success by identifying new opportunities”
Observable Behavior: Following the program, the participant actively solicits new ideas and maintains a list of “Innovative Ideas.”
Monitoring/Measurement: The participant discusses the list with one or more of the following: a supervisor; staff; learning partner(s); or coach.
Learning Objective #2 (skill): “Conceptual Problem Solving”
Tie in to Entrepreneurship Competency: “Positions the organization for future success by identifying new opportunities”
Observable Behavior: The participant identifies a significant problem within his/her agency and, without regard to the practicality of implementation, describes 2-3 possible solutions that address the problem in innovative ways and lists the potential benefits of each.
Monitoring/Measurement: The participant discusses the innovative ideas with one or more of the following: a supervisor; staff; learning partner(s); or coach.
Measuring Outcomes
Of course, the more the organization formalizes and values these activities, and the more they are integrated into actual work, the more valuable they become. Ideally, the best of these ideas would be selected for implementation and championed as examples of entrepreneurship. These successes can, in turn, be attributed directly to the learning program as business results.
Most importantly, by engaging in this process with organizational leadership and the performance management team, the learning and development team ensures that they are aligned to strategic objectives and that they are measuring and evaluating the success of the programs themselves in a robust and credible manner.
Register for our FREE WEBINAR to learn how you can “Put Hard Numbers on Soft Skills”
About the author:
William Kasper is the Practice Manager for Dashboards and Analytics for PTG International. At PTG he brings to bear a unique blend of knowledge and experience in Learning and Development team management, training evaluation, and in the design and implementation of business performance management solutions. Prior to joining PTG, Mr. Kasper worked for The Advisory Board Company, where he helped some of the country’s most prestigious hospitals and health systems create business performance dashboards and data analytics. His experience as a government contractor includes management of technical training for numerous clients, including Los Angeles County Human Resources, Raytheon, California Department of Transportation and a number of Federal agencies.
Mr. Kasper holds a B.A. from Oberlin College, an M.F.A. from California Institute of the Arts, and has completed additional educational programs at M.I.T. and the Robert H. Smith School of Business.
About PTG:
PTG International provides clients with a range of services and technologies to help them design and implement robust, multi-level training evaluation solutions that are sustainable and cost-effective. Whether providing guidance for a single, mission-critical training program or implementation and management support for an enterprise-wide system coordinating multiple technology platforms (LMS, Performance Management, etc.), PTG strives to help government agencies and private companies accurately measure, monitor and value their investments in learning and development
