Oh expert in the crystal ball, please tell all...tell all!
Questions abound in the learning and performance world. Here are a few we have been asked several times recently along with our answers. We welcome additional questions as well as comments and reactions to our responses.

To ask your question, please click on the crystal ball.

To access our Ask Harold Archives, click here.

In what kind of company would the technical skills of top managers be more important than human relations or conceptual skills? Answer

I'm a training professional working in the pharmaceutical industry and am seeking a basic competency model detailing what performance looks like for classroom trainers. Ideally the model would describe performance at three different levels: basic/fundamental, intermediate and advanced. Can you point me in the right direction? Answer

I'm putting together a pre-test and post-test for one of our training programs. It will be given to some, but not all of the participants to determine if the participant knowledge increased as a result of the training. Since the training we do is not based on clear, measurable objectives, the only test I can create is a knowledge test of the content that is covered. Do you have any information on when a pre- and post-test is effective and when it is not? Answer

Has HPT had a well-documented link to ethical business or corporate social responsibility? If so, who has made this link and where is it documented? Answer

Where can I find current research on transfer of training? Who are its key researchers? Answer

How can the use of effective job feedback designs or job aids replace on-the-job training? Under what conditions would formal training be a better solution? Answer

Is there a method to determine if an individual's mindset is activity-based versus results-based? Answer

In which types of companies are technical skills more important than human relations or conceptual skills for their top managers? Answer

I am teaching a class in training and development and am using your books, Telling Ain't Training and Beyond Telling Ain't Training Fieldbook for my graduate students in a Master of Arts program. On page 141 of the Fieldbook, you advocate selecting training topic assignments and then state "Distribute a 10-minute section of the training program to each participant." Where might one find appropriate topic assignments? Could you suggest several topics that have worked for you in the Video Practice Session (VPS) and provide an example. Answer

In the world of Competence Assurance and job profiles, what percentage of total tasks/procedures must a candidate be assessed against to assure competence? Is a representative sample of critical tasks sufficient or should you assess against every task/procedure available? Answer

I would like some advice on paper versus electronic job aids. I'm concerned that paper gets outdated and people still hold onto the old information, even when updates are provided. The electronic format is good, but it's not as convenient so may not be used as much. Please share your thoughts on this subject? Answer

In what kind of company would the technical skills of top managers be more important than human relations or conceptual skills?

This a tough on; there is no fixed rule for this. However, in general, small, entrepreneurial companies that have some form of cutting edge technical capability can often grab a lot of new business because of the leaders' recognized technical competencies. The innovation excites customers and the demand is such that business acumen, interpersonal skills and even communication or marketing skills are of lower priority. Often, the organization takes on the form of a cult with the leaders as technical gurus surrounded by dedicated disciples who work crazy hours to build success.

As the organization grows, this highly personalized, technically focused adventure begins to outgrow the initial visionary and innovative thrust that launched it. Reality and day-to-day concerns emerge. To survive beyond the launch requires more mature conceptual, business, marketing, financial and interpersonal skills. Other players enter the market and the cutting edge uniqueness of the service or product now faces competitors. Demand softens and the realities of the competitive marketplace begin to diminish the focus on the technical aspects alone of the enterprise. Financial supporters have to be wooed and appeased; new people who are brought in have expectations that must be met beyond the great vision of the start-up. The leaders find themselves faced with the hard-core and mundane requirements of running a business. Not all visionary and technically brilliant entrepreneurs can make the shift.

Hence, to finally answer your question, those companies that have caught fire because of the technical brilliance of their leaders (e.g. General Electric, Hewlett-Packard) come to mind first. Those companies that are largely technically driven also require leaders who are highly skilled in their technical areas. Professions such as law, accounting, medicine and certain fields of consulting, including Human Performance Technology, certainly must have technically capable leaders to inspire and generate credibility in the marketplace. However, in all of these instances, as maturity and growth occur, other skills must be added. Sustainability requires that conceptual, interpersonal, business and a wide range of organizational and management skills be present. Without these in the leadership (or if the leaders are wise, then brought in to help in decision making), the company is unlikely to prosper long term. Technical skills may be necessary. However, in the long run they will probably not be sufficient.

I'm a training professional working in the pharmaceutical industry and am seeking a basic competency model detailing what performance looks like for classroom trainers. Ideally the model would describe performance at three different levels: basic/fundamental, intermediate and advanced. Can you point me in the right direction?

There are many beliefs and models out there for you to pick and choose from. However, if you are looking for one that is structured, has a long track record, is both process and performance centered, and has been used with a large number of organizations, I recommend the program provided by the Canadian Society for Training and Development (CSTD). Here is the link, http://www.cstd.ca/certification/index.html. This is a program to certify trainers at increasingly higher levels that has been around for many years and has had solid success. I believe that CSTD offers a variety of arrangements for organizations.

Another example of a series of graduated modules for building ever-increasing training capabilities is the Trainer Competency Track (TCT). The link to this is http://www.nystc.org/committee/minutes/pdt/competency.htm. It is arranged in five levels and, at the top, includes performance consulting, organizational development and many other areas for professional growth. You'll have to do some investigating to learn more about the modules and the model that underlies it. My knowledge is only cursory concerning the programs.

Another "model" is the one devised by CHART, the Council of Hotel and Restaurant Trainers. CHART is an industry-wide non-profit group that encompasses most of the hospitality organizations and has been around for years. As you can imagine, this is an area in which a huge amount of training occurs. Here is a link to a "white paper" about the CHART model: http://batrushollweg.com/Insights&Research/WhitePapers/BlueprintForTrainerDevelopment.pdf

I could pepper you with a host of models. Many are industry specific. I leave you with a link to the directory of the Society of Pharmaceutical and Biotech Trainers, http://spbt.klickit.com/local/files/vendor%20directory/vendorproj_proof2.pdf. It has a section of trainer programs arranged in a hierarchical format. The Society itself has tackled issues such as trainer development, performance improvement and competency modeling. As this is closer to your organization's universe, I recommend connecting with SPBT and getting your hands on what it has produced.

I'm putting together a pre-test and post-test for one of our training programs. It will be given to some, but not all of the participants to determine if the participant knowledge increased as a result of the training. Since the training we do is not based on clear, measurable objectives, the only test I can create is a knowledge test of the content that is covered. Do you have any information on when a pre- and post-test is effective and when it is not?

First, let's define pre-test and post-test. A pre-test is a post-test equivalent that is administered prior to submitting "subjects" to a "treatment." In the training world, this means that you create a test based on what it is that trainees are to master as a result of the training and administer it before and after the training. This allows trainees who have already mastered portions of what is included in the training to skip these. To ensure that trainees have not memorized the responses from the pre-test when you administer the post-test, you should create a different but equivalent set of test items for each of the tests. To do this requires some knowledge of both test item construction and a bit of statistics to ensure equivalence of tests. Usually, we construct test item banks and then draw from these at random to ensure equivalence. It may sound complicated, but it is not so bad. You need to have someone help you with this until you know what you are doing.

Now for the more important matter. Training has long history of being criterion-referenced. This means that specific and measurable criteria for mastery are established, normally derived from performance-based task analyses that lay out the required outputs and outcomes of a job (or set of tasks within a job). Training is designed to ensure that trainees meet these criteria. The criteria are expressed through verifiable objectives that are trainee centered, contain the verifiable performance and state a standard of acceptable performance given the level of the trainee group. Test items are created to perfectly match the objectives. Our book, Engineering Effective Performance Toolkit, details all of this with lots of job aids.

If you do not have performance-based objectives, it is difficult to create any kind of valid test. In addition, testing declarative (talk-about) knowledge of a content area that requires procedural (doing, using) knowledge is of little use except to say, "I tested them." There is very little correlation between being able to talk about what you can do and being able to actually do it. The declarative knowledge test will only be useful for those portions of the content that are to be remembered verbally. They have very little impact on performance results (e.g., talking about the features and benefits of a product versus actually being able to demonstrate the product in a convincing manner).

As for sampling trainees, that is fine if the sampling is random and meets sampling criteria. You can find guidelines for this on the web or in books on testing and sampling.

To conclude, what is the purpose of the activity? If it is to show that there are knowledge gains from a course so that someone can say,"See, they learned," then the approach being taken will work. Trainees will probably remember something that they did not know before. If it is to verify whether or not the training had an impact on their ability to perform, then the results of such an exercise will not be helpful.

As a side note, we are currently conducting a research study on transfer of training. As has been found elsewhere in many studies, declarative knowledge learning and retention is high. The trainees learn and retain enough to do well on the tests and significantly better than on the pre-tests. However, when we examine application on the job, the portrait changes markedly. They may "know," but they do not "apply." We are studying the variables that facilitate or inhibit on-job transfer.

Has HPT had a well-documented link to ethical business or corporate social responsibility? If so, who has made this link and where is it documented?

Your questions about HPT and ethical business and social responsibility are very relevant in today's highly competitive enterprise climate. As HPT professionals, not only do we have to act ethically (and be seen to do so), we must also make sure that we provide guidance and counseling to our clients that in turn help them to act appropriately. I cannot vouch for a "well-documented link" between the writings on HPT and corporate social responsibility, but the theme has often been addressed in HPT publications. All of the Handbooks of Human Performance Technology have chapters dealing with ethics and standards. The latest one, edited by James Pershing, is no exception. Please check out chapter 44, Standards and Ethics in Human Performance Technology, written by Ingrid Guerra. In particular, examine the section: Social, Corporate, and Professional Consciousness starting on page 1,032. The references section is also a great starting place.

You might wish to contact Ingrid at Wayne State University. Another person who has long been interested in the ethical dimensions of out field is Peter Dean in the College of Business, University of Tennessee. He was a senior fellow in the ethics program of the Wharton School's Department of Legal Studies at the University of Pennsylvania for several years.

Like most professions, HPT is very concerned about its practitioners operating ethically and helping clients do things correctly. From my own consulting experience, I have unfortunately had to deal with potentially unethical situations. In some cases, I assisted the clients to make the right decisions. In others, I had to walk away from the projects because I could not in good conscience participate in activities that were not in accordance with HPT's ethical standards. I made my case, attempted to change the clients' directions and when I could not, simply withdrew - but documented the reasons for my withdrawal to the client. There are other HPT professionals who have shared similar cases with me over the years.

Where can I find current research on transfer of training? Who are its key researchers?

Transfer of training - transfer of learning is a huge and multi-faceted focus of interest for different kinds of researchers. In 1997, I edited a special issue of Performance Improvement Quarterly on the subject. Since then, research has continued. There are theoretical questions embedded in the transfer arena as well as practical ones. Some researchers are interested in the mental mechanisms of transfer, others in the external factors affecting transfer of training to the workplace. Some are mainly interested in children or students and school learning whereas others are concerned about workplace and return-on-investment issues. And when we examine transfer, are we concerned about lateral transfer, vertical transfer, near or far transfer, or fluid or crystalized transfer to name only a few of the types of transfer researchers study. In the Performance Improvement Quarterly issue I mentioned above, Achi Yapi and I reported on a study we conducted in Ivory Coast on the effect of behaviorist and cognitive based cases on transfer of learning (Stolovitch, H.D. and Yapi, A., 1997. Use of case study method to increase near and far transfer of learning. Performance Improvement Quarterly, 10 [2], pp. 64-82). The hypotheses we stated were confirmed by our study.

To come back to your question, I strongly recommend that you read broadly on the topic. For this, conduct an online search to identify articles dealing with more general facets of transfer. Use these to help define terms and identify research agendas as well as key players. The most promising databases are ERIC, Psych Abstracts, Dissertation Abstracts and Current Index to Journals in Education. I especially like to pore through Review of Educational Research.

Notice that I am not giving you fish, but rather am suggesting ways for you to do your fishing. I am not current enough on the latest research in this area and so I think you are best served by delving into the literature on your own.

How can the use of effective job feedback designs or job aids replace on-the-job training? Under what conditions would formal training be a better solution?

You pose your questions as if we might be pitting job aids against on-the-job training and both against some form of formal training. I suspect that this is not your intention. Rather I think that you are looking to discover how these three all fit together. And, yes, they all do. Job aids are basically external memory storage devices. They come in many varieties, but all do the same thing. They guide you to perform without your having to remember all of the steps, decisions, or details. They all range in complexity and sophistication from very simple ones such as an address book that remembers phone numbers and addresses for you to much more complex ones that can guide you to correctly calculate your income tax or make an important technical or buying decision.

Examine an office photocopy machine. You will see a small screen that helps you select the right paper, choose whether or not to collate, staple, or sort, etc. Based on what you want to do, it lists the steps to follow. If you make an error, it helps you fix it. I use a paper job aid that I keep in my wallet for making phone calls when I am traveling abroad. It lists countries, matches them with access codes and numbers, and provides me with costs per minute. It even has a little troubleshooting guide to help me out if I cannot reach my party.

On-the -job training (OJT), for the most part, is informal. A person is assigned to the mail room or the warehouse, where she or he is placed into the hands of an experienced employee who "shows her/him the ropes." How organized this type of training is depends on the organization and its managers. OJT can include shadowing an experienced worker, having a work buddy assigned, or simply having access to people who know how to do "it" when you get stuck. There are, of course, more formal structured on-the-job training (SOJT) programs for which materials, trained workers in SOJT practices, and evaluation systems have been implemented.

All of these work to a greater or lesser degree. The more informal the OJT, the less certain you are of the effectiveness of what is taught. OJT, however, is the most common form of training to be found in most organizations. The U.S. Department of Labor Statistics estimates that four to five times more time and money is spent on informal OJT in American business and industry than in formal training programs.

Helping someone to learn on the job works. So do job aids. Job aids are best suited for providing information that a worker must access but would have difficulty perfectly retaining due to excessive detail or infrequent use (e.g. hazardous materials and how to recognize each and every one along with what intervention is best when something goes wrong; sequence of steps for restarting an assembly line after troubleshooting). They are also useful for making decisions when a number of variables have to be considered.

My recommendation is not to substitute job aids for OJT, but rather to use them together. Create excellent job aids and have experienced workers demonstrate their use with novices and then have the novices practice using the job aid under the watchful eye of the OJT coach. This improves consistency of performance and decreases idiosyncratic ways people develop for doing work.

As for formal training, the general rule is if there is a need for specialized skill and knowledge on the part of the trainer beyond what a skilled worker possesses, then formal training sessions are probably best. Once again, these can be combined with OJT. "Theory" is best done outside of the immediate work environment. Accurate and consistent rules and principles are best taught in more formal settings (although the sessions may take place at the work site).

To sum up, one is not a direct substitute for the other. Provide formal training to lay the foundations for job skill and knowledge, advanced knowledge and skill building, and legal reasons. Use formal training (live, self-paced or virtual) to allow workers to try out new skills in settings where they will not disrupt the work flow nor be distracted by what is happening at their work sites. Use OJT to ensure workers are applying their skills and knowledge appropriately. Create job aids for tasks that can be done by following the steps or applying the job aid to specific work tasks (e.g. a color chart card to lay against hydraulic fluid in a glass tube to determine what action to take). Have these introduced formally or on the job and provide OJT to practice appropriate use of these job aids.

Our book, Beyond Training Ain't Performance Fieldbook, has more about job aids, how/when to create them, and what varieties exist.

Is there a method to determine if an individual's mindset is activity-based versus results-based?

The most direct way of doing this is to provide sample cases to the individual and ask him/her to describe (or work on if there is time) how s/he would deal with it. Give the person free rein and encourage him/her to deal with it if there were no constraints. This is for those whose work you cannot access. You can also ask the person to describe a performance problem from the past and have him/her present what s/he did to solve it. See literature on performance-based interviewing.

Another alternative is to provide a case to the individual with two alternative approaches, one activity-based and the other results-based. Have them select which one s/he prefers with a rationale. Of course, you have to be careful not to make the preferred one obvious.

If this is someone who is working inside your company, ask to see work samples. Obtain documents and verify with clients what was done. Probe for the rationale behind the approaches taken. Factor in organizational or client constraints.

For me, performance-based interviewing and hiring offers a higher probability of success in candidate selection than any other method I have encountered. Don't worry about "mindset." Focus on what the person actually demonstrates to you when faced with an operational issue. After a couple of cases, you should have a pretty solid portrait of how the person goes about performing.

One note of caution: since people who are being tested usually want to do well, you have to avoid building in bias toward an activity versus results orientation. If this is an internal person who knows that the organization prefers or in the past has favored a given approach, then s/he may opt for it to "look good."

In which types of companies are technical skills more important than human relations or conceptual skills for their top managers?

The answer is in very few, if any. Perhaps a very well-known and well-established research and development company might qualify or a company that is engaged in technical activities related to the military or other technically strategic areas for which there is little or no competition and are subsidized by government funding (e.g. Livermore Labs; Sandia). Realistically, however, top management has to "have it all," but not necessarily equally in all of the members of the senior management team. The leader may be strong in the financial areas and various members of the company's leadership may have differing strengths. Together, like a sports team, they integrate their capabilities to do a great job.

That said, a top leader must possess vision, a sense of the market and have connections with investors as well as means for inspiring his/her team, shareholders and key personnel. The head of the company may not be pleasant (e.g. Disney's former CEO, Michael Eisner; Rupert
Murdoch) but they do have vision and a strong conceptual handle on their markets as well as their capability as an organization.

To conclude, while technical skills can be extremely beneficial for top managers, especially in highly technical fields, they are insufficient (except for the rare cases I cited, and even then...) to ensure company success long term. Conceptual skills are essential to make the company prosper. The ability to inspire, lead or drive people to perform and recognize and reward high performance are essential for company leaders. I would guess that an examination of the most successful companies will show that in most cases, the top managers are not the most technically qualified. Rather, they are the ones who find the technically talented and draw from them their very best.

I am teaching a class in training and development and am using your books, Telling Aint Training and Beyond Telling Ain't Training Fieldbook for my graduate students in a Master of Arts program. On page 141 of the fieldbook, you advocate selecting training topic assignments and then state "Distribute a 10-minute section of the training program to each participant." Where might one find appropriate topic assignments? Could you suggest several topics that have worked for you in the Video Practice Session (VPS) and provide an example.

The VPS is like a shell. Any content that the participants normally teach is usable and can be placed within it. If a new course or program is being introduced, you would chop up the new program into discrete 10 minute chunks. Here are examples from some sessions:

• A real estate company introduces a new program for certifying agents. The trainers are introduced to the program and trained on it. They then receive slices of the program for practice during the VPS.
• Trainers are training in car dealerships. They are teaching about Warranty Claim processing. Again, they go through the program as learners and then examine the leader guide, are assigned parts of the course and teach a 10 minute slice during the VPS.
• Recently, I had Store Managers from a successful bookstore chain. They developed their own 10 minute slices on things they teach to their staff. Topics consisted of: selling the customer card; inventory checking; book location; ordering books; handling customer complaints.
• When I had a group of people who were not yet training in an organization, I had them select small topics for which they had expertise. Some examples were: decorating a cake; checking oxygen tanks before scuba diving; solving a math puzzle; how to read resistor codes; using a slide rule and other similar fun topics.

The key is for them to have a model for training, such as our Five Step Model in Telling Ain't Training, receive coaching and have opportunities to practice before they are videotaped. Then they are placed in a VPS setting. We find the VPS most useful when a company is introducing a new program to its trainers and after taking them through it, slice up the program and have them teach parts of it. It is also very useful to bring experienced trainers in for a training clinic. Have them select a part of what they already train and then have them go through the VPS for coaching.

In the world of Competence Assurance and job profiles, what percentage of total tasks/procedures must a candidate be assessed against to assure competence? Is a representative sample of critical tasks sufficient or should you assess against every task/procedure available?

This is a knotty issue. It may sound like a cop out to say, "It depends," but that is the only response I can offer. The more risk that is involved in the job (for example, Bomb Disposal Technician), the more you must test for "everything". The reverse is also true. If the job is one of low risk (for example, magazine sales person), then it is fine to sample only those tasks critical to the success of the job. Here is what I recommend. Perform a job analysis that systematically lays out all of the job tasks. Go at least three levels down: major tasks, sub-tasks and sub-sub tasks. Using experts who know the job well, top performing incumbents and, if feasible and appropriate, "customers" of those in the job - internal or external - have them rate each task on three dimensions: frequency, importance and risk. Use a simple rating system of High, Moderate and Low. Any task that receives three Highs should definitely be assessed. Any task with two Highs or two Moderates should also probably be checked. Feasibility and time become issues here. You can probably let any task with two Lows or more and no Highs go. This is not a scientific procedure, but does allow you to make assessment choices based on credible input. I do have one caution. If any tasks have legal or safety implications, assess for these.

On a side note, I am not a big fan of "competency" assessment. They are suppositions. We suppose that these competencies will produce desired results. I prefer Performance Modeling. Here is my explanation. Let us assume that you are assessing those in an automotive sales position. A competency model might include, "excellent oral communication." The thought is that a car sales person should be able to speak clearly and talk to the customer in a convincing manner. Perhaps this is true. Perhaps not. Is it possible to determine a customer's needs and "hot" buttons and make a sale without being a wonderful speaker? The answer, based on our studies, is "yes." The performances we are looking for are overt and verifiable and lead to job success in demonstrable ways. Performance is valued accomplishment derived from effective behavior. Both are overt and can be empirically assessed. Competencies are assumed to be correlated with successful performance. Therein lies the danger.

I would like some advice on paper versus electronic job aids. I'm concerned that paper gets outdated and people still hold onto the old information, even when updates are provided. The electronic format is good, but it's not as convenient so may not be used as much. Please share your thoughts on this subject?

Paper or electronic, both are media and each has its place in the array of job aids we use. Part has to do with the use people will make of the job aid. If you are baking a cake or making a stew and have stuff all over your kitchen counter, a paper cookbook or index card is probably more convenient than an electronic device. Similarly a job aid pasted to a telephone with emergency numbers is more practical than some electronic instrument. On the other hand, job aids for trouble shooting equipment are more helpful in an electronic mode, particularly if they allow access to databases and expert advice to deal with highly variable circumstances.

So how should you decide which medium to use? Here are some rules of thumb:

1. For simple procedures that do not change much over time, paper job aids work well. They also tend to be more cost effective and can be carried around, pasted up or mounted in a prominent place very easily.
   
   
2. For complex procedures or requirements to access a variety of procedures, some of which are complex, electronic job aids are generally better.
   
   
3. Consider the level of technical sophistication of the end-users and their preferences. Some people cannot adapt to PDAs. They prefer paper daybooks and planners. For less technologically experienced users, paper job aids are generally easier to use.
   
   
4. Consider the environment in which the job aid will be used. In some, electronic devices will not work well (e.g. in an underground tunnel; at a buzz saw). In other environments, electronic job aids are by far superior (e.g. inside the cockpit of an airplane; at a terminal). Analyze the implementation context to determine which medium is more compatible with environmental characteristics.
   
   
5. Consider the volatility of the content. The more frequently the content of the job aid is likely to change, the more appropriate electronic job aids are if there is a means to update these automatically. If not, they are no better than paper ones.
   
   
6. Finally, consider cost. If electronics are at hand and are cheap and easy to use, then that's the way to go. If some of the end users will not have access to the electronics and you can produce a paper-based job aid quickly and cheaply, then paper is your best bet.

The bottom line is that practical considerations reign when it comes to selecting the appropriate job aid medium. Implementation variables usually dominate here. The most important aspect of a job aid is its design for accuracy, clarity, ease of use and accessibility. Focus more on these than which medium is better. The right choice will naturally emerge.

ASK HAROLD !