Posts Tagged ‘single loop learning’

My 2010, a year of blogging

January 6, 2011

Today marks my 1st anniversary of blogging, and 2010 was an “interesting” year, as they say.

I changed continents (again), and took on multiple roles in addition to being a global director of Knowledge Management, and wound up job hunting – but not in that order.

My roles this year included my main job, that of a director of Knowledge Management, as well as unofficial Chief Learning Officer which saw me in many meetings with universities and vendors of learning materials, head of Localization & Translation in which I inherited a recently emptied department and a strangled budget, and made friends with several translation vendors across the world.
Another role was that of program manager of the offshoring and outsourcing activities that took me to India, and involved building a team of over a hundred software engineers while I also managed the contract and relationship of a similar-sized group in Bali, tried vainly to move some outsourcing to South-Africa, and celebrated the building of a 15-person team in Chile.
During this time I received a few “what are you doing” phone-calls from the Australian embassy in India, and many people avoided me in case I was looking to offshore their job.

In July I my relationship with Mincom ended, and having waved goodbye to Brisbane, found myself back in Denver and job hunting.
Since then I have interviewed with dozens of firms – been hugely interested in some, horrified by a few, and left others feeling vaguely relieved not to be working there and having to breathe in their toxic culture on a daily basis.
Some interviews ran into several months and included large panels only to end with me as the runner-up, while others ended in a fizzle when the budget vanished, the position was cancelled, or the VP herself resigned after missing several chances for an interview.
Some had really sharp and focused job descriptions (HP, Invensys, and Philips for example), some had a copy/paste smorgasbord, and some had job descriptions that were a complete mystery.
Some organizations were clear and transparent about their process, others seemed to be playing it by ear and making it up as they went along.

Generally, the people were nice but clearly unsure about what they are trying to achieve – one guy spent 30 minutes posing an elaborate scenario that he fed me piece by piece until we arrived at the answer he apparently had in mind. According to him this was the first time anybody had given the correct answer but he was seemingly unhappy with that so I didn’t get the job.
Maybe just as well, all things considered.

I often wonder how much a firm’s recruitment practices are a reflection of what it is like to work there, and what effect recruitment practices have on their clients.
According to a few research papers I read, it is and it does.

Keeping Busy

Besides looking for a permanent employer, flying around for interviews, and making copious resume modifications to satisfy recruiters, I blogged on KM-related topics, networked, and read several IO Psychology and KM textbooks from cover to cover. Some people have hobbies, some play golf, and I read textbooks – go figure.

Some people take a break from work when they are between jobs, I mostly designed questionnaires and wondered about Communities of Practice.
I also thought about Sharepoint a lot – can you believe it, 130 million licenses and likely to hit 97% adoption rate this year?
Again, go figure!

As part of a Master’s in Knowledge Management I covered various maturity models and although I really liked the KMMM by Lange & Ehms, the K3M by Liebowitz & Beckman, and the various KMMI attempts, they all seemed to be heavy on the Conservation side and light at the Innovation end. I also felt that they neglected the point made by Argyris that processes will inevitably obscure and hide those systematic problems that are essentially never spoken about – things that we become systematically blinded to by the way we measure and think. As a result I built my own KM Maturity model based on the Carnegie-Melon CMMI, with two added layers bookending the CMMI, and blogged incessantly about the implications of Argyris and his Single and Double Loop Learning concepts. The blog about externalization and avoidance I was sure might get me lynched by recruiters.
Seems I must like Argyris, because he comes up in my blogs more than any other author.
Owing to my longtime interest in IO Psychology and research methods, the offspring of the KMM was a questionnaire instrument (currently in Beta), which of course lives on a KM wiki (CoP-M).

To get a better way to benchmark and examine the current state of KM in an organization, I developed a KM & OL Climate questionnaire called the KMOL-C which is now in its RC2.1 version with an RC-3.0 in planning.

During this time I also started thinking about starting my own LLC, firstly because even providing pro bono consulting in the US means one is vulnerable to being sued personally.
Secondly it would allow me to do paid consulting and contracting.
I am still stuck for a company name however, so feel free to suggest one.

Having Fun

Mostly I read textbooks for fun, but I also had many enjoyable discussions, debates, and arguments with HR people on LinkedIn – Since I was dealing with them a lot it seemed logical.
I also played with some new applications – bibus an opensource equivalent to EndNote, Qiqqa a nice CAQDAS tool for qualitative interviewing, R a free statistical package that means I can’t afford SPSS or SAS, and ggobi a graphical add-on for R. No spinplots like VisualStats had, but VisualStats seems to have stopped.
I added all my books to GuruLib – mostly by using a webcam to scan in the ISBN barcodes.
Using R, I pulled correlation numbers for a survey I did for a LinkedIn discussion and discovered that self-identified HR people are more than twice as risk averse as operational managers.

Reading academic papers was very enjoyable, and in case you think they are all boring and filled with indigestible facts and arcane theory, here is one I particularly enjoyed:

My most favorite piece of research findings was this one about penguins (Meyer-Rochow & Gall 2003)

Now besides the fact that next to ducks, penguins are the next funniest animal, there is something inherently funny about research that clocks the speeds and distances of penguin evacuation.
Also, knowing that penguin poop is more dense than blood but less dense than honey, and is ejected at pressures that approach that of a car tire, is just fascinating.
The paper also won an IgNoble award in 2005

What’s in Store for 2011

I hope to register my own LLC soon, start a PhD, and get a job with a really interesting and innovative company, and I hope to use my survey instruments with several organizations, volunteer time to worthy organizations, and stay healthy.

More to the point, I hope to carry on enjoying knowledge management, IO Psychology, and discovering interesting ideas and people.

… and blogging, of course.

~~~

Matthew Loxton is a Knowledge Management expert and holds a Master’s degree in Knowledge Management from the University of Canberra. Mr. Loxton has extensive international experience and is currently available as a Knowledge Management consultant or as a permanent employee at an organization that wishes to put knowledge assets to work.

Advertisements

Building a CoP Maturity Model and Questionnaire Instrument – WIP

November 2, 2010

This blog is a slight departure from normal, and discusses a work in progress rather than a statement of fact or opinion. It is also an invitation for readers to participate and to contribute in something that everyone can reuse when done.

Those who wish to contribute can do so on the CoP Maturity Model Workshop Wiki

Introduction

In anticipation of my next job, I have been doing several little projects, one of which was looking for a questionnaire instrument with which to measure the Community of Practice (CoP) status in an organization.

The idea was to get a snapshot of where the firm is in terms of CoPs, and then to both measure progress from that point as well as to use the knowledge to better shape an approach and to know where to push and where to pull.

The first step in any research or endeavor is usually to conduct a review of current literature – look to see what has already been done and to leverage that if possible rather than waste time re-inventing something that already exists.

What I found was absence of evidence – no questionnaires to be seen, and none of my 500+ LinkedIn contacts or network of Knowledge Management practitioners responded with any knowledge of such a tool.

CoP Construct Structure

Questionnaire design is hard work but quite enjoyable if you have the time, and presently I have some of that, so I set about the first step of questionnaire construction.

No, not writing a bunch of questions, but doing a mind-map of the constructs – two key principles in questionnaire design are to have high construct-validity and reliability over time.

The mind-map at time of writing looked like this:


[fig 1]

The major dimensions or constructs of “CoP Status” I believe are:

  1. Maturity
    Structure to be determined
  2. Penetration
    How far across the departments CoPs have spread, as well as how far up and down the hierarchical structure they concepts and practice has spread. Maturity could thus differ horizontally and vertically, with empty spots, immature patches, as well as enclaves of highly mature CoP presence.
  3. Activity/Energy (can’t decide between the two yet)
    Basically how busy are CoPs and how much effort are people putting into them. A CoP may be mature and have penetrated well, but still be subdued and exhibit low activity.
  4. Externalization
    A measure of whether CoPs had stayed within the organization or have spread outside to domain-specific bodies (e.g. SIGs, Fraternities, professional organizations, etc.), suppliers, customers, or business partners.

The bit that seemed least fleshed out was a maturity model, and again I surveyed for existing literature – and again came up relatively empty. Those people that already had a maturity model regarded it as proprietary, and those that didn’t, didn’t.

First-Pass CoP Maturity Model

Not to duplicate effort, I borrowed the basic CMM model, added a level 0 which I took to be a naïve baseline that would look something like Argyris’s Model-1 state rather than a null situation – The thought being that a naïve state would be structured in a typical silo fashion by department.

  1. Level 0 – Learned Incompetence
  2. Level 1 – Awareness of process
  3. Level 2 – Repeatable process
  4. Level 3 – Defined process
  5. Level 4 – Managed process
  6. Level 5 – Optimized process

I have added to each level a first pass substructure of what I think would be going on at that level of maturity.

 

  1. Level 0 – Learned Incompetence
    1. Unaware of CoP concepts
    2. No efforts to organize or associate
    3. SMEs isolated from others beyond silo boundaries
    4. Silo’s, Silo’s everywhere, nor any flow between
    5. ..?
  2. Level 1 – Awareness of process
    1. Awareness of basic CoP concepts
    2. Initial efforts to self-organize
    3. Initial definitions of purpose
    4. Invitation by social network across boundaries
    5. Sporadic flow between silos
    6. Celebration of desired outcomes
    7. Connectors and Mavens identified
    8. .. ?
  3. Level 2 – Repeatable process
    1. Scheduled meetings
    2. Membership criteria
    3. Rules of interaction
    4. Codes of conduct
    5. Controlled vocabulary
    6. Enforcement of norms
    7. Inter-silo flows regular but dependent on Connectors
    8. Salesmen identified and empowered cross boundaries via Connectors
    9. … ?
  4. Level 3 – Defined process
    1. Defined CoP purpose
    2. Domain defined
    3. Objectives defined
    4. Policies formalized
    5. Norms formalized
    6. KPIs and success factors defined
    7. Inter-silo flow mechanisms established and documented
    8. … ?
  5. Level 4 – Managed process
    1. Formalized processes and norms enforced according to a formalized process
    2. KPIs monitored and wayward activities brought under control accordingly
    3. Celebration events controlled
    4. CoP Champion formally identified
    5. Budget provisions for CoP activities established
    6. … ?
  6. Level 5 – Optimized process
    1. CoP is integrated into strategic decisions
    2. CoP utilized to gain competitive advantage
    3. Hiring mostly done via CoP external connections
    4. CoP forms integral part of marketing and branding
    5. … ?

What’s Next?

The next step is more minds and a bit of basic collaboration – I need some input and thoughts to flesh out, and then flatten, sharpen, and contextualize the framework so that we get something worth testing.

  1. Workshop, Brainstorm, Re-jig, and then Formalize the Model
  2. Test it out by seeing if it mirrors what exists out there.
  3. Construct the questionnaire instrument
  4. Test the instrument on a small population of well known organizations
  5. Rinse and repeat until it seems to have adequate reliability and show signs of construct validity
  6. Give it to everyone and hope they find it useful

Conclusion

CoP success in a firm is vital to many things and having tools to measure what level of maturity, penetration, and level of activity there is in this regard is important. Furthermore, CoPs can play a vital role in the branding and market image of a firm and CoP activity in external bodies can be a critical source of innovation at low investment cost.

This blog is a starting point to develop a usable instrument with which to carry out such measurements.

 

Please contribute to my self-knowledge and take this 1-minute survey that tells me what my blog tells you about me. – Completely anonymous.

~~~

Matthew Loxton is a Knowledge Management professional and holds a Master’s degree in Knowledge Management from the University of Canberra. Mr. Loxton has extensive international experience and is currently available as a Knowledge Management consultant or as a permanent employee at an organization that wishes to put knowledge to work.

 

Death, Learning, and Corporate Survival

October 26, 2010

Why do mature companies die or grow frail and get eaten?

After all, once they have passed through the helter-skelter of childhood and have attained stability after the hectic days of early formation, why don’t they just live on forever?
This was a topic that interested Arie de Geuss of Royal Dutch Shell and he asked a similar question to one that led to a breakthrough in medical science almost four centuries ago – could the same hold for how we look at corporations?

Death as a subject

In 1662, John of Graunt built tables of mortality for the city of London, listing for each year the numbers of deaths by cause. This required not just the collection of data about death, itself a valuable exercise, but also required him to think in terms of categories of causes of death. Although many of the categories have changed over time, this process of thinking once set in motion, led to steady revision and improvement.

For example, from the year 1632, Graunt lists these as the top five causes of mortality:

Chrisomes*, and infants        2268
Consumption
**                     1797
Fever                                    1108
Aged                                     628
Flock
s†, and the small Pox    531

*Infant mortality before 1 month of age
**Tuberculosis
†Means “sediment”, but it is unclear what Graunt meant by this in conjunction with Smallpox

This systematic approach paved the way for tracking and intervention, and gave birth to the science of demographics and enabled epidemiology to develop.
You could say that Graunt was a necessary and key player in the development of modern medicine.

The Mortality of Companies

In his analysis of companies in terms of mortality, de Geuss created categories from the data that led him eventually to conclude that companies die because they develop learning disabilities – they became deaf and blind, and stopped learning – and therefore eventually succumbed to external forces that they were unable to notice or against which to marshal an appropriate response in time.

I view this in terms of Organizational Learning (OL) – which is why I describe my occupation as “Knowledge Management and Organizational Learning”, and I break it into five major components:

  1. Stimulus-Response Learning
  2. Vicarious and Promiscuous Learning
  3. Scenario Planning
  4. Ongoing Professional Development
  5. Innovation Intent

Stimulus-Response Learning

This is the kind of thing that even an earthworm can do, but which many organizations seem to lack.

If an earthworm touches an electrified wire, it eventually learns to avoid the wire, no matter which part of its body did the touching. In contrast, some companies will repeat the same mistake over and over again, seemingly needing to reiterate the same mistake several times with each and every business unit and team before the message finally gets through and becomes part of its adaptive repertoire.

Being smarter than an earthworm should not be that difficult for a corporation made up of smart people, but it means that internal communications and repositories are done in such a way that if one part of the organization makes a mistake or encounters something that poses a risk, that all other units and geographies have access to that same information in a way that they can actually use (and do!).

This turns out to be more difficult than one might assume and the “plumbing” side of providing email, portals, knowledge-bases, and content management are only about a third of the solution. The remainder is a corporate culture that is able to learn across divisional boundaries, and for this you need both leadership and vibrant Communities of Practice

Many organizations never get this far, and die because the rock that they stubbed their toe on last year, came back and hit them in the head this year.

Vicarious and Promiscuous Learning

Once one has evolved past the realm of Annelids, the next big advantage is to learn from other people rather than needing to take the lumps yourself. This saves money and time, and is therefore a direct competitive advantage.
Rome learnt from Carthage, apprentices learn from their tradesmen, and hopefully a company can actively look for examples of what to do and what not to do by observing others. Except where patents and copyright are an obstacle, the keyword is to “shamelessly borrow” ideas and then modify them to fit localized conditions.

This is best done by the leadership team, and by the Communities of Practice who can effortlessly dig their roots into the pool of expertise and experience that lies outside the organization but within their domain of excellence. When an SME comes back from attending a trade show or seminar they can mutate the ideas to suit the organization and spread them throughout the organization via the interdepartmental CoP structure.

Just achieving this stage will provide a significant competitive advantage and add decades of life-expectancy.

Scenario-Planning

So far we have dealt with the past and the present, and the next evolutionary phase is to consider the future beyond the next departmental quarterly review. Scenario-planning is a toolset that attempts to break at least partially free from the learned helplessness and practiced defensiveness that Chris Argyris outlines as part of “Single-Loop Learning“. By posing “what if” scenarios, there is the possibility, if you are nimble, to catch yourself before the auto-protective blinds come down and to notice the stealthy approach of a hidden predator, or surprise yourself with an outcome that was unexpected.

This is the playground of the giants mainly, because everyone else is too busy “just surviving” to look several years down the pike and try to make out the fuzzy shapes on the horizon or in the shadows. The irony is that it can lead to complacency (look at BP and the recent gulf of Mexico debacle), in the same way that seatbelts and airbags led to less careful driving in some people.

Scenario-planning requires a mix of dogged fact-finding and logical step-wise thinking, systems-thinking, and imaginative brainstorming. Plenty of DIY books exist on the topic, but usually a firm needs external help at least in the beginning. It also requires a mix of culture and technique that is frankly beyond most firms. After producing various scenarios and plotting the likely outcomes, and then working back to find solutions, it requires a very peculiar kind of management culture to stare the scenarios in the face and put money and executive sponsorship behind remedial action.

Although this is a critical component of achieving and maintaining longevity, its very success is a risk, since dodging future bullets makes a firm more likely to become complacent and also to value the process less. People in westernized countries are less likely have their children immunized because they have forgotten or have never experienced the real diseases – dodging them makes them seem less like the killers they are.

Ongoing Professional Development

Another dimension in successfully competing is simply having better skills and intellectual assets than your competitors. This runs the gamut of identifying people with better SKAs than your competition, to acquiring and keeping them, to putting them to work more efficiently and effectively than the next company in your market space. However, time passes, things change, tools rust, and if you want to keep ahead of the competition, having a workforce composed of people who actively pursue their own ongoing professional development is surely the best.

This is also the key element in forming a CoP, and without a culture of ongoing learning, the intellectual assets of a company will slowly gather dust and be buried.
The absence of a vibrant and concerted effort to maintain professional expertise is an early sign of cognitive degeneration in a firm, and a harbinger of senescence. If your staff don’t actively pursue their own ongoing professional development, you are already a dead-man walking.

Innovation Intent

The final dimension is the desire for change, and perhaps the hardest of all to achieve.

As companies age, like people, they tend to grow more conservative in outlook and more comfortable with the tried and true over the new and exciting.

This is a perfectly logical risk-aversive approach since most novelty, most innovation either fails or is deleterious. Mutations, for example, seldom produce an improvement – usually they just result in cancer. So sticking to what has already proven to work adequately is a very safe bet – in the short term.

However, this leads inevitably to rigidity in the face of change and decreased ability to formulate new solutions when the old ones no longer apply. Think of this in terms of bacteria – over time bacteria will acquire resistance to existing medications no matter how effective they were originally, and unless novel attacks are discovered, eventually the bacterium starts gaining ground and flourishes.
For this reason one has to have a deliberate intent to innovate, to test out new approaches and ideas before the old ones are exhausted and overrun.

However, this requires a cultural environment in which experimentation is supported, controlled, and encouraged. An early warning sign is if mistakes are typically punished rather than treated as learning opportunities – If punishment is the first and foremost reaction, then you have a safe bet that there is little innovation and the firm is already gathering moss and accumulating risk.

A word of caution is appropriate here – Major innovations don’t typically come from individual work, nor from steady evolutionary refinement over time, but from importing mature ideas from other domains and collaboration between people and across domains and organisations.
If individual work is rewarded and there is a winner-take-all culture, you already have a massive handicap.

Conclusion

Studying the causes of death in firms serves two valuable purposes – knowing the facts of death itself, and the formation of a classification on which to build remedial efforts. This provides a framework against which to take preventative and generative action, and with careful action, a firm can greatly extend its productive lifespan.

Most of the steps require a cultural component, and all require leadership and executive support that can look beyond the next quarterly earnings. But for those companies that have the character and desire, the processes listed can provide not just a new lease on life, but significant competitive advantage.

~~~~~~~~~

Matthew Loxton is a Knowledge Management professional and holds a Master’s degree in Knowledge Management from the University of Canberra. Mr. Loxton has extensive international experience and is currently available as a Knowledge Management consultant or as a permanent employee at an organization that wishes to put knowledge to work.

Externalization and Avoidance – How we ignore and avoid facing our own faults

September 22, 2010

Externalization and avoidance are key manifestations of how we ignore and avoid facing our own faults and project them onto others, and has ramifications in how organizational learning does (or doesn’t) take place.

The Problem

In a journal article about experiences with professional business-consultants, Chris Argyris describes a senior manager asking a group of business consultants what problems they encountered and what changes they could make to their practices to improve service. In response, several of the consultants explained what the customers could do to improve consultant’s access to information or client acceptance of advice. The manager tried several more times to solicit ideas on changes the individuals could make to their consulting practices, but was again given details on things customers and others could do.
The consultants seemed oblivious to how painstakingly they avoided discussing their own practices.

“As long as efforts at learning and change focused on external organizational factors—job redesign, compensation programs, performance review, and leadership training—the professionals were enthusiastic participants. Indeed, creating new systems and structures was precisely the kind of challenge that well-educated, highly motivated professionals thrived on.
And yet the moment the quest for continuous improvement turned to the professionals’ own performance, something went wrong.”
(Argyris 1991)

This externalization behavior is explained by the concept of “Single-Loop Learning” or “Model-1” behavior and defenses people employ to objections or errors in Model-I actions operating under single-loop learning processes, in which they “create defensiveness, self-fulfilling prophecies, self-sealing processes, and escalating error” (Argyris
2000)

Not only are the errors unmentionable, but the very unmentionability is unmentionable also, hence “self-sealing”. (Argyris 1999)

Argyris poses this as a defense mechanism against embarrassment and feelings of threat which is deeply entrenched, finely practiced, and in which the individual is highly skilled to the degree that the mechanisms are automatic, instantaneous, and spontaneous. (Argyris
1990)

When people are challenged about these self-sealing actions or an unmentionable is mentioned, they “become defensive , screen out criticism, and put the ‘blame’ on anyone and everyone but themselves” (Argyris 1999), and thus inclined to scapegoat rather than engage in either self-reflection or analysis of the “tried and true” methods of received wisdom.

An example of this in practice is a discussion on LinkedIn by self-declared HR professionals debating what they “really hate in a CV“.

Rather than the more pertinent question of how they should go about identifying and acquiring better human assets than their competition, they found fault with external actors and produced a long shopping-list of what applicants should be doing differently rather than what they should be doing to address the core challenge of obtaining better assets. Many framed the issue in terms of how the applicants should “sell” themselves, or in what self-promotion the applicants should engage – and thereby ignore the obvious question of what recruiters should do to ensure that they procure the best candidates in spite of obstacles.

If asked why they are critiquing the resume-writing skills of applicants rather than trying to change their own practices to increase hiring quality, they again turn the discussion to failings of the applicants and frame the discussion in terms of the shortcomings of applicant’s self-promotional techniques.

Further probing or argument about why they choose to focus on resume-writing errors rather than acquisition problems, leads predictably to complaints of rudeness, general defensiveness, and further explication of what applicants should do to be more acceptable to recruiters.

In short, ego-defenses.

What to do?

Argyris argues in his early works in much the same vein as Freud – that making people aware of their defensive mechanisms will lead to mastery over them and resolution – self-revelation as a therapeutic bulwark against systematic Model-1 behavior.

Unfortunately this has not proven to be a very successful approach, and as I discuss in an earlier blog entitled “Dealing With Failure”, escape is not at all easy since these are both deeply ingrained and perhaps inborn defensive mechanisms.

I propose three main approaches for mitigation

  1. Engineer the human out wherever possible
  2. Place procedural traps to trigger corrective actions
  3. Copy aspects of the scientific method of critical tests

Engineering

This is an old trick that comes from safety & quality practices, and involves building processes and procedures that simply eliminate the human in situations where they are prone to error, or engineering the risk downwards by providing guards, interlocks, and other devices for keeping fingers away from blades, and eyes away from flying debris. Build processes that reduce the opportunity for human error.

Traps & Triggers

Stuff happens, and despite all your engineering, people will still have cognitive biases and be prone to avoid noticing them – so build traps in the procedures that will catch them in the act and trigger self-correcting processes. With all the will in the world, you cannot wish away your biases, but you can plan ahead by setting traps for yourself.

Critical Tests

Science has had the principle of critical tests for some 400 years now, perhaps the concept should be adopted in more areas of human activity. If you believe that your skills at sizing up character traits in job interviews gets you good employees, then you must record what you thought at the time and test against actual outcomes. It simply isn’t enough to believe it to be true – you must follow up with measurements that could theoretically prove you wrong. You must also test the converse; you must sample some of those candidates whom you rejected to see what became of them. If you don’t, you could well be systematically excluding the best candidates and would never know. If the people you rejected went on to become Nobel Laureates, CEOs, and celebrated practitioners of their craft, then perhaps you are biased in how you reject candidates – but if you don’t sample your rejects you would never know.

Conclusion

Simple discussion is incapable of penetrating the protective layers and well-rehearsed defenses of Model-1 behaviors, and to avoid embarrassment people will defend positions by use of well-oiled and very subtle mechanisms such as reframing the issue to externalize its causes. However, protecting processes by engineering out human biases, and exposing behavior to traps and triggers and subjecting it to critical tests, can significantly lower the risk of falling prey to the many cognitive and behavioral biases to which humans are prone.

Please contribute to my self-knowledge and take this 1-minute survey that tells me what my blog tells you about me. – Completely anonymous.

Bibliography

Argyris, C. (1990). Overcoming Organizational Defenses, Prentice Hall.

Argyris, C. (1991). “Teaching Smart People How to Learn.” Harvard Business Review
69(3): 99-109.

Argyris, C. (1999). On Organizational Learning, Blackwell.

Argyris, C. (2000). Flawed Advice and the Management Trap, Oxford Press.

~~~

Matthew Loxton is a Knowledge Management professional and holds a Master’s degree in Knowledge Management from the University of Canberra. Mr. Loxton has extensive international experience and is currently available as a Knowledge Management consultant or as a permanent employee at an organization that wishes to put knowledge to work.

Aspects of Questionnaire Design

August 19, 2010

I have blogged before about using executive presentations and other artifacts as part of organization-wide organizational learning (see below), and in this blog I will cover some aspects of questionnaire design.

I will assume that the reader either already knows or will research the basic questionnaire-design constructs like having clear agent of action, avoiding double-barreled questions, speaking in the active voice, and the like.

Why Create a Quiz?

The objective of a quiz is to perplex and mystify the reader, or so you have been led to believe over the years of taking them yourself and being mystified at the purpose of some questions that seem to delve the depths of irrelevancy, and perplexed by the minutes of your life ebbing away as you answer them.

However, this is not at all the purpose of setting questions.

Setting questions achieves the following:

  • A second shot at highlighting to the audience what you regard as the important take-away points. You should set questions only on the things you want the reader to know are the most important bits – by posing a question you are saying to the audience “…and this is the important part
  • Finding out if there is something inside your material that is systematically misleading the audience. If significant numbers of people get the wrong answer, then you have misled them someplace and you need to fix that.
  • Finding out if there is a bias of some kind in the audience population. If only one department, or a specific age-group, or only people over six feet tall get certain answers wrong, or pick a specific incorrect answer from a list, then something is going on that you need to look into – which is probably something you told them previously.
  • Finding out if what you said made a lick of sense.
  • Discovering if the person felt confident about their answer or not

Of course this gets a tiny bit more complicated, but then that is why you are in this business – you like complicated things.


Which bring us to How.

Rules

  1. Only ask questions that test understanding on something you regard as a vital point – don’t waste your time and theirs on setting questions on irrelevant material.

  2. Never offer frivolous alternatives in a multiple-choice question, each alternative should be something the person is likely to pick due to a misunderstanding that you have already discovered.

  3. If in doubt, leave it out.

  4. Test and retest before launching.

  5. You need the SME to be involved in building a questionnaire because only they can know which questions are significant, and which answers are valid.

How to Create a Quiz

I have a book on my shelf that is written by the guru on questionnaire design, A.N. Oppenheim (Oppenheim1998) and one of the few books exclusively focused on the topic of designing questions. The preface to the 2nd edition starts off with the following:

The world is full of well-meaning people who believe that anyone who can write plain English and has a modicum of common sense can produce a good questionnaire. This book is not for them”

The basic drift is that it isn’t that simple to construct a good questionnaire, and boy, isn’t it in spades!

Ask a bad question, and you will get nonsensical answers and be left wondering what the audience thought you meant.
You will also have wasted your chance, and have wasted the respondent’s time – for which there is no excuse whatsoever.

There are plenty of texts (such as published by O’Reilly) dealing with the technical side of questionnaire tools both SaaS ( Survey Monkey, etc.) or embedded within Learning Management ( Moodle, WebCT, etc.), Trouble-Ticketing ( Remedy, OTRS, etc.), and other suites.
But that’s the easy part, albeit the part with the thickest manuals.

What I am going to cover here is the more tricky part of how to build the dialogue involved in asking questions in an eLearning context.
You cannot see the puzzled look on your respondent’s face in an eLearning situation, so you will have to plan for it when you design your questions.

Step 1 – Critical Elements

Identify the critical concepts or facts that you want the audience to understand and retain, jot these down.
If you get past 15 or 20, consider breaking your course into more than one part – a tutorial with more than a dozen critical points is starting to get really big, and unlikely to stick. Five is a good number, try to keep it that focused.
Keep it tight, keep it light, and rather build more tightly-focused courses than trying to solve the world’s problems in one fell swoop.

Step 2 – “By George She got it”

For each question, consider what supportive information you can give for a correct answer.
You are getting another shot at contextualizing and once more to drive a point home, don’t waste it.
You should present the respondent with a text of your choice and you should conform to the dialectic form of “
yes, and …”.
Affirm the correct answer and then provide the context of why that answer is right, and drive the point home a little deeper.

Step 3 – “um… no, because …”

For each incorrect answer you provide or which might occur (you will enable them to pick a wrong answer, right?), you need to furnish targeted corrective information.
Try to present wrong alternatives not to confuse, but to identify what you think are common mistakes or potential mistakes you want to address, so that once more you can drive your point home and provide a context.
The idea is to provide them with enough information (including referring or linking to other sources), so that you get the issue cleared up in their mind before they move on.

Step 4 – Concluding Summary

Many questionnaire tools will give you the option (which you will naturally take) to provide a feedback statement after they have finished answering it.

This is once again, an opportunity to provide additional context or remind them of the facts.
It allows you to place the question and answer in perspective in the broader picture, and provide the respondent with an additional link in why this is important and how to picture it.

Step 5 – Confidence

There is a big difference between getting something wrong when you are taking your best guess and being wrong and simultaneously being very confident about your answer, and it is very useful to know which is the case.
Consider constructing your questionnaire to add a rider to each question to measure how confident the respondent is – a simple five-point
Likert Scale should be fine.

Conclusion

By now you can see another reason why asking irrelevant questions is a waste of effort – for each question you need a comment for a right answer, comments for wrong answers, and a comment to put the whole question into a meaningful perspective.
A whole bunch of work that you only want to do if the question is worth the effort.

Remember, you are in an asynchronous dialogue with the respondent, and the objective is to pass on not just facts, not merely information, but knowledge – and you can only do that by also providing perspective and context.

Good Luck!

Some previous posts on Organizational Learning:

“How to get added value from corporate presentations”

“Knowledge Transfer”

“Niche Mastery – How KM can add a few hundred million dollars to corporate worth”

“The Corporate College is dead, long live the Corporate College!”

“Corporate blogging and web2.0 – training wheels first

Please contribute to my self-knowledge and take this 1-minute survey that tells me what my blog tells you about me. – Completely anonymous.

Bibliography

[Oppenheim1998] Oppenheim A. Questionnaire design, iterviewing and attitude measurement. . Pinter, 1998.

~~~

Matthew Loxton is a Knowledge Management professional and holds a Master’s degree in Knowledge Management from the University of Canberra. Mr. Loxton has extensive international experience and is currently available as a Knowledge Management consultant or as a permanent employee at an organization that wishes to put knowledge to work.


%d bloggers like this: