A New Modeling Framework for Organizational Self-assessment: Development and Application

October 2001
Volume 8 • Number 4

Contents

A New Modeling Framework for Organizational Self-assessment: Development and Application

by C. H. R. Siow, J. B. Yang, And B. G. Dale, University of Manchester Institute of Science and Technology

This paper reports the development of a multiple-attribute decision-making (MADM) modeling framework and methodology for application to the scoring process against the criteria of the European Foundation for Quality Management (EFQM) Model for Excellence. The MADM model is based on the evidential reasoning approach with the focus of the initial development and application on the people management enabler. The model has been tested with real data from two award application documents from an electricity distribution utility. The scores achieved using the methodology have proven the validity and reliability of the model.

Key words: EFQM Excellence Model, multiple-attribute decision making, self-assessment process

INTRODUCTION

Ahire and Rana (1995) and Hwang and Yoon (1981) outline how a multiple-attribute decision-making (MADM) model allows the decision-maker to evaluate, along common multiple criteria, various competing alternative courses of action to achieve a pre-specified goal. These and other writers describe how MADM methods are technical decision aids for evaluating alternatives that are characterized by multiple qualitative (or fuzzy) and quantitative attributes. Eom (1989) also points out that multiple-attribute, decision-making problems with these types of attributes are commonplace in business practice. Self-assessment against the European Foundation for Quality Management (EFQM) Excellence Model, as typically outlined by Hakes (2000), can be considered to be part of the general class of organizational evaluation problem. Hwang and Yoon (1981) argue that all problems share the following characteristics.

  • Each problem has multiple objectives/attributes and the decision-maker must generate relevant attributes for problem setting.
  • Multiple criteria usually conflict with each other.
  • Each attribute may have a different unit of measurement.

These characteristics are found in the self-assessment process against the criteria of the EFQM model. Therefore, it is possible to apply the MADM modeling logic to develop a methodology for assisting the scoring process against the criteria of the EFQM model. Ahire and Rana (1995) determined that a MADM model application to a decision problem should:

  • Identify the decision hierarchy in terms of criteria, subcriteria, and sub-subcriteria.
  • Determine the relative importance of weights of these elements of hierarchy.
  • Assess and rank various alternatives using these criteria.

The first step in applying the MADM logic to a problem is to evaluate and quantify the state of an attribute. This approach may be conceptually clear and easy to understand; however, Yang and Singh (1994) and Yang and Sen (1994) demonstrated the types of difficulties that can be encountered in this process. The main difficulty is the requirement for attributes to be potentially independent and quantitative. Measurements of qualitative attributes are highly subjective and imprecise and, as such, it is more effective to evaluate them using subjective judgments with uncertainty. This is typical of the problem under consideration in this paper—self-assessment against the criteria of the EFQM Excellence Model—in which the process of scoring is influenced by the following factors.

  • Assessors’ background and prejudices
  • How they interpret an organization’s approach
  • Deployment, assessment, and review
  • Assessors’ level of understanding of the TQM/excellence/continuous improvement philosophy and the subcriteria under consideration
  • How they interpret the details in an organization’s submission document

Consequently, there is some degree of subjectivity and uncertainty in the scoring activity and, as a result, the process is not scientifically robust with respect to numerical evaluation. The evidential reasoning approach (Yang and Singh 1994; Yang and Sen 1994; Yang, forthcoming) provides an alternative way to deal with such uncertain synthesis problems by means of evidence combination for multiple attributes.

To date, academic researchers have ignored the application of MADM methods to the self-assessment process against the criteria of a quality/excellence award. This may be due to the recent rise in popularity of self-assessment and the technicality of the MADM methods that tend to favor scientific decisions and with which few quality management researchers are familiar. The authors’ review of MADM methods has shown their potential for application to the self-assessment environment; this also established that the evidential reasoning approach was the most appropriate. This paper reports the development of a MADM modeling framework and methodology to assist in the scoring of the subcriteria of the EFQM model. The focus of the initial development and application has been the people management enabler of the model.

EVALUATION FRAMEWORK

In a MADM model, criteria can be described on the basis of attributes. These attributes are broken down into their simplest form in such a manner that they can be readily and easily evaluated by means of either numerical values or subjective judgments with a level of uncertainty. By using the evidential reasoning approach, the evaluations of these basic attributes can then be aggregated into an overall evaluation of the people management criteria (with respect to the 1998 version of the EFQM model). Yang and Sen (1994) developed a new, general, multilevel, and distributed evaluation process based on an evidential reasoning framework, where uncertain subjective judgments for the evaluation of qualitative attributes can be accommodated. With this new approach, qualitative attributes may be evaluated by uncertain subjective judgments through multiple levels of factors, and each of the judgments may be assigned by single or multiple assessors in a rational way within the evidential reasoning framework. This approach has recently been enhanced so that both qualitative and quantitative attributes can be dealt with under the same framework (Yang, forthcoming), and has been integrated in a newly developed Windows-based software package called Intelligent Decision System (IDS) (Yang and Xu 2000). The process of constructing the MADM model, methodology, and evaluation will be based on this platform.

Undertaking a self-assessment against the EFQM Excellence Model is based on the transformation of original data into equivalent subjective statements with uncertainty. As precise numerical values are not usually available for the enablers (that is, leadership, policy and strategy, resources, people management, and processes) of the model, it is then better to articulate subjective judgments with uncertainty as original evaluation data, as detailed by Yang and Sen (1997). Furthermore the scoring of such qualitative issues will generate high uncertainty during evaluation (Van der Wiele et al. 1995). Therefore, the evidential reasoning approach is the ideal tool to handle this situation.

With the MADM approach an overall set of evaluation grades is first defined, and each of the base-level attributes is judged relative to these grades, which may range from excellent to very poor, or from fundamental to minor. According to Yang and Sen (1997) the MADM approach works on the assumption that “if the evaluation of a basic attribute is, to a certain extent, judged to be good, then the evaluation of its associated upper-level attribute will, to some degree, also be good.” The evidential reasoning approach provides a systematic way of synthesizing such uncertain evaluations of basic attributes to produce an evaluation for an associated upper-level attribute.

The following are some of the points on which the construction of the evaluation framework has been based.

  • Attributes on the same level should be of similar importance. If they are not, then too much credence can be given to any one of them.
  • A decision must be made as to when to stop the breakdown of attributes. The attribute can be broken down to an infinite number of levels but there is a stage where further breakdown will not provide increased accuracy. On the other hand, too little breakdown can also result in an evaluation that is too subjective and inaccurate.
  • The framework must be an accurate representative of the people management criteria being modeled.

The EFQM Excellence Model (1998) consists of five enablers and four results criteria broken down into 32 subcriterion parts (for example, the people management criterion has six subcriteria). This structure has been employed as the guideline for the construction of the evaluation framework. The people management criteria and its six subcriterion parts are referred to as level 0 and level 1 attributes, respectively. The level 1 attributes can be further broken down into the level 2 attributes of areas to address, which are detailed in the evidence for each subcriteria. An example of the detailed breakdown of each attribute of the people management criteria is given in Table 1, and their respective descriptions can be found in EFQM (1998). A key decision is whether or not to carry out a more detailed breakdown of these attributes, requiring a clear understanding of the attributes and their underlying factors. If the attributes are broken down beyond level 2, researchers can produce a different framework for the evaluation of the same enabler and, as a result, the benefit of the generic MADM model is lost.

The subcriterion 3(a)—how people resources are planned and improved—is used as an example to illustrate how the framework has been formulated. For this level 1 attribute, the EFQM model breaks this down to the following five (level 2) attributes; in self-assessment terms these are the areas that need to be addressed in collecting evidence in support of the 3(a) subcriteria.

  1. How the organization aligns the human resources plan with policy and strategy
  2. How the organization develops and uses employee satisfaction surveys
  3. How the organization ensures fairness in terms of employment
  4. How the organization aligns its remuneration, redeployment, redundancy, and other terms of employment with policy and strategy
  5. How the organization uses innovative work organization strategies and methods to improve the way of working


Each of these level 2 attributes is further broken down into two level 3 attributes—approach and deployment. These are defined in the EFQM (1998) as follows:

  • “The enabler criteria are concerned with how the organization approaches each of the criterion parts.
  • “Information is required on the excellence of the approach used and the extent of deployment of the approach—vertically through all levels of the organization and horizontally to all areas and activities.”

EVALUATION GRADES AND DEFINITIONS

The MADM model requires a generalized set of evaluation grades before an assessment can be undertaken. Central to decision theory and to the evaluation of grades in the MADM model is the concept of value. This is the measure of what is good or desirable about a design or its attributes (Siddall 1972) and demonstrates that the desirability of each outcome can be decided by attaching to it some value. For example, when assessing a quantitative attribute such as: the “number of team briefings conducted annually,” there is little chance of evaluating whether the reported result (for example, 24 times) per year is good or bad. This indicates the need for a range of values to evaluate the performance of an attribute. According to Siddall (1972) utility value can be described as that relating to the function or usefulness of an attribute. Using the example of team briefing assessments, here is a range of values from 0–24, with 24 having a utility value of 1 (the best), 0 as having a value of 0 (the worst), and other values having utilities between 1 and 0.

While the same guiding principles apply to qualitative judgments, there are some differences. The judgments given by assessors must also relate to utility. The best judgment possible is related to the utility value of 1 and the worst to 0. For example, if a five-point evaluation scale is chosen and calibrated starting with the endpoints, then 1 point may be given to the maximum value that is practically or physically realizable and 0 point to the minimum value. The midpoint (0.5) will also be a basis for calibration, since it will be the breakpoint between values that are favorable (or better than average) and values that are unfavorable (or worse than average).

With reference to the examples of qualitative and quantitative judgments, the conventional procedures that derive these numerical values use addition and multiplication across the attributes. This type of scaling assumes that a value of 0.75 is three times as favorable as a value of 0.25. The combination of values across attributes implies that the difference between any two specific values is the same for each attribute. It is obvious that the assignment of such values is arbitrary. Many scales are possible, and it is important to provide some consistency checks or definitions in these scales. These checks/definitions are desirable but they make the scaling procedure a tedious activity, posing many questions to the assessor.

The evaluation grades and definitions have been formed in three stages (EFQM 1998; Porter and Tanner 1998).

  1. Each level 2 attribute is assessed according to a group of definitions or checklist items.
  2. The scale for the evaluation grades is established. Five grades were selected—world-class, award winners, improvers, drifters, and uncommitted.
  3. Each checklist item is then defined in terms of the five grades, using the scoring matrix approach.

Checklists for Each Qualitative Attribute

To provide some consistency and definition in the evaluation scales, a checklist of items is used to evaluate each level 2 attribute in the people management enabler. These items have been formed from the work by Godfrey et al. (1998) and various checklists acquired from management consultancies. An example for the criteria “How the organization aligns the human resources with policy and strategy” is shown in Table 2. These lists of items can be further developed from the documents produced by award-winning companies (for example, BT United Kingdom 1997; TNT United Kingdom 1998).

The checklist enables the assessors to have a clear idea of what to look for in the evaluation of each attribute. The checklist also serves as a reference to enable the assessors to make reliable and accurate judgments and thereby minimize the subjectivity of the assessment process. The checklist can be modified to suit an organization’s operating environment; however, changes should only be done as a last resort when there are too many “un-assessable” items. It is recommended that during assessment, “un-assessable” items should not be graded and taken into consideration during the final assessment.

Evaluation Grades Scale

Dale and Lascelles (1997) have identified six different levels of characteristics and behaviors—world-class, award winners, improvers, tool pushers, drifters, and uncommitted—in relation to the adoption of total quality management (TQM). These give definition to the evaluation grades and show hierarchical differences between the levels. This is required as there is a hierarchical difference in the utility value of each evaluation grade in the MADM model: The best judgment possible is related to the numerical value of 1 and the worst to 0. To prevent the evaluation scale from becoming too complex and difficult to differentiate between the levels, the six levels of TQM adoption have been reduced to five levels; based on the advice of Dale and Smith (1998). The tool pushers and drifters’ levels have been combined. The scale of “world-class” to “uncommitted” is used to describe the evaluation grades. Details follow:

The general scale of evaluation grades, H, is defined as

H = {H1 H2 H3 H4 H5}

{World-class, Award winners, Improvers, Drifters, Uncommitted}

P{H} = {1 0.75 0.50 0.25 0}

The description of each grade level is taken from Dale and Lascelles (1997) and Dale and Smith (1998).

Evaluation Grades Definitions

During the evaluation of complex qualitative attributes such as people management, there are no mathematical formulas, and any attribute can be assigned to the general set of grades without validity. To think of an attribute merely in terms of “world-class” to “uncommitted,” places an undue burden on the assessor to translate judgments directly into single-word evaluations. A better approach is to define each evaluation grade with more precision of what a world-class attribute can achieve and, on the other hand, what an uncommitted organization cannot achieve.

In this MADM model, all the level 2 attributes have been defined in terms of the checklist of items, with each item in turn defined in evaluation grades from H1 to H5. For example, considering the attribute—how the organization aligns the human resources with policy and strategy—a world-class grade definition for each of the checklist items is as follows:

  • Item 1. Comprehensive evidence of the total integration of human resources management (HRM) and quality management as part of the same initiative and moving toward a common goal of excellence
  • Item 2. Comprehensive evidence of the human resources (HR) function having an influence in the development of business policy and strategy
  • Item 3. Comprehensive evidence of an HR strategy incorporating plans for selection, training, and succession; clear evidence of regular review of these plans leading to improved business effectiveness

On the other hand, the uncommitted evaluation definition of this attribute is the reversal of world-class.

  • Item 1. No evidence of the integration of HRM and quality management
  • Item 2. No evidence of the HR function having an influence or input in the development of business policy and strategy
  • Item 3. No evidence of an HR strategy incorporating plans or selection, training, and succession; also no evidence of assessment and review

The definition of each grade takes into account the guidance specified by the EFQM (1998) in terms of approach and deployment.

Based on EFQM (1998) and Porter and Tanner (1998), Table 3 shows the scoring matrix that is used to evaluate the approach, differentiating between the different evaluation grades.

The attribute is concerned with the extent to which the approach has been implemented, taking into account its full potential and the appropriateness and effective application of the approach. The following are used to define the different evaluation grades of H1H5.

  • World-class—Complete and full integration into normal operations
  • Award winner—Well integrated but not totally integrated into normal operations
  • Improvers—Average levels of integration
  • Drifters—Some areas of integration
  • Uncommitted—No integration

Weightings

To design an accurate MADM model the weightings of each attribute have to be considered as they allow the alteration of the composition of attributes evaluation in favor of certain subattributes. Without weightings, all attributes are considered equal in making up the overall evaluation of the higher-level attributes. All the attributes in this evaluation framework are given the same weightings as the EFQM model (that is, in the people management criteria there are six subcriteria). The 9 percent or 90 points weighting for the people management enabler in the EFQM model will only be considered when all the enablers are combined.

There is a need to have a clear understanding of what is important when assessing people management to effectively determine the weightings for each item on the checklist. The total weightings given to all the checklist items for one parent attribute should be no more than 100 percent. The weightings given are subjective since they are based on presumption, understanding, and experience. Hence, the relative importance of each checklist item will vary, depending on the researcher and organization. To reflect its own culture, operations, and processes it is possible for an organization to change the weightings for the more important items.

COLLECTION AND EXTRACTION OF TEST DATA FOR EVIDENCE

The data used for testing this model were taken from the 1997 and 1998 EFQM model application documents of an electricity distribution utility. Unlike other MADM projects, such as design selection where the information obtained is in no particular format, hence making the extraction of data a difficult task, this MADM model has been designed in line with the 1998 EFQM Excellence Model. The evidence for the assessment of a particular criterion part will be in the same format, and the assessor will just have to classify this evidence with respect to the defined grades. There are no set rules on the usage of data, and it is possible that data from one criterion part in the application document can be used as evidence for assessing a different criterion part.

The framework has been tested by evaluating the data to the set of evaluation grades. While it was easy to extract the relevant data from the two application documents, this was not the case in mapping the extracted data to the set of evaluation grades. For example, when assessing the attribute—evidence of a systematic approach to succession plans—the following data were presented in the 1997 application document.

The appraisal process helps to identify people for succession planning. By assessing appraisal notes and their own personal knowledge the utility identified potential senior managers and specific training plans were developed to enhance their careers. With the introduction of the new organization in October 1996 this planning enabled the utility to appoint staff of a suitable calibre into all new posts. As a result of SVS [voluntary severance scheme] the age profile of the company changed and so a new project is under way to create a comprehensive succession planning process.

The wording and terms used in the document do not always coincide with the definition and meaning of the evaluation grades as given.

  • World-class—Comprehensive evidence of a systematic approach to succession plans. Clear evidence that these plans are being regularly reviewed, leading to improved business effectiveness.
  • Award winners—Extensive evidence of a systematic approach to succession plans. Clear evidence that these plans are being regularly reviewed and refined.
  • Improvers—Evidence of a systematic approach to succession plans. Clear evidence that these plans are being regularly reviewed.
  • Drifters—Little evidence of a systematic approach to succession plans. Clear evidence that these plans are being occasionally reviewed.
  • Uncommitted—No evidence of a systematic approach to succession plans. Any plans are not reviewed.

The assessor needs to interpret this extracted evidence and categorize it into one or more of the five grades. For such qualitative evaluation as people management, precise numerical values are not available, and in this process of evaluation an item’s grade can lie between “world-class” and “award winner.” Using subjective judgment the assessor can assign degrees of confidence or belief to these grades. If it is felt that the item lies closer to the world-class definition, then the evaluation may become world-class (0.8), award winner (0.2), indicating a degree of confidence/belief in the evaluation. The total degree of belief for one grade could be less than 100 percent but should not be more than 100 percent. This is one of the main advantages of the evidential reasoning approach. The degrees of belief are multiplied by the weighting to produce an assessment for each of the level 2 attributes. These assessments can be aggregated, and the final assessment for the people management enabler can be generated using the IDS package.

An assessment can be characterized using either a distribution in terms of degrees of belief or expected utilities (scores). There are three possible utilities—worst possible, average, and best possible—and these are obtained by taking into consideration the missing evidence. As a result of missing evidence, some degree of belief may be left ungraded, and could be assigned to any of the grades. The IDS software uses the theory of evidence to analyze these uncertainties by taking account of two extreme scenarios. One scenario is that the missing evidence could support the worst grade, uncommitted; and the other is that the missing evidence could support the best grade, world-class. This leads to the generation of two utilities—worst possible and best possible—between which a utility interval is comprised. This allows the organization to assess where it stands in terms of the worst and best possible scenario, without having total certainty of the assessment. The average result is obtained by giving no preference to any of the grades. This is done by assigning equally ungraded degrees of belief to the missing evidence.

During an evaluation there are some checklist items that cannot be fully evaluated and assessed with confidence without a site visit. A site visit is an opportunity to talk with people in the organization and to confirm the validity of the data and clarify aspects that are unclear. If in the evaluation of an attribute the evidence justifies a degree of belief of 50 percent, but this evidence is questionable because it is not clear how widely the attribute is deployed within the organization, then as a result, the remaining 50 percent degree of belief is left ungraded, resulting in a total degree of belief for that attribute of less than 1. If a site visit can make clear the scope of deployment, the remaining degree of belief may be assigned to individual grades.

There may be a case when there is no wish to conduct a site visit to verify the evidence. In this situation it is possible for the assessor to implant some personal judgments into the evaluation process. Here, the assessor’s perception in terms of inference and deduction for the missing evidence is based on his/her belief of how well the organization can possibly perform in that attribute. This method of assessing missing evidence is subjective but useful if a site visit is not possible.

TESTING THE MODEL

The model has been tested using the utilities’ two application documents. Analysis of the 1997 document reveals that the average overall utility value obtained for the assessment is 0.366 or 36.6 percent and is midway between the drifters and improvers levels of TQM adoption (Siow 1999). Figure 1 provides a graphical representation of this type of analysis.

The highest score for the level 1 subattribute (3f)—How people are cared for—is 45.8 percent (Figure 2). This score is obtained from its child level 2 attributes scores, which have a highest average score of 62 percent and a lowest average score of 31 percent (Yang, Dale, and Siow, forthcoming). The lowest score of the level 1 attribute (3a)—How people resources are planned and improved—is 27.7 percent, which is obtained considering all its level 2 attributes (Siow 1999). The highest score among these level 2 attributes is 45.4 percent, and the lowest score is 14 percent. As explained by Yang, Dale, and Siow (forthcoming) the overall score of 36.6 percent is within the range of the lowest and highest score of each lower-level attribute, indicating that the mathematical process for this model is accurate and the score of 36.6 percent is valid.

The MADM model can also help identify strengths and areas for improvement (areas for improvement) by analysis of the appropriate attributes. This can be done in two ways.

  1. By examining the lowest or highest score of each level’s attribute through the framework to the base level (hierarchical framework method).
  2. By identifying all the subattributes that score higher (strengths) or lower (areas for improvement) than the overall score (relative score method).

The hierarchical framework method is less accurate than the relative score method. With the former method only those attributes that are in most need of improvement and those in which the organization has done well are identified. By following the highest score of each attribute through the framework to the base level attribute, the assessor can identify the checklist items that have scored high, along with the evidence and comments made (see Figure 3). The areas for improvement are identified in the same manner by following the lowest score of each attribute through the framework to the base level attribute to identify which checklist item received the lowest score.

The hierarchical framework method has two weaknesses. Firstly, the assessor is only able to identify areas of attention through its parent attribute. That means that if the parent attribute is scored low or high then more investigation is needed on the respective child attributes. However, there may be cases where the parent attribute has been scored high but one of the child attributes scored low. This means the low-scoring subattribute will be disguised among a group of high-scoring subattributes. Secondly, by focusing on the level 1 attribute with the lowest or highest score, this method allows the remainder of the level 1 attributes to be left out of the identification process and, as a result, some areas that need attention may not be identified. Having made these points it must be said that the more focused hierarchical framework method allows a quick assessment to be made.

This problem can be minimized by modifying the method so that it starts from a level 1 rather than a level 0 attribute. This allows an examination of every level 1 attribute that gives the assessor a wider view of the scored document. This method can be rather tedious and is only recommended if a very detailed feedback report is required.

The relative score method uses the relative score (overall score) of the level 0 attribute to identify strengths and areas for improvement. It allows all the attributes of the same level (usually the lower- or base-level attribute) to be compared to a relative score with those attributes with a higher or lower score identified for further investigation. To identify the strengths from the scoring of the 1997 application document, all the level 2 attributes with scores higher than 36.6 percent are considered to be strengths. It is also possible to examine in this way all the level 3 attributes (that is, approach and deployment); however, it is felt that the findings could be too detailed. Areas for improvement are located by identifying all attributes that have scored lower than 36.6 percent.

It is clear that the hierarchical framework and relative score methods have advantages and disadvantages. The former method is able to establish a list of items linked to a parent attribute, emphasizing the weakest or strongest parent attribute; however, it suffers from being too narrow in its scope of investigation. On the other hand, the relative scoring method is more elaborate, taking into account all criterion parts and is also more flexible as any strengths and weaknesses above or below a specified score are considered. This will lead to a large number of areas identified, which, dependent on the assessor’s objectives, can be minimized by concentrating on a smaller number. Building on their respective advantages the two methods can be used together to produce a feedback report.

COMPARATIVE ANALYSIS OF THE AWARD SIMULATION DOCUMENTS

The IDS software employed in the analysis can be used by an organization to keep track of its progress over a period of time, offering a logical and systematic way for comparative analysis. With the relevant data input into a database, comparison can be done between alternative courses of action and benchmarking undertaken. This form of comparative analysis is given in Figure 4. It can be seen that the 1998 submission has improved from a score of 36.6 percent to 45.8 percent. The score of 36.6 percent is somewhere between that of drifters and improvers, while that of 45.8 is close to improvers.

With the hierarchical framework and the IDS software, the areas for improvement from the previous year’s assessment can easily be identified from the difference in the scores. The methodology can also be used to simulate different improvement strategies, based on the number of areas for improvement identified from the self-assessment process. Each checklist item of attributes relating to a subcriterion can be categorized into short- or long-term improvements. The items identified are then assessed to an evaluation grade that the assessor believes can be achieved within the chosen time frame.

IMPLEMENTATION CHALLENGES AND LIMITATIONS

The MADM modeling framework developed on the basis of the evidential reasoning approach allows the assessors to map evidence extracted from self-assessment documents and site visits to the defined evaluation grades. This mapping process provides a natural way of interpreting a wide range of evidence, most of which is of a qualitative nature. In this process, the assessors are not forced to pre-aggregate qualitative evidence into a single numerical score. However, the mapping process does require the assessor to provide his or her assessments in a different way from conventional scoring, and it is not straightforward to map the extracted evidence to the set of evaluation grades. Therefore, appropriate training is needed to implement the MADM modeling framework. It should be noted that the evidential reasoning approach can only help to reduce but cannot completely eliminate subjectivity in self-assessment. This is because the mapping process still requires the subjective interpretation of extracted evidence in terms of the evaluation grades by individual assessors. It should also be noted that in the current evidential reasoning modeling framework evaluation grades must represent distinctive standards in assessment. In other words, they are required to be mutually exclusive. This is satisfied in the EFQM model as the five evaluation grades are related to percentage scores from 0 percent to 100 percent. Research is currently being conducted to allow the use of dependent (or nonexclusive) evaluation grades in the framework.

As discussed, the use of the checklists (or guidelines) can help assessors to improve the accuracy and consistency of their self-assessment. It has been recognized that it requires in-depth research and takes time and resources to identify commonly used guidelines for assessment criteria and areas to address. As part of such effort, several research projects are being conducted by the authors to investigate and organize self-assessment guidelines for constructing a knowledge base and decision models. Obviously, there is a need to stimulate more research in this area to develop widely accepted knowledge bases to support the self-assessment process. It should be noted that for comparative analysis within an organization or among a number of organizations, it is important to keep the modeling framework consistent and intact, which means that the same checklists and weighting systems should be used to assess all organizations in the analysis.

Finally, the criteria aggregating process of the evidential reasoning approach is not as simple as the additive weighting method as used in the conventional scoring method. The implementation of the proposed modeling framework and the methodology relies on the availability of easy-to-use computer software packages. The intelligent decision system mentioned in this paper provides a Windows-based package to support self-assessment. However, it takes time to get the awareness and acceptance of this and similar software packages. With the rapid development of computer technology, however, it will only be a matter of time for practitioners to employ computer software packages like IDS for self-assessment.

CONCLUSIONS

The process of defining the evaluation grades was done as part of a three-stage process: firstly, by defining the attributes with a precise list of checklist items; secondly, by deciding on the evaluation grades; and lastly, by defining each checklist item in terms of the different grades. Subjectivity is an inevitable problem because of the qualitative nature of the people management enabler, which was selected for investigating the feasibility of the application of MADM. By designing a MADM model with precise definition for each qualitative attribute and using a mathematical method via the evidential reasoning approach, it has the potential to reduce the subjectivity and biases of the scoring activity in undertaking a self-assessment against the criteria of the EFQM model and will improve the validity and reliability of the scoring process. The MADM model is also able to identify areas of attention for feedback. This is done using either the hierarchical framework method or the relative score method, or using a combination of the two methods.

A comparative analysis of the host utilities’ two award-simulation documents showed that the MADM model, coupled with the use of the IDS software, provides a logical and systematic way for comparative analysis of the results. This is extremely useful as it enables an organization to keep track of its progress over time.

The assessors, using the MADM model, are required to allocate an evaluation grade that they believe to be appropriate, and in this way the model is able to effectively simulate the results of a real-life self-assessment. By simulating different improvement strategies it allows an organization to plan effectively which improvement and changes will give the best results in terms of achieving excellence. Two different scenarios had been designed into the MADM model in terms of the time taken for improvement to be implemented (that is, short term and long term).

Acknowledgment

The constructive comments on this paper from the anonymous referees are greatly appreciated. This work is partly supported by the UK Engineering and Physical Science Research Council (EPSRC) under the Grant No: GR/N27453/01 and the Grant No: GR/N65615/01.

References

Ahire, S. L., and D. S. Rana. 1995. Selection of TQM pilot projects using an MCDM approach. International Journal of Quality and Reliability Management 12, no. 1:61–81.

BT United Kingdom. 1997. Quality Award 97. Belfast: British Telecoms.

Dale, B. G., and D. M. Lascelles. 1997. Total quality management adoption: Revisiting the levels. The TQM Magazine 9, no. 6:418–428.

Dale, B. G., and M. Smith. 1998. Spectrum of quality management implementation grid: Development and use. Managing Services Quality 7, no. 6:307–311.

Eom, H. 1989. The current state of multiple criteria decision support systems. Human Systems Management. 8, no. 2:113–119.

European Foundation for Quality Management. 1998. Self-assessment: Guidelines for companies. Brussels: EFQM.

Godfrey, G., B. G. Dale, A. Wilkinson, and M. Marchington. 1998. An audit tool for self-assessment of human resources policies within a total quality management environment. Manchester: Manchester School of Management, University of Manchester Institute of Science and Technology.

Hakes, C. 2000. The business excellence handbook. 5th edition. London: European Quality Publications.

Hwang, H., and K. Yoon. 1981. Multiple attribute decision making—Methods and applications. New York: Springer-Verlag.

Porter, L., and J. Tanner. 1998. Assessing business excellence. London: Butterworth-Heinmann.

Siddall, J. N. 1972. Analytical decision making in engineering design. Englewood Cliffs, N.J.: Prentice Hall.

Siow, C. H. R. 1999. Development of a distributed evaluation model for self-assessment based on the EFQM model for business excellence. M.Sc. diss., Manchester School of Management, University of Manchester Institute of Science and Technology.

TNT United Kingdom. 1998. Submission for the 1998 European Quality Award. Warwickshire: TNT.

van der Wiele, T., A. R. T. Williams, F. Kolb, and B. G. Dale. 1995. Assessor training for the European Quality Award. Quality World Technical Supplement (March): 12–18.

Yang, J. B., and M. H. Singh. 1994. An evidential reasoning approach for multiple attribute decision making with certainty. IEEE Transactions on Systems, Manufacturing, and Cybernetics 24, no. 1:1–17.

Yang, J. B., and P. Sen. 1994. A general multi-level evaluation process for hybrid MADM with uncertainty. IEEE Transactions on Systems, Man, and Cybernetics 24, no. 10:1458–1473.

———. 1997. Multiple attribute design evaluation of complex engineering products using the evidential reasoning approach. Journal of Engineering Design 8, no. 3:211–230.

Yang, J. B., and D-L. Xu. 2000. Intelligent decision system via evidential reasoning, version 1. Cheshire, UK: Intelligent Decision System.

Yang, J. B., Forthcoming. Rule- and utility-based evidential reasoning approach for multiple attribute decision analysis under uncertainties. European Journal of Operational Research (EJOR Paper #98287).

Yang, J. B., B. G. Dale, and C. H. R. Siow. Forthcoming. Self-assessment of excellence: an application of the evidential reasoning approach. International Journal of Production Research.

Biographies

C. H. R. Siow is a postgraduate student of the Manchester School of Management (MSM), University of Manchester Institute of Science and Technology (UMIST), in the United Kingdom. He received a bachelor of science degree in mechanical and manufacturing engineering at UMIST and his master of science degree in operations management at the MSM. This work forms part of the requirements for his postgraduate project.

J. B. Yang is senior lecturer of decision and system sciences at the MSM, UMIST. Over the past 16 years, he has been conducting research in quantitative and qualitative decision analysis and optimization with applications in various areas including product design and management. He is currently leading two postdoctoral research projects funded by the UK Engineering and Physical Science Research Council (EPSRC), one for measuring and assessing business performance using self-assessment and decision modeling, and the other to develop a hybrid methodology for multiple criteria decision analysis under various types of uncertainties. He has published three books and more than 100 papers in national and international journals and conferences. He is experienced in constructing decision models in different areas and in recent years has developed a large software system in C for design selection and synthesis and a Windows-based intelligent decision system in Visual C++ and Visual Basic for general multiple criteria decision analysis under uncertainties.

B. G. Dale is the United Utility Professor of Quality Management and the Head of the MSM, UMIST and the Rotating Research Professor at Erasmus University in Rotterdam. He has been researching a complete range of activities associated with TQM since 1981, and has an international reputation in the subject including organizational self-assessment for business excellence. During this period of time he has received research funding to the order of £4m, and has published 14 books and 350 papers.

The authors may be contacted as follows: University of Manchester Institute of Science and Technology, PO Box 88, Manchester M60 1QD, United Kingdom; 0161 200 3424; Fax: 0161 200 8799; E-mail: Barrie.Dale@umist.ac.uk.

If you liked this article, subscribe now.

Return to top

Featured advertisers

Article
Rating

(0) Member Reviews

Featured advertisers





ASQ is a global community of people passionate about quality, who use the tools, their ideas and expertise to make our world work better. ASQ: The Global Voice of Quality.