Volume 7 • Number 2
This article examines customers quality-related experiences and discussions in Usenet newsgroups. The objective of this article is to provide a catalyst for consumer product producers and an example of how quality-oriented information about a particular consumer product can be gathered and analyzed. An example of the process is provided, as are the results of what might be a typical effort of gathering and analyzing Usenet conversations for a specific product of interest.
Key words: customer feedback, product information, Usenet newsgroups
by BYRON J. FINCH, MIAMI UNIVERSITY
During the last decade, as businesses have responded to quality-based competition with more customer involvement, customers have enjoyed increasing access to each other to discuss quality-related experiences through widespread use of the Internet. Businesses have also capitalized on the Internets growth until it has become dominated by the World Wide Web. The Web dominance of the Internet has led to its transformation to a theme park of video, sound, and interactive experience. Despite the glitz and glamour of the Web, one portion of the Internet has maintained its early character. Usenet remains a freewheeling mass of thousands of interest groups, with little structure, no authority, millions of interactions per day, and communication via simple text. One can find a discussion on virtually any topic imaginable taking place somewhere among the newsgroupsits just a matter of searching for it and tuning in.
Usenet newsgroups are analogous to public bulletin boards. Anyone with Internet access can post anything to any newsgroup. Everyone accessing that newsgroup can read any of the posts and respond. The result is that conversations, called threads, are created. Usenet provides an opportunity for geographically separated individuals who might never meet to come together and converse about a similar interest.
With millions of interchanges per day, some businesses have come to realize that somewhere in the babble their consumer products or services are likely being discussed. One can quibble, all day about sample bias and validity, but if you were sitting in the stands at a ball game and heard your companys name mentioned by someone three rows up, your ears would perk up and youd strain to hear what was being said. Even though its just one person, youd want to know what he or she was saying. You wouldnt be thinking, This is a biased sample, so I shouldnt listen. Usenet can offer that same opportunity, but with greater efficiency and in greater volumes. The content detail and utility of whats being said may surprise you.
The objective of this article is to provide a catalyst for consumer product producers and an example of how one might gather quality-oriented information about a particular consumer product. This article provides an example process and the results of what might be a typical effort of gathering and analyzing Usenet conversations for a specific product of interest.
A previous study, based on monitoring Usenet newsgroups for a period of one year, searched for one company name. Any Usenet message containing the company name was retrieved, read, and categorized on a variety of constructs (Finch 1999). The study provided a breakdown of the types of messages, subjects of messages, tone of evaluatory comments, and so on. The intent of that study was to explore the potential of using Internet conversations as a means of gathering customer feedback in a passive way and to develop the constructs that would be useful in an examination. It concluded that indeed, useful information was present in these conversations. For that study, reference.com was used to monitor Usenet traffic. Today, deja.com offers that capability.
In this article, the focus shifts to a more pragmatic one by extending to the level of detail that would most likely be needed to incorporate this type of data into quality improvement processes. Here, the full extent of the information available on one specific product is uncovered to demonstrate what can be learned, as well as an example of how the data might be analyzed. The company, a manufacturer of consumer tools, will be referred to as ToolTech, a pseudonym. The product of focus was its table saw.
The information gathered consisted of any Usenet posts that mentioned the ToolTech company name from August 1, 1996, through July 30, 1997, with the exception of an 11-day period from May 13-May 24, which resulted from a server change. A total of 1641 Usenet posts contained the ToolTech name. Of those, 88 were deemed unusable because they were duplicate posts, in a language other than English, or unintelligible, leaving 1553. Of the 1553 useable posts, 1134 mentioned specific ToolTech products. Forty-eight different products were named. In addition, some products that had several models were discussed in general. Table 1 provides the frequencies of the ToolTech products mentioned at least 10 times. Because some products and combinations of products were unique to ToolTech, their names were disguised to preserve ToolTechs true identity. The table saw was by far the most frequently mentioned ToolTech product during that year with 401 mentions.
Because of the depth of information contained in the posts, the table saw data can be mined for an extensive amount of product-related customer feedback. This is best accomplished in two phases. The first is to find out about the information itself. In other words, what type of information was there, how best should it be compressed, organized, and filtered into a useable form that will lead to what consumers may be thinking and enhance quality-related feedback. The second phase is to examine the content of the condensed data.
CONDENSING THE DATA
In order to understand the data, posts were categorized by variety of constructs, presented in a model in Figure 1. It provided a variety of filters that improve the efficiency of any further analysis. The process used to translate the conversational data into a more understandable form was to first categorize each post on a number of criteria, and then process the results through a series of filtering sorts. That process is discussed in the following sections, with examples of posts to illustrate various constructs.
The decision as to whether it is best to search for any posts that mention the company name, and then sort by specific product, or to set up separate sorts for each product depends on several factors. In some cases it may be difficult to search for a specific product because it isnt going to be named uniquely in posts. Searching for a product by its model name may not work well because product users may not call it by that name. On the other hand, a product that is readily identifiable and well known by its model name, as was the case in this example, could be searched for in a model-specific search. In a broader sense, however, many firms would want to be able to view anything that mentioned the company name, making that type of initial search, followed by specific sorts by product, more productive.
Regardless of the approach used to gather posts that mention a specific product, posts that mention the product of interest must be isolated to form the preliminary level of analysis. After they were sorted by product, two other sorts can be performed. The first of these is sorting by type. Type refers to whether the post is a response to another post or an initial post that started a thread of conversation. The type can be identified through an examination of the subject of the post. Initial posts provide the word or words identifying the subject, as specified by the author of the post. Responses automatically show an Re: before that subject. For the table saw, of the 401 posts, 71 (approximately 18 percent) were initial posts while 330 (approximately 82 percent) were responses. This 1 to 4 proportion was typical for posts on all ToolTech products. It basically means that for every post made, there are, on average, approximately four responses. It is useful to know because it provides sort of a baseline for evaluating threads recognizing a hot topic. Some threads that are controversial or of great interest may last for many responses. Others die early from lack of interest. Some search utilities offer the ability to track a particular thread from its inception to its conclusion, making it much easier to follow a conversation, if that is important. Examples of both types are presented as follows.
Sample initial posts
I am an average woodworker and I would like to start making some furniture. Nothing too fancy, a mission bed, armoire, patio furniture, coffee, and end tables. I have been looking at the xxxx saws and a new ToolTech that had a lot of features. I know experience with a tool is the best advice. Any recommendations would be appreciated. I would like to stay under $800, and I would prefer not to have to buy another saw for about 10 years. I have plenty of room for it also. Thanks for the input.
Im trying to decide which table saw to get. Im not really a woodworker, but do need to do jobs around the house sometimes. The saw would not get a lot of hard use, so I dont need the best saw. On the other hand, I dont ever buy the lowest grade of anything. What would be a good mid-grade saw? Does ToolTech make a good saw? I hear xxxx is good. Is it better to NOT have the direct drive type of saw? Any comments would be appreciated.
Sample response posts
If youre looking for a good hobby TS, consider the ToolTech. Ive had one for 3 or 4 years and am quite satisfied. Its accurate, lightweight (about 75 pounds), easy to move around if you get the caster set, has a really outstanding dust-collection system, and enough power to do anything I call on it to do. It sells here (Delaware) for just a hair under $500, including the stand and a fairly decent blade. Take a look and see what you think.
Simple: The ToolTech, which I own, is full of little plastic parts, high tech, and constantly needing adjustment to work right, when it does. The yyyy is an older-fashioned cast iron model, with a good reputation for durability, which I wish I had purchased. Dont repeat my mistake. Look to the multiple ToolTech flames here.
The second construct of interest at this level was purpose. Purpose identifies the primary intent of the post. In the case of the ToolTech data, the various alternative purposes were identified by sorting the first couple months of data and identifying the major categories. Ultimately, there ended up being six purposes for the data.
The first category for purpose was evaluative. A post was determined to be evaluative when the primary intent dealt with evaluation of the product. The second purpose was for sale, and was selected when the post was trying to sell the product. The third purpose was information, and was selected when the product was mentioned relative to useful information about it. The fourth purpose was flame, and was identified when the product was mentioned in the context of a post whose purpose was to be derogatory or insulting. The fifth purpose was humor, and was used whenever the primary purpose of the post containing the mention of the product was to be funny. The sixth and final purpose was other, and was used for posts that did not fit any of the other purposes. Posts that were classified as other included mentions in legal posts or in online resumes. Categorizing purpose was admittedly a subjective process, but in most cases it was quite easy to identify the overriding purpose for a post. Sample posts, with different purposes, follow.
Sample evaluative post (initial)
Can anyone comment on the xxxx contractors TS. Pros vs. cons, and also the ToolTechone clearly better? While I received a couple recommendations regarding the xxxx, I am somewhat disappointed that it costs quite a bit more, and has stamped steel extensions that are not level with the main iron deck.
Sample evaluative post (response)
A wee bit underpowered. 8/4 hardwoods can be a bit challenging. The throat plate sits lower than the table. Needs shimming. No miter slot. I prefer the sliding table personally. Uses a single wheel to adjust depth of cut and blade angle. A bit more flimsy than your average contractors saw. Ive had mine for about a year now and its done everything Ive asked of it with no complaint.
Sample for sale post
This Table Saw System is virtually brand new and includes:
Basic Saw & Stand, Quick Fold Rear Work Table, Router / Jig Saw Mounting Kit, Dado Insert. Offers ???
Sample information posts Initial
Hokay ToolTech gurus, I hope you can help. The center table of my ToolTech TS is about 1/16" taller than the tables on either side. I cant find a way to adjust that. Can you tell me how?
Yes there is a way. Actually I was told that the sliding table may be too high and you could remove strips of Mylar under where it rests on the rails to lower it. I got this from a ToolTech tech on the phone as we trouble shot, but I had another problem. Adding or removing Mylar is how you adjust it.
Sample flame post
The question isnt where to buy it, its why to buy it. Piece of @$# %!!
Sample humor post (response)
DAMN! Did they do it again? Last time South Carolina seceded from the union, other states followed and we had the bloodiest war in the history of our country. <G>
Sample other post
We are currently looking for an SAP Basis Administrator for permanent hire. We are looking for the following qualifications and experience:
Clearly, the most useful purpose, from a quality information perspective, was the evaluative post. The informative post may offer insight that would benefit efforts toward improving consumer education, but the evaluative posts offer information directly applicable to product quality improvement.
Combining the constructs of type and purpose provides added insight into the characteristics of Usenet posts. For example, posts that request a product evaluation can be distinguished from posts that respond to a request for an evaluation. Table 2 provides the results of this analysis for the table saw data.
The third level of analysis deals specifically with the tone of product-oriented posts whose purpose was designated as evaluative. Evaluative posts were categorized as being positive, neutral/mixed, or negative. Positive was assigned if the post was entirely positive toward the ToolTech product. Neutral/mixed was assigned when there were neutral comments or when there were both positive and negative comments. Negative was assigned when the comments were entirely negative. The 204 evaluative posts yielded 96 positive posts, 54 neutral/mixed posts, and 54 negative posts. The tone construct will be examined further and broken down by type later in this article. Example posts with positive, neutral/mixed, and negative tones are presented as follows.
Sample positive tone
....I believe that due to its light weight and transportability, its ability to hold its settings so well under heavy use, combined with the standard features included in the basic package not available with other comparably priced TSsthe ToolTech will eventually become the contractors choice for an on-site table saw....
Sample neutral/mixed tone The one very appealing feature of the ToolTech to me was the dust collection and ability to attach a vacuum. I chose the xxxx contractors saw because of other reasons, I now need to vacuum up dust from the floor, however, the other appealing features of the xxxx made it a better purchase for me. Someone who is an absolute neat freak may think that the ToolTech is a better purchase because all contractors saws make a mess.
Sample negative tone
I too am a newbie and was making the decision about a year ago. I opted for the ToolTech and would now gladly sell it to anyone for $500. Another post has it dead on: the ToolTech cuts wood but is more work to keep in tune and the lack of a miter slot is a deficiency.
Evaluative messages are also sorted by whether or not a comparison was made between the ToolTech product and the product of another company during the evaluation. Of the evaluative posts, those that mention the products of other companies provide the most in-depth discussion of strengths and weaknesses of the ToolTech products. The negative comments become more meaningful as they are placed in that context. For the table saw data, of the 204 evaluative messages, 69 made comparisons, 135 did not. An examination of the evaluative messages by tone and compare uncovers an interesting phenomenon.
Table 3 provides the results of the breakdown by tone and compare. Posts that compared the ToolTech table saw to those of other companies appear more likely to be negative than those that evaluated it without the use of a comparison. For noncomparative posts, positive posts outnumbered negative posts by a ratio of greater than 3:1, with neutral/mixed posts accounting for slightly less than one-fourth of the total. For comparative posts, negative posts outnumbered positive posts by greater than a 2:1 margin, and neutral/mixed posts accounted for more than one-third of the total.
One could speculate as to the causes of this difference, but two seem most likely. It could be that people are just naturally more negative when evaluating a product using a comparison than they would be when evaluating a product without using a comparison. This creates a bias in the data, but probably isnt something that ToolTech would be able to change, nor does it provide information useful to the company for improving products. The second possible cause would be more of a concern to ToolTech and may make an important statement as to the quality of its products when compared to others in the market. It could be that consumers like this saw, and generally think it is a very good product until theyve had an opportunity to use another one and form a basis for comparison. It may be that at that point they realize what theyve been missing. That would label the ToolTech table saw as a product for beginners. The advanced or more sophisticated user may become dissatisfied and want to trade up to a better machine. Being the preferred choice for beginners is not necessarily a bad place to be, but it creates an undesirable reputation as far as quality is concerned, if that is not the market segment ToolTech seeks.
Before moving on and investigating the comparative posts further, the evaluative posts will be reexamined and sorted by type and by tone. In that way any indications that requests (initiating posts) for an evaluation have any pre-existing biases can be identified. One would expect that responses to requests for evaluations would show feelings that would indicate positive or negative perceptions. That was, in fact, the case, with 96 positive, 54 neutral/mixed, and 54 negative posts. One would also expect that people seeking the evaluation of a product from others would be neutral and not yet have a predetermined bias about the quality of the product. If they did, they wouldnt need another persons opinion. That can be determined by breaking down evaluative posts by type and then by tone. As is shown in Table 4, initial posts do seem to be less committed to a positive or negative bias than responses.
ANALYSIS OF CONTENT
In phase 2, specific content issues are examined, focusing on evaluative posts that compared the ToolTech table saw to others. It may be that for other products the sorting and filtering process used in phase 1 would result in a different focus for phase 2, but for the ToolTech table saw, this clearly appears to have the greatest potential.
The first and most obvious analysis of the content of evaluative posts that compared the ToolTech product to other products was to identify the other companies mentioned in the comparisons. Table 5 provides the frequencies of company mentions for those posts. There are more company mentions than posts because more than one company was mentioned in some posts. From the 69 comparative posts, eight companies were mentioned two or more times. Two companies clearly provided the bulk of the comparative references. From the perspective of consumers, these two companies provide ToolTechs greatest competition for the table saw market.
The second analysis of the comparative post content deals with the product attribute being discussed. Evaluations of a comparative nature typically identified a basis for the comparison. The basis of comparison can be considered to be a product attribute that, if it is significant enough for consumers to mention in a comparison, should also be considered significant by the producer. An analysis of posts mentioning either of the dominant two companies identified in Table 5 provides the list of attributes discussed and frequencies presented in Table 6. These attributes provide an interesting list of specific issues of concern to table saw users. The general attribute was assigned when a positive or negative comment was made in reference to the product in general, with no specific attribute descriptor.
The final level of analysis to assist in understanding consumer perceptions was done to provide an indication of how consumers felt the ToolTech table saw compared with its two largest competitors on the attributes presented in Table 6. This was accomplished by identifying the direction of the comparison being made for each instance of an attribute being mentioned.
For each post, any mention of a company (ToolTech or either of the two main competitors), positive and negative comments relative to the attributes presented in Table 6 were tallied.
Table 7 provides an indication that several issues, including a problem with holding settings, a lack of power, the price of accessories, and noise are viewed as negative when compared with the two main competitors. The table saws positive attributes appear to be portability, versatility, and dust collection.
The example provided here was not intended to show that statistically significant inferences can easily be derived from these data. They probably could be through sophisticated content analysis. It makes the most sense to look for clearly defined patterns and trends, as were pointed out in Table 7, and use them as an indicator of the need for further investigation. Clearly, the sophistication of this analysis would not be sufficient to initiate a product design change, but it provides substantial evidence to delve further into an investigation of outstanding consumer concerns. The clear pattern of consumers feeling that the ToolTech table saw does not hold its settings and requires constant readjustment is an example. The evidence here points to a need for a deeper investigation of that problem.
The subjectivity required in the analyses will always be suspect, particularly when results dont show any clear indications of differences or problems. However, surveys and focus groups also have their weaknesses. They are expensive, may also be prone to bias, and provide only a snapshot in time of customer perceptions. The Internet provides an additional, not a replacement, means of monitoring consumer reaction to products, and can enable the producer to monitor that reaction on a continuous basis. Depth of discussion and willingness of consumers to expound in great detail, combined with the low cost of the information, lend support to using Internet conversations as one more source of customer feedback.
Finch, Byron J. 1999. Internet Discussions as a source for consumer product customer involvement and quality information: An exploratory study. Journal of Operations Management 17, no. 5:535-556.
Byron J. Finch is a professor of management and the director of faculty development for information technology in the department of management at the Richard T. Farmer School of Business at Miami University, in Oxford, Ohio.
Finchs research interests include spreadsheet models and applications in operations management and quality management potential for Internet technologies. He is particularly interested in how Usenet and other discussion media can be used to enhance product quality.
Finch has been published in many journals, including the Journal of Operations Management, International Journal of Quality and Reliability Management, Production and Inventory Management Journal, and Quality Progress. He has authored or co-authored five books and is currently the managing editor of the Operations Management Center, a resource portal sponsored by Irwin/McGraw-Hill.
He earned a doctorate in operations management from the University of Georgia. He may be contacted as follows: Richard T. Farmer School of Business, Miami University, Oxford, Ohio 45056-3628; 513-529-3159; Fax: 513-529-2342; E-mail: Finchbj@muohio.edu.
If you liked this article, subscribe now.
(0) Member Reviews