Big Data and Quality Professionals

Based in Dallas, TX, Ponmurugarajan Thiyagarajan  (Rajan) is a business development manager for Digital Enterprise at Tata Consultancy Services and a senior member of ASQ.  He is passionate about quality, digital reimagination solutions and is a “Mac head.”

He blogs at Quality Matters, http://pmr-blog.blogspot.com/

Were you pleasantly surprised when the receptionist at a hotel proactively identified you with a greeting as you were about to check-in?   Did a relevant coupon pop-up in your smartphone when you were shopping at a retail store recently?  Did you receive a reply to your tweet in social media from your telephone company with an apology note for the service inconvenience caused?  If you could answer “Yes” to any of these questions, then big data is mostly that magical element that helped those companies to manage and deliver this customer experience for you.  Big data has evolved as an effective tool that can be used by companies to continuously improve aspects such as customer experience, product quality, business processes etc.

Big data is in play when data size is huge (Volume), moves in high speeds (Velocity), comes in variety of forms (Variety) and in varied quality (Veracity) which conventional database systems cannot efficiently process.

Analytics built over big data enable organizations to process structured and unstructured data to derive useful intelligence and provide actionable insights for end-users.  The advent of high-speed network connectivity, commodity computer hardware, and open source software such as Hadoop and Non-Hadoop (for example: NoSQL) technologies have made big data a popular technology choice.

There are interesting use cases of big data that can help organizations that are committed to differentiate, innovate, and embrace disruption of conventional processes.  For example, wearables (watches, bands, etc.) and connected devices (Internet-of-Things glucometers, connected cars, connected homes, etc.) utilize big data technologies to collect and process huge amounts of real-time data from machines (logs), people (social), and other sensors (internet of things).   From these data, organizations get to understand customers’ 360 degree view and derive the ability to contextualize and deliver a personalized experience.

That being said, big data is still a buzzword for many and often perceived as a misused terminology.  While some organizations have tested it to work, to a good extent, other companies are still researching it, and some are even hesitant to adopt it at all. One of the key challenges that I can think of is the accuracy and uncertainty around the quality of data that is gathered and processed.

Lack of good data governance is a major cause for this challenge.  Also, outliers and incorrect data misdirect users during the decision-making process.  Business users demand high quality of data to derive actionable insights.   Being an emerging technology area, I believe that big data has to be further researched from a quality point of view.  I have these questions for the quality professionals:

All of this has interesting implications for quality professionals who may become involved with big data efforts. Assurance of quality is key in such projects: data clean-up must happen in an automated fashion and reconciliation reports to be produced in real-time to track quality parameters. Thus, relevant tools needs to be built for quality assurance. It will be interesting to see how quality tools such as Plan-Do-Check-Act, the 7 quality tools (Fishbone diagram, Check sheets, Control charts, Histogram, Pareto Charts, Scatter Diagrams, Flow Charts) etc., can be customized for a big data project.

I believe there’s a lot of possibility in this area for quality professionals, as I’ve yet to see any concrete maturity models around big data. This is a potential topic for future research.

To conclude, let me state an example of a large corporation that probably is making the best use of big data.  It is Google that really attempts to help users, like me, to plan vacation or business travel in a modern digital way.  Right before a recent trip, Google provided relevant notifications and guidance to my smartphone on when to start to the airport, the best route to take to avoid delays, the status of the flight with gate information, hotel booking information, etc.

Google seems to collect a lot of information from users’ mobile devices, emails, internet browsing history etc., to derive and offer useful analytics.  It is interesting to note that users, like me, are ready to slightly compromise on privacy (by opt-in) for the benefits we can enjoy.  I think this is another good example to demonstrate big data in action.

I invite you to share further thoughts and views on big data and how quality professionals can play a vital role in this digital era.

This entry was posted in Customer Service, innovation, quality tools and tagged . Bookmark the permalink.

8 Responses to Big Data and Quality Professionals

  1. Steven Noel says:

    With so much data (terabytes) statistical inference may be a thing of the past. With that said, assuming normal distribution of data and using the entire population, would not the errors in the data be diluted or fall outside significance?

  2. Steven Noel says:

    With so much data (terabytes) statistical inference may be a thing of the past. With that said, assuming normal distribution of data and using the entire population, would not the errors in the data be diluted or fall outside significance?

  3. Iris Byfield says:

    Thanks Rajan for presenting Big Data in its simplest, basic concept. The foundational info you presented will help to build further knowledge on it, eliminating the fluff and noise that accompany it most times.

    Being a quality professional, I think it is fair to comment that we always work with data, using tools to process it to obtain meaningful information in relation to customer satisfaction and process efficiencies. Big data just means we need to up the ante in building our skills on this including learning about new tools that will facilitate this learning.

    There is no alternative for the quality professionals – customers are getting digital savvy; we need to embrace the digital technologies and create collaborative inroads with the technology services group in our own organisations who most often are the assigned custodians of big data.

  4. Martin Andrew says:

    Thanks for an excellent article. This summarises the possibilities and issues really well. (I have recently led discussions on Big Data at several Australian Organisation for Quality’s ‘Café Quality’ events).

  5. Martin Andrew says:

    Thanks for an excellent article. This summarises the possibilities and issues really well. (I have recently led discussions on Big Data at several Australian Organisation for Quality’s ‘Café Quality’ events).

  6. torrents says:

    Currently it sounds like WordPress is the preferred blogging platform available right now.
    (from what I’ve read) Is that what you’re using on your blog?

  7. Shaheen Alam says:

    Thanks Rajan for nice write up, obviously there is no good complete tools observed yet to manage Big Data in efficient manner.

    Still, quality professional is managing their analysis with the available tools they and and it’s definitely required verification big putting into decision table.