Based in Dallas, TX, Ponmurugarajan Thiyagarajan (Rajan) is a business development manager for Digital Enterprise at Tata Consultancy Services and a senior member of ASQ. He is passionate about quality, digital reimagination solutions and is a “Mac head.”
He blogs at Quality Matters, http://pmr-blog.blogspot.com/
Were you pleasantly surprised when the receptionist at a hotel proactively identified you with a greeting as you were about to check-in? Did a relevant coupon pop-up in your smartphone when you were shopping at a retail store recently? Did you receive a reply to your tweet in social media from your telephone company with an apology note for the service inconvenience caused? If you could answer “Yes” to any of these questions, then big data is mostly that magical element that helped those companies to manage and deliver this customer experience for you. Big data has evolved as an effective tool that can be used by companies to continuously improve aspects such as customer experience, product quality, business processes etc.
Big data is in play when data size is huge (Volume), moves in high speeds (Velocity), comes in variety of forms (Variety) and in varied quality (Veracity) which conventional database systems cannot efficiently process.
Analytics built over big data enable organizations to process structured and unstructured data to derive useful intelligence and provide actionable insights for end-users. The advent of high-speed network connectivity, commodity computer hardware, and open source software such as Hadoop and Non-Hadoop (for example: NoSQL) technologies have made big data a popular technology choice.
There are interesting use cases of big data that can help organizations that are committed to differentiate, innovate, and embrace disruption of conventional processes. For example, wearables (watches, bands, etc.) and connected devices (Internet-of-Things glucometers, connected cars, connected homes, etc.) utilize big data technologies to collect and process huge amounts of real-time data from machines (logs), people (social), and other sensors (internet of things). From these data, organizations get to understand customers’ 360 degree view and derive the ability to contextualize and deliver a personalized experience.
That being said, big data is still a buzzword for many and often perceived as a misused terminology. While some organizations have tested it to work, to a good extent, other companies are still researching it, and some are even hesitant to adopt it at all. One of the key challenges that I can think of is the accuracy and uncertainty around the quality of data that is gathered and processed.
Lack of good data governance is a major cause for this challenge. Also, outliers and incorrect data misdirect users during the decision-making process. Business users demand high quality of data to derive actionable insights. Being an emerging technology area, I believe that big data has to be further researched from a quality point of view. I have these questions for the quality professionals:
All of this has interesting implications for quality professionals who may become involved with big data efforts. Assurance of quality is key in such projects: data clean-up must happen in an automated fashion and reconciliation reports to be produced in real-time to track quality parameters. Thus, relevant tools needs to be built for quality assurance. It will be interesting to see how quality tools such as Plan-Do-Check-Act, the 7 quality tools (Fishbone diagram, Check sheets, Control charts, Histogram, Pareto Charts, Scatter Diagrams, Flow Charts) etc., can be customized for a big data project.
I believe there’s a lot of possibility in this area for quality professionals, as I’ve yet to see any concrete maturity models around big data. This is a potential topic for future research.
To conclude, let me state an example of a large corporation that probably is making the best use of big data. It is Google that really attempts to help users, like me, to plan vacation or business travel in a modern digital way. Right before a recent trip, Google provided relevant notifications and guidance to my smartphone on when to start to the airport, the best route to take to avoid delays, the status of the flight with gate information, hotel booking information, etc.
Google seems to collect a lot of information from users’ mobile devices, emails, internet browsing history etc., to derive and offer useful analytics. It is interesting to note that users, like me, are ready to slightly compromise on privacy (by opt-in) for the benefits we can enjoy. I think this is another good example to demonstrate big data in action.