Technologies That Media Companies Should Understand, pt. 1 – Sentiment Analysis

I’ve written previously about the consumer-centric environment businesses now find themselves in. With 3 billion connected devices out there, 3G wireless penetration and over half of the US adult population using social networks, information can be shared instantly, globally and from trusted sources. Consumers have information and choice, and as a result are starting to demand much more tailored and relevant products.

As a result businesses are getting serious about trying to understand their customers. Traditionally this was done through focus groups and test panels. This approach can be expensive, limited and usually biased. Luckily there is a new tool that can help. Sentiment Analysis uses machines to process and understand natural human language…in essence understand what people are saying and how they feel. While there are lots of advantages to this technology, from a marketer’s point of view it means we now have the ability to scrape the entire internet (all those tweets, facebook posts, message board conversations, user reviews, articles, blog posts, etc.) and understand not only what people are saying, but their emotions and how they feel. This isn’t a focus group or sample size, it’s the entire population (at least the engaged internet population) and it can be analyzed in near-real time.

For a film company this means the ability to understand how a trailer is resonating, what the response to messaging is, what elements work and which don’t, and how they can adjust. For a broadcaster it means knowing what audiences feel about one show versus another in the same time slot. For a record company it means knowing how fans are reacting to the latest stunt by a drunken lead singer. Once you have some historical data you can begin to correlate against sales and other metrics to add in a predictive element (e.g. based on a sentiment strength score of X we can predict an opening box office of Y). Pretty cool stuff.

I won’t get super technical, but the basic process revolves around text analytics and semantic analysis all built on dictionaries of positive and negative sentiment (i.e. ‘killer’ means good, ‘f*ck this’ means bad), but much more sophisticated. You begin with key terms associated with a product or brand. For a pair of Levis it might be ‘Levis’, ‘jeans’, ‘denim,’ ‘501’, ‘button fly’, etc. You then do a data pull gathering any relevant message or posts containing those terms, which means pulling millions and millions of conversations usually via a third party aggregator, or maybe in real-time from the Twitter Fire Hose. Those conversations are then analyzed with sentiment scores attached and patterns and features identified.

A process called machine learning allows the platform and dictionaries to become more refined as the computer identifies correct patterns and evolves its algorithm. Where things really get interesting is when you start to look relationships between associated terms and concepts (people who felt strongly about X also felt strongly about Y, Brand A is associated positively with concept Z while Brand B is associated negatively with the same concept). You can even start to understand what else people are interested in even if they don’t know themselves. This all creates rich data to make real-time product and marketing decisions based on worldwide consumer feedback.

The real crux here is that the text captured from a blog post or website is unstructured, meaning it is not in a neat table that a computer can use to define meaning. It is simply a lot of text characters in a row. It requires fairly beefy computing power that can process huge volumes of data that is highly variable (a term known as Big Data). However, once a system is set up it can process millions of conversations in a matter of hours. Many, many companies have jumped into this space and offer Social Media Sentiment Analysis as a service, making it highly affordable.

Sentiment Analysis is fairly widely known in marketing circles but I’m starting to see it pop up into product design, pricing, and distribution decisions. I would recommend anyone in a decision making role in any of these areas start to understand the strengths and limitations of this technology.