Big Data: From Hype to Value

Do you think ‘big data’ is over-hyped? Are you tired of hearing about its potential and want to know how it affects the bottom-line? Will Hayes (CEO, Lucidworks), Phil Kim (Cap One), Michael Morrison (CEO, Datawatch), Adam Towvim (CEO, Trust Layers) share their perspectives on how companies are deriving value from big data today, moderated by Andrew Burton (CEO, Logentries).

As the panel traversed topics ranging from hiring practices in big data to data governance and protection, there was one major reoccurring theme: big data is not the same as the big answer. As Adam Towvin eloquently put it, “sometimes it’s not about finding the needle, it’s about building the right the right haystack.”

Much of the reason that big data as a field feels like hype is because of the ways it’s being described as a way to find answers and solutions for a huge range of previously unsolved challenges – and those answers have not been forthcoming since “data science” became an industry buzzword.

But as our panellists today have outlined, that’s partially because of the way big data is being misinterpreted and partially because of limitations in data usage and lack of expertise in the field.

Will Hayes kicked things off reminding us, "don't get infatuated with technology, get infatuated with results,” a sentiment echoed by the rest of the panel. Phil Kim agreed that big data is in stage where people don’t understand what it is and how it can be used – and although there are great tools for extracting value from data, people aren’t always sure on how to use them.

The biggest benefit of big data at the moment, said Towvin, is that data helps companies innovate faster and improve business in ways it was impossible to do before.

Andrew Burton turned the panel to the question of how “offensive or defensive” data should be in bringing data science practices to other parts of the business.

While the group agreed that an offensive approach was important for educating others about the value and limitations of big data, as well as formulating questions and helping customer-facing teams build better products, there was some discussion of the tension that other departments (particularly legal) might feel when large pools of data across the company are thrown together and leveraged in this way.

When it came to data governance and protection, the panel acknowledged the importance of protecting data but Towvin in particular felt the aggressive ways in which data was encrypted and hidden away undermined those data true potential. While protection was important, there is a sense we are selling customer short by using that data to make better decisions in areas ranging from health to consumer products to personal finances. Kim highlighted Capital One’s practice of putting risk management and marketing teams in the same room so that they can jointly solve problems of how to protect and use data effectively at the same time.

The audience was curious about how to prevent people from confusing correlation and causation – a challenge the panellists admitted is almost impossible to solve. Their solution? Hire the right people, experts with deep insights to help them avoid these biases.

Of course this naturally led to a discussion of how to build the right team and make the right hires for the field. Kim explained the challenges he faced explaining to hiring departments what a data scientist actually was, but all of the panellists agreed that the most important trait for a data scientist was a deep intellectual curiosity, as well as scientific yet creative mind that can absorb the quantitative challenges of big data while still looking for interesting patterns.

Fortunately for the students in the room, Michael Morrison spoke to the need of these big data companies and teams to hire “the next generation.” He pointed out that many of these companies need to do as much to market themselves to talent as candidates need to do to market themselves to companies. Morrison highlighted an example from Datawatch – they opened an office in Boston as they knew it was a more appealing location for the type of people they wanted to attract.

Looking forward, the panel spoke passionately about the benefits that real time analysis of streaming data will provide to the industry.

Thank you to Will, Phil, Michael, Adam and Andrew for a great overview of this fascinating and fast-moving field.