New Vantage Partners began its big data survey series last year. In the first survey, 85% of respondents said their organizations were just beginning to explore big data, with only 35% investing $1 million or more. Just a year later, the current survey’s respondents have painted a different picture. Here are highlights of this year’s survey findings:
91% have a big data initiative planned or in progress
60% have at least one big data initiative completed
32% have an initiative in production
88% expect to spend > $1 million on big data by 2016
50% expect to spend > $10 million on big data by 2016
14% expect to spend > $50 million on big data by 2016
The survey results are especially interesting for organizations considering an investment in big data — time, technology or dollars — but worried about a lack of talent to push projects forward. Sixty-eight percent of respondents plan to retrain and upgrade existing staff to handle big data initiatives. Only 30% are actively seeking to recruit data scientists, or have already done so to augment their existing staff, and 19% report they already have expertise in house.
At the same time, organizations are looking at new processes, and in some cases new roles, to ensure adoption and use of big data. Over half of the respondents have established a Big Data Lab or Center of Excellence; 61% have established data governance standards and committees, and 56% have established a big data executive committee with management oversight. Nearly half (48%) indicate that their companies have established or are considering a Chief Data Officer role.
Between 60% and 70% of respondents listed their top areas of investment this year as: development of more sophisticated analytics; more effective integration of existing data sources; and the creation of analytical sandboxes to support data discoveries. A slightly smaller proportion (50%–60%) of respondents reported that their top areas of investment include migration of batch processes to big data platforms; improved fraud detection and risk analytics; and advancement of more sophisticated analytics (social network and time-series analysis).
The survey results are interesting in that they afford an intimate view into what big companies are doing with big data, from the mouths of those executives leading the charge. At the same time, there may be a lesson to be learned for smaller companies seeking big data insights. That is, while talent may be less of an issue for large companies, it may still be a stumbling block for smaller organizations that may not have the personnel to develop. In that case, seeking outside analytics talent may be the best option.