Businesses are clamouring to get their hands on as much data as possible, and are taking advantage of emerging Big Data technologies to gain insight into their customers’ demands. But can Big Data programmes be run using traditional IT teams, or do organisations need to take a new approach to hiring?
Earlier this year, projections by the McKinsey Global Institute said that, by 2018, there will be a need for 190,000 more workers with analytics expertise and a further 1.5 million data-savvy managers in the US alone. What’s driving this demand for data-oriented employees? According to the research firm, it’s all down to Big Data.
It might be something of a buzzword, but Big Data has certainly had the industry talking this year. Even in the Middle East, we’ve seen a number of high-profile Big Data implementations that have helped to transform the way that businesses look after their customers. And there are still plenty more uses for Big Data going forward.
According to IBM, the world is churning out 2.5 quintillion bytes of data every day, and 90 percent of it has been created over the past two years. Such vast swathes of data should provide good insight. However, it is incredibly difficult to leverage that data and turn it into something meaningful, hence the need for data-savvy employees. According to research firm Forrester, less than 5 percent of data is used effectively, and that’s mostly down to a lack of training and skills.
But do businesses looking to get serious about Big Data really need to shell out so much for expensive data scientists or sending existing staff on expensive analytics training courses? Well, according to Karthik Krishnamurthy, Vice President, Enterprise Information Management, Cognizant, it all depends on what you want to use Big Data for.
“Team alignmentputting the right team in placeis driven by an organisation’s use case landscape. When the majority of use cases are focused on business outcomes and objectives, the ‘right team’ requires a data science capability. On the other hand, if a company’s use cases are primarily positioned towards cost savings, processing efficiency, and platform optimisation, the ‘right team’ does not require a data scientist,” he says.
“For example, a data scientist is needed when the business use case seeks increased customer value, cohort analysis, micro-market segmentation, or modelling customer behavior based on social media. The data scientist has the job of understanding what data is relevant, designing statistical models, and visualising the data to communicate business-understandable benefits. In this case, the data scientist is the critical member of the right team. For IT-oriented use casesETL offloading, data archiving, mainframe conversion, real-time operational data storingthe right team is the IT group and Big Data technologists. A data scientist is not required in order to achieve these benefits. Whether driven by outcomes and objectives or cost savings, the ability to leverage Big Data is contingent upon connecting the right use case to the right team members.”
According to Krishnamurthy, depending on the skills of a current IT team, many organisations might have talent that is cross-compatible with a Big Data implementation. That said, it’s unlikely that any pre-Big Data IT team will be 100-percent prepared for what’s ahead. For one thing, new Big Data tools and technologies are emerging all the time, meaning it is difficult for anyone to be seriously experienced in how to use them.
“The aspect of skill set gaps is a result of newer Big Data technologies. It requires different knowledge in tools such as Hadoopand its many componentsand NoSQL, and these are just the basics. Beyond the basics comes the need for programming expertise in Map/Reduce, Java, R, PIG and Javascript. Visualisation is another capability that IT teams don’t typically have in their toolset. Yes, they are familiar with user interfaces and application design. Yet, Big Data visualisation requires a new level of data interaction, and a data cartographer,” he says.
It’s important to distinguish between the skills of traditional IT professionals and Big Data pros, Krishnamurthy adds. He says that IT produces answers to tangible business questions, whereas Big Data is more about searching for answers for questions that might not have been asked yet. Because of this, Big Data pros have an alternative mindset, and tend to think a little more creatively, Krishnamurthy says.
“There is another contrast between traditional and Big Data teamsthe approach. With traditional IT, it’s about requirements gathering and building, and you have more of an ask-answer relationshipif you don’t ask, they don’t answer. Big Data is more about agile development methods, self-service and business enablement through the development of solution platforms. You can incorporate most of the needed Big Data skills into an existing IT team, but you still have to plug the skill set and mindset holes. Companies can do this by cross-training the team and by bringing in new blood,” he explains.
So, if a Big Data project is going to be pursued, it would make sense that an organisation hires a data scientist to make sense of all that Big Data. But there might be a problem with this – according to a number of IT recruitment firms, data scientists are some of the most highly demanded employees of the moment. One magazine even rated data scientist as the sexiest IT job of the decade. This means that a good data scientist will be expensive.
But before organisations even start looking for a data scientist, Krishnamurthy believes that they need to work out what exactly they expect the new employee to do.
“To understand the demand for data scientist, it is important to clarify the role and its responsibilities. There are five skills required of a data scientist: business domain expertise, analytical capability, data visualisation, communication skills, and technology acumen,” he says.
“The data scientist is very important when it comes to Big Data. Having one person who knows how to find the right data sources and can then cleanse, merge, analyse and visualise the answer is extremely valuable. What also makes them valuable is not what they’ve learned but how they think. In kind with unstructured data, their thinking is also unstructured. Data scientists are creative, inquisitive and can see connections where others cannot. They are always looking for something interesting. Their ability to communicate their findings to others makes them even rarer. With the amount of complexity with which some insights are derived, results must be presented in a straightforward and simple manner. Data scientists get this.”
That said, there are a number of other positions to fill if a Big Data project is to be taken seriously. Indeed, Krishnamurthy believes that one job posting is even more important than the data scientist one, and that’s the Big Data architect.
“The Big Data architect is the glue between the organisational goals, the Big Data engineers, and the data scientist. They have cross-everything knowledge, spanning domains, industries and functions. These are the people who see the project through from beginning to end, from requirements gathering to deployment and everything in between,” he says.
In addition to the data scientist and Big Data architect, Krishnamurthy advises looking at hiring data governance and security professionals. He says that these employees are becoming increasingly important due to the increasing power of Big Data. Regulations are tightening due to the availability of personally identifiable information, and this means that companies need to be careful about what they do with their vast swathes of data, as well as how the protect it.
What’s more, as companies search for Big Data talent, they’re tending to target application developers and software engineers more than IT operations professionals, says Josh Wills, senior director of data science at Cloudera, which sells and supports a commercial version of the open-source Hadoop framework for managing big data.
That’s not to say IT operations specialists aren’t needed in Big Data. After all, they build the infrastructure and support the Big Data systems. Critically, these employees will use tools such as Hadoop, setting up the infrastructure to allow for such in-depth analysis. IT staffers can quickly learn Hadoop through traditional classes or by teaching themselves, the experts say. And burgeoning training programmes at the major Hadoop vendors are proof that many IT folks are doing so.
It would seem, then, that to really sink your teeth into a Big Data project, you will inevitably need to hire new talent. Sure, some deployments across the region have been done on the fly using existing IT talent, but if the Middle East is going to get serious about Big Data, it would seem that having the right talent is essential.
“A take-away here is that Big Data is beginning to create a second wave of resource needs,” Krishnamurthy says.
