Gulf Air shunned the expensive solutions of the big vendors jumping on the Big Data bandwagon, and became the first Middle East organisation to implement the technology by embracing open-source software with an in-house team.
There is no doubt that Big Data is the hottest IT topic of 2013. Everybody is talking about it as the next big technology to revolutionise the enterprise.
Unsurprisingly then, every man and his dog wants a piece of the action. All the industry heavyweights have been touting their own solutions, ranging from hefty appliances to in-built analytics technology.
Many are saying the truly mammoth frequency of data being generated by organisations warrants a completely new approach to IT infrastructure. Others are saying the related skills are so sought-after that ‘data scientists’ — a job spawned from the trend — are now amongst the most in-demand and, subsequently, highly-paid professionals in the IT world.
The trouble is, it’s expensive. Very expensive. Which is fine when the business case and ROI is clear — but right now, that is not the case.
Yes, we all get the benefits in principal. The data is at the disposal of the organisation and can be turned into actionable information thus generating business value. But “in principal” isn’t enough to get senior management to sign off the budget for an extremely pricey project.
What they need is guarantees; clear use-cases of enterprises that have made the investment and seen the returns.
So therein lies the hurdle. Big Data is so young in its infancy, that CIOs — whilst enthused by the technology — simply cannot justify the investment it warrants at this stage.
In step Dr Jassim Haji, IT Director of Gulf Air, the national carrier of the Kingdom of Bahrain.
Like everyone else, Dr Haji took an interest in the subject. “You won’t find any issue of any IT magazine these days that does not have an article or piece of news about Big Data,” he says. “Its growing popularity is very interesting and shines the light into its wide and attractive potential.
“However, we have to be careful not to lose our way in this big hype. It is easy for IT managers to become over-excited about such hypes and dive directly into the implementation of Big Data without aligning it with the business vision. Everyone is eager to get a piece of the action, but this has to fit into the greater picture of the business.”
Dr Haji took the standard approach when in the initial stages of an IT implementation, and looked at what was on the market. One initially took his eye. “To be honest,” he says, “we tried with Microsoft.
“But with Microsoft, like others, there were lots of hidden costs, and it was going to take time, and we were restricted with lots of technological restraints from Microsoft.
“So we came to a stage where we said, if doesn’t work with Microsoft and it doesn’t work with others, maybe it’s not worth going for.”
It is this stage where many CIOs are getting to and, just like Dr Haji says, deciding to put a project on hold until the technology matures and more success stories emerge.
Not Dr Haji, however. With the airline evolving towards restructuring, and opening and closing routes, he was determined to draw value from the hordes of comments they were getting on social media.
“They started tweeting and retweeting, so we found thousands of those comments going around,” he says. “It’s difficult to track them and understand where they’re coming from and how business should respond. And it’s not quantative — it’s something very qualitative that is hard and subjective to understand.
“We said, well, we’ve got so much data, we look at it every day, everybody thinks about it differently — how do you go about it?”
Furthermore, more than 80 percent of social-media posts about Gulf Air are in Arabic, Dr Haji reveals, and, as such, he wanted to experiment in Arabic sentiment analysis.
Having neglected the big vendors, Dr Haji took the bold decision to assign a project manager and four or five staff, from his existing team, and execute a part-time assignment to train themselves on Big Data.
“I sent them to various seminars on Big Data and they went online and researched and looked at different software,” Dr Haji says. “Then we had a backroom of staging to try these kind of things.
“When you have limited resources and you don’t have a big budget, it’s a good way to do it. To be honest, there was no proof of concept that I could take to the management and say let’s spend that much and this will be the result. It was all unknown.
“So I told the team let’s go after this and if we can do it good and well, we will be the first and would have done something unique. If we couldn’t do it, we had at least tried.”
Having done their research and self-training, the team opted for Apache Hadoop-based software from Cloudera.
Apache Hadoop is an open-source framework that allows organisations to process large amounts of data, regardless of its structure, at a low cost, and has largely been attributed to driving the growth of the Big Data industry.
However, Hadoop has presented the industry with challenges related to a lack of talent in those that can use the technology.
“We ended up with an open-source system where the vendor doesn’t support you beyond certain levels,” Dr Haji says. “You get the structure from the vendor, but you have to rely on yourself for everything else.”
“It was just a trial — it wasn’t something structured where a vendor had come up with a solution and said here is the ideal solution for your problems. We just made sense out of it and created from scratch, and whoever we talked to said they had never heard of anybody using Big Data for such things. But that didn’t justify that we shouldn’t go for it, so we started.
The “off-and-on exercise”, as Dr Haji calls it, took around a year between him appointing the team and the solution going live.
“They’ve never been entirely dedicated to the job and still aren’t now,” he says. “When you do a trial-and-error it takes lots of times, and you can’t afford to put five or six people out of their jobs.
“It was a challenge taken up by the team and myself to just go after it, keep it quiet, and see what we come up with. If it was successful we would announce it and tell everybody, and if it wasn’t we wouldn’t have told anyone,” he laughs.
The biggest challenge Dr Haji and his team faced was getting the system to compile all the different Arabic slangs and dialects that make up the language in order to understand the comments and translate them into English.
Having overcome that, the project, which required an extremely small investment, has been a great success.
However, the work is not done. The IT department has done the hard work, but for real business gain to come out of it, other departments must use the information to generate value.
“The implementation has already been a success,” Dr Haji says. “The data we got is very useful and valuable. Now, I’ll have to push others to do their job to make sure the success comes to the company. Whether they need other tools to go integrate with this to make sure we increase our sales or reputation, or not, that’s a different story.”
Having taken the route he did, Dr Haji is now firm in his opinion that it is not viable at this stage to be investing in the expensive Big Data solutions seen on the market.
“The investment they are asking the customers to make towards what they are going to get out of it is too much,” he says. “I think even Big Data at its highest capabilities that vendors are offering does the same thing. It doesn’t do the work for you, it brings you the data.
“If we draw value from 10 percent of the comments, we’ve already made a return. Sometimes, through difficult times you learn innovations. When you don’t have that much money to spend, there is a pressure on you. To deliver, you come up with creativities. That’s where we ended up being.”
He is also keen to emphasise that Big Data will have different meaning to each organisation, and each must be clear of the expected benefits of such a project.
“Despite the term, Big Data does not necessarily have to be big in size or volume; it is how to use the outcome of Big Data that is of big importance,” he says.
Looking forward, Dr Haji plans to extend the sentiment analysis to handle the English language, as well as expanding to other modes of media, like Facebook, newspaper articles and travel sites.
Furthermore, he wishes to take another challenge, which could see Gulf Air’s IT team help other organisations in the region embrace Big Data.
“I said, since we got to this stage, why not build on it. We have the expertise, maybe we can come up with something they are offering in the market at high prices. I’m not saying we’re going to sell it tomorrow, but maybe we can do part of that.”
