Big data in the cloud

There’s a wealth of data that companies can use to better understand customers and identify emerging business opportunities and threats. But how to access and work with all the data? An emerging type of service called data as a service, or DaaS, promises to help.

With DaaS, organisations can gain access to information they need on an on-demand basis, much like they acquire applications via software as a service (SaaS) and storage, servers, and networking components through infrastructure as a service (IaaS). Data is stored by the service provider and accessible to users from the Internet.

Experts say that although DaaS is still an emerging concept, it’s becoming more relevant as organisations leverage big data – gathering and analysing massive amounts of information to help run the business, provide services to customers, identify trends, and open up new market opportunities.

As business and economics research firm McKinsey Global Institute pointed out in a May 2011 report, the amount of data in the world is exploding, and analysing large data sets “will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus.” The growing volume and detail of information captured by organisations, the rise of multimedia, social media, and the “Internet of things” will fuel exponential growth in data for the foreseeable future, the firm says.

“With the growth of size, speed, and spread of the big data sets and the never-ending quest for a competitive advantage, organisations are turning to large repositories of corporate and external data to uncover trends, statistics, and other actionable information to help decide on their next move,” says Srini Prabhala, head of the technology practice in the financial services and insurance group at the consulting firm Infosys.

Because businesses increasingly want to capitalise on information they don’t own – for example, a financial services firm going beyond its transactional data to analyse social data to better understand what customers like and don’t like – DaaS is likely to thrive.

How should IT and business users prepare for DaaS? Here are some recommendations from consultants and other experts.

1. Create a “data mind-set”

To get the most out of a DaaS deployment, IT management and staff need to think more in terms of data rather than hardware, software applications, networking, and other IT components, says Paul Gustafson, director of the Leading Edge Forum at CSC, a firm that provides IT outsourcing and consulting services.

“To move to DaaS means that data – not applications – leads. That’s a significant shift” in thinking, Gustafson says. IT departments need to adjust their focus from managing computing infrastructure to making sure the organisation leverages data in the best ways possible to foster innovation, he says.

That includes making data available to users and business partners whenever appropriate, providing scalable architectures, adopting cloud storage, and presenting case studies of data-driven business success stories to business and IT staff.

2. But don’t neglect infrastructure

Even with data taking centre stage over infrastructure issues and companies accessing much of this data via the cloud, many organisations still need to deploy greater internal storage capacity and bandwidth to leverage massive volumes of data if the information is available for download from the service provider.

The Virginia Bioinformatics Institute conducts a lot of genome analysis and DNA sequencing using some 100TB of data gathered each week from all over the world. To manage that volume, it is looking into using DaaS to help with its data collecting and analysis, says Harold Garner, the institute’s executive director.

Having adequate storage and processors with lots of memory will be an issue as the volume of data continues to grow with DaaS, Garner says. “You’ve got to have lots of local storage because you need to move stuff [to local storage] anyway,” he says. “You always have to take this data and combine it with other data.”

3. Try before you buy, check references, and insist on SLAs

Be prepared to do your research. “Ask for sample data or even access to the data service from each provider to see how it will work for your application and developers,” advises Chris Corriveau, chief technology officer at StockTwits, which operates an online community of stock market traders and investors and uses DaaS via cloud provider Xignite.

“Not all services are the same, and data format and access will really vary,” Corriveau says. “Shop around. As data becomes a commodity in some industries, you can strike deals and find the data provider that will fit your budget and data quality.”

DaaS vendors should offer some kind of prepurchase trial, Infosys’s Prabhala says. “Because the data is already available on the cloud, there should be no issues with giving prospective customers a test run,” he says. “Any vendor that avoids doing so either has to offer a very good explanation or has something to hide.”

Because DaaS is a relatively new service, be sure to check out references of other paying customers. “It’s still an emerging model with few established best practices, so it can be difficult for a vendor to work out the right pricing model and proposition to start getting market traction,” Prabhala says. “If it has referenceable customers on its books, that’s a good sign that it’s found its feet.”

Once you select a data provider, always insist on a service-level agreement (SLA). The cost of monitoring and administering an SLA might increase the overall cost of the service, Prabhala says, but if an outage occurs companies will have benchmarks to tell how hard the provider is trying to restore service.

A big concern with DaaS has been performance, says Noel Yuhanna, a principal analyst at Forrester Research. Because data services add another layer that can slow down response time to service requests, enterprises should enable distributed data caching with data services to improve performance and scale, he says.

4. Build a strong governance mechanism

With DaaS, extremely large amounts of data come in to organisations from a variety of sources and with varying degrees of criticality and requirements for privacy and security.

Organisations need to have strong governance around standards, guidelines, and policies related to DaaS. “Data governance plays a critical role in data services, ensuring that applications, users, and processes get the right data which they have access to and [that] the data is trusted,” Yuhanna says.

Security of the DaaS offerings is contingent on how data access controls are implemented, Prabhala says, and security of accessing the data service itself needs to be standardised. Concerns over the security of all cloud computing are already significant at most enterprises; those issues apply to DaaS as well.

“The drawbacks of data as a service are generally similar to those associated with any type of cloud computing, such as the reliance of the customer on the service provider’s ability to avoid server downtime,” Prabhala says. So governance related to terms of ensuring scalability and availability of the data sources applies to DaaS as it does to PaaS, SaaS, and IaaS.

But DaaS brings its own special governance concerns that require companies to reconsider the effectiveness of traditional data protection mechanisms, Prabhala says. “The characteristics of [this] deployment model differ widely from those of traditional architectures,” he says.

5. Emphasise data quality

Data quality should be part of the DaaS governance effort, but it deserves separate mention. If quality is not a high priority, DaaS might end up being a waste of time and money.

“Businesses should understand the quality mechanisms that a data provider has in place,” Corriveau says. “Poor quality leads to poor [results] and/or poor user experience.”

Specifically, check into whether the data service provider is cleaning its data so that customers don’t have to spend resources engineering filters, building monitoring tools, or managing issues related to poor data quality. “Businesses should demand quality data and ask a provider about how they maintain data quality,” StockTwits’ Corriveau says.

6. Ramp up your analytics skills

Much of the data that organisations acquire will need to be analysed in some way and put into context to create more value for the business. Although some vendors provide analytics-as-a-service offerings and your company might already have data analytics capabilities, you’ll need to build up internal analytics resources and skills like never before.

A growing number of organisations are leveraging the R programming language and software environment for analytics and statistical modeling, CSC’s Gustafson says. He expects this to accelerate as DaaS services gain momentum.

Some companies might opt to create entirely new entities to handle the data analytics required for DaaS. For example, Gustafson says, consumer goods company Procter & Gamble, a big user of DaaS, has established a text analytics group to deal with the new realm of data gathered from outside the company. Much of the data the group analyses comes from resources such as social media.

7. Know when to use DaaS and how to measure results

IT needs to work with its internal business partners to identify business need for DaaS. “A solid understanding of business data and the use and value of business data to various roles and stakeholders is critical in determining opportunities to leverage DaaS for the business,” says Mike Sabin, senior vice president of global sales and marketing at Dun & Bradstreet, a provider of commercial information on businesses. D&B, as both a user and provider of DaaS, deploys data services to deliver information on demand via the cloud to users through its Web services.

Once you’ve indentified opportunities where DaaS can provide critical business value – for example, helping internal research, human resources recruiting, supply chain management, sales prospecting, and marketing campaigns – Sabin says it’s vital to define and measure the expected return on investment. “Like any IT project, there should be stated goals and outcomes to measure criteria for success or failure of the program,” Sabin says.

DaaS use is on the rise to gain competitive advantage.

Clearly, the use of DaaS is on the rise in a range of industries as organisations look for ways to gain a competitive advantage by accessing data via the cloud. Forrester estimates that more than 1,500 enterprises worldwide are using such data services to support requirements for agile business intelligence, enterprise search, high-performance applications, real-time reporting, and dashboards.

“Business users should use data services to support all their enterprise data needs, because it offers consistent real-time data to support various queries and reports,” Yuhanna says.

Through the right preparation and ongoing maintenance on the part of IT and the business lines, organisations can take advantage of DaaS to turn big data into a big advantage.