How hot is your data?

According to industry commentator James Taylor, “Analytics simplify data to amplify its value”. The problem is that data today for many organisations is so overwhelming a problem that even getting it under control is a problem, let alone getting it into the state where it can be analysed properly for business advantage.

The problem with storage is two-fold. Firstly, the explosion in data means that organisations struggle with the sheer volume and, with regulatory and compliance issues butting up to a seemingly endless exponential growth in internal and external data, the traditional solution of throwing more storage at the problem is not working any more. As Teradata’s CTO Stephen Brobst puts it, “Just adding more storage merely adds to I/O issues – you get to the stage where you just can’t access information fast enough”.
Secondly, as companies have battled to solve for years, just storing data is not enough – it needs to analysed in a timely fashion in order to be of business benefit. That’s why data silos become data graveyards unless business analytics are used intelligently.
An answer to both problems may have arrived from Teradata, the company who created and now dominates the data warehouse market. It has delivered two new Active EDW (Active Enterprise Data Warehouse) platforms to enable smarter decisions at hyper speeds. Critically, these are the first to combine solid state drive (SSD) and hard disk drive (HDD) technology with the industry’s only intelligent virtual storage solution that automatically migrates data between drive types to achieve optimum performance.
The Active EDW 6680 combines SSD and HDD technology and intelligent virtual storage solutions; the new Active EDW 6650 uses HDD storage and can be upgraded to integrate SSDs and Teradata Virtual Storage.
“This delivers what we’re calling the multi-temperature warehouse,” explained Scott Gnau, Teradata’s Chief Development Officer at the recent launch of the company’s EMEA conference in Barcelona. “Realistically, most data is cool, needing only to be accessed once in a while. A reasonably small amount of data is hot, meaning constant access is critical. And the rest is warm, implying regular usage. What the new Active EDWs do is store hot data in SSDs and cool in HDDs.”
“The effect is to get the best memory utiilisation at the best price,” Brobst said. “Sure, the cost of processor power is always decreasing but you pay for the I/O overhead. This way, you’re using the fastest storage for the most important data and the cheapest storage for the less used data.”
While the volume of data continues to grow, the number of business applications and users that leverage that data are also growing exponentially. For instance, it’s not uncommon for a business to have 20 or more BI apps, ten times more than just a few years back. Because of the need to do more with the data approach of the integrated data warehouse which ratchets up the need for more performance per terabyte of data space, the new Active EDW platforms enable all data to be integrated into one warehouse and then used simultaneously by multiple applications to maximise the value of the data to the business.
According to Gnau, “The Active EDW 6680 offers up to four times boost in throughput and unleashes performance for reporting, operations and complex analytic applications. That means customers can do more, do it faster and do it better, thanks to the combination of large memory and efficient cacheing.”
The new products also avoid an issue being faced by customers of other SSD-based storage solutions, because the TVS (Teradata Virtual Storage) solution automatically tracks the usage of data and then intelligently migrates it to the appropriate storage type, instead of the manual data migration required by other systems. This frees up database admin time, reduced errors and cuts TCO.
According to Gnau, in time the data migration will be predictive, rather than just self-managing. Brobst admits that some users may feel uneasy about losing the manual control, but insists that “you can no longer manually manage your data”. He also points out that so-called Big Data (unstructured from social networking, etc) brings bigger data management problems. “The bigness of the data is not the most interesting part of Big Data. It’s more about the diversity than the size.”
The benefits of automated predictive data management will be important to a lot of industry sectors where timeliness is critical. Take network security, for example: current detection systems used by network providers and government watchdog agencies fall short in effective threat screening, because of the unprecedented number and sophistication of cyber attacks. Users need to simultaneously analyse the continuous stream of incoming packet data along with the massive longterm historical data to find patterns that lead to the discovery of threats.
Or consider the complexity of today’s 24/7 business model that demands managers make up-to-the minute decisions regarding sales, inventory, operations or financial data. For example, a retailer can now instantly spot sales trends and react with inventory, supply chain and pricing decisions to optimise their revenue and margin for products. The reliance on reports based on data from overnight batch processing gives way to operational business intelligence dashboards supported by active data warehouses. And, as dashboards enabled by SSD-based solutions replace day-old reporting, managers can be far more interactive and agile as they manage their business for enhanced profit.
The Active EDW 6680 scales from 7 Tb to 36 Pb user data; the Active EDW 6650 platform scales from 7.5 Tb to 92 Pb of user data.