Exponential growth of Big Data and analytics has sparked the need for businesses to collect, store and manage this data, as well as having the tools available to accurately analyse it in real-time. How can an efficient log management solution enhance this process?
When looking around the data centre, it’s difficult to ignore the potential in all of the Big Data available from the array of systems. The importance of analysing unstructured data is undeniable, but in many cases, manually digging through huge volumes of information is overwhelming.
“Take a common application like Microsoft Exchange, where larger Exchange deployments generate upwards of 1 GB of logs per day,” says Deepak Narain, regional presales manager, MENA, VMware. “According to LexisNexis, 1 GB is the equivalent of approximately 677,963 pages of text. Simply put; manually sifting through logs is time-consuming, complex, and often not worth the hassle.”
Yet, to simply discard this volume of data could result in vital insights being missed, and this need to improve operational processes is one of the main drivers behind the increased implementation of log management solutions in this market.
“When faced with troubleshooting, an IT administrator is sometimes forced to spend hours digging through and correlating logs, trying to find the needle in the haystack to gain insight into the specifics of what is going on,” adds Narain. “How many man hours are wasted that could be better spent achieving IT or business objectives?”
Ensuring network security is another – and perhaps the most prevalent – driver for these implementations. “Detecting and mitigating security threats and protecting confidential data integrity largely depends on auditing devices and activities happening across the network,” says Manikandan T, director of product management, ManageEngine. “Though net flow data monitoring contributes to the aforementioned security analysis, log data is the primary source of information.” Whilst most of this information may be irrelevant, “a tiny anomaly could be the difference between preventing a security breach and seeing your name on the front page of tomorrow’s newspaper,” says Narain.
So how can CIOs look to ensure this fate is not their own when choosing a log management solution provider? One paramount feature should be ‘artificial ignorance’ – a method used by most log-management analytical solutions, which aims to minimise the possibility of missing out on high severity security incidents due to false positives.
“Artificial ignorance provides the ability to validate the context and then declare an event as an anomaly,” says Manikandan T. “If a solution doesn’t have this capability, then security administrators will be overwhelmed with false alarms, and a real security incident would go unnoticed.”
The need to comply with regulatory mandates is another main influence for a log management implementation. A proactive management system can help organisations avoid costly fines and restrictions imposed by failing audits, whilst helping them to adhere to internal best practices and mandated regulatory compliance standards.
“Most compliance systems require organisations to collect, analyse and archive log data, as it is the footprint for every event happening on the network,” explains Manikandan T. “Furthermore, investigating security breaches and attacks – one of the major requirements of most compliance mandates – requires conducting forensic analysis over the archived log data.” In other words, CIOs should be on the lookout for a comprehensive log management solution that not only performs log collection and analysis, but also archival and search if they are to fully adhere to compliance standards. “The ability to simply gather and archive certain log data is not a productive way of implementation,” adds Manikandan T. However, it is worth remembering that while most solutions offer compressed and secure storage techniques, archiving this data for a longer period of time will result in the need for secondary storage devices – involving additional costs and management.
Enterprises in the Middle East are fast-recognising the need for log management solutions that tick all of the requisite boxes. The slow-but-steady adoption of cloud technologies in the region seems to be driving organisations towards a cloud-based platform for this service, which presents many benefits to the enterprise.
“A huge benefit of moving to a cloud-based platform is not only low cost and affordability, but one predictable cost per user or per device,” says Narain. “With cloud-based solutions, an organisation is typically looking at shorter implementation cycles. But, more importantly, it will not need IT expertise within the staff to administer the solution.”
Enterprises are now in need of features beyond simple log collection, and are instead looking to extend their support scope to include the cloud platforms in their list. “Support to platforms such as AWS and Azure is the need of the hour,” says Manikandan T. “Enterprises are now looking for exclusive predefined reports that can provide valuable insights into their cloud development, and alerts that help to find security loopholes in their cloud infrastructure.” These predefined reports help to extract the necessary and meaningful information from the data whilst meeting security, compliance and auditing requirements.
CIOs should also be aware that a comprehensive log management solution should have the capability to collect, parse and normalise log data from various types of devices, including perimeter devices, critical servers and applications in real-time and in a central location. “The real-time alerting module of a log management solution will help in containing the security threats at the initial stage or even mitigate threats proactively,” says Manikandan T.
Here in the Middle East, it seems that the benefits of a log management solution are becoming more apparent in the enterprise. “Log management is something that is being adopted very rapidly across the region now, in the wake of security breaches and hacker attacks,” says Nairain. Enterprises are grasping the need for network security and are now actively looking for solutions that combine log management with security information and event management. “Such solutions enrich the log data even more with their threat intelligence and vulnerability management platforms,” says Manikandan T.
