SIEM emerged as companies found themselves spending a lot of money on intrusion detection/prevention systems (IDS/IPS). These systems were helpful in detecting external attacks, but because of the reliance on signature-based engines, generated a large number of false positives.
The first-generation SIEM technology was designed to reduce this signal-to-noise ratio and help surface the most critical external threats. Using rule-based correlation, SIEM helped IT detect real attacks by focusing on a subset of firewall and IDS/IPS events that were in violation of policy. Traditionally, SIEM solutions have been expensive and time-intensive to maintain and tweak, but they solve the big headache of sorting through excessive false alerts and effectively protect companies from external threats.
While that was a step in the right direction, the world got more complicated when new regulations such as the Sarbanes-Oxley Act and the Payment Card Industry Data Security Standard mandated stricter internal IT controls and assessment. To satisfy these requirements, organizations are required to collect, analyze, report on and archive all logs to monitor activities inside their IT infrastructures.
The idea is not only to detect external threats, but also to provide periodic reports of user activities and create forensics reports surrounding a given incident. Vendors say this is what is driving the SIEM market today. “The number of regulation and industry mandates generating compliance requirements continue to expand, increasing the time and cost burden on today’s organizations. Effectively managing logs is critical in providing proof of compliance. This is not an easy task because log files come from a variety of sources, in different formats and in massive volumes. One area of rising importance is managing the entire lifecycle of log data primarily to demonstrate compliance and improve security in the most cost-effective manner,” says Ganesan Lakshmanan, Principal Consultant, Security Management, CA.
Gabrille Dechant, EMEA Marketing Manager for LogLogic, offers another perspective: “The need for increased visibility and control of all data within organizations networks and databases. The purpose of collecting logs is to analyze them, as well as to correlate them in real time in order to have not only visibility of your network, but also the ability to control and take action on what is happening. Security must be based on content knowledge and user behavior in order to be efficient and reactive.”
Though SIEM technologies already collect logs, they process only a subset related to security breaches. They weren't designed to handle the sheer volume of log data generated from all IT components, such as applications, switches, routers, databases, firewalls, operating systems, IDS/IPS and Web proxies.
With an emphasis on monitoring user activities rather than external threats, log management entered the market as a technology with an architecture to handle much larger volumes of data and with the ability to scale to meet the demands of the largest enterprises.
As companies implement log management and SIEM solutions to satisfy different business requirements, they are also finding the two technologies work well together. Log management tools are designed to collect, report and archive a large volume and breadth of log data, whereas SIEM solutions are designed to correlate a subset of log data to surface the most critical security events.
If you take a look at an enterprise IT arsenal, you'll likely see both log management and SIEM. Log management tools often assume the role of a log data warehouse that filters and forwards the necessary log data to SIEM solutions for correlation. This combination helps optimize the return on investment while also reducing the cost for implementing SIEM.
In these tough economic times it's likely we'll see IT trying to stretch its logging technologies to solve even more problems. It will expect its log management and SIEM technologies to work closer together and reduce overlapping functionalities. ” SIEM is the next step after Log Management and people realize that Log Management only does not provide enough value by itself,” says Markus Nispel, Director of Solutions Architecture, Enterasys.
Next-generation SIEM and log management
One area where the tools can provide needed help is in compliance. Corporations increasingly face the challenge of staying accountable to customers, employees and shareholders, and that means protecting IT infrastructure, customer and corporate data, and complying with rules and regulations as defined by government and industry.
Regulatory compliance is here to stay, and under the Obama administration, corporate accountability requirements are likely to grow. Log management and SIEM correlation technologies can work together to provide more comprehensive views to help companies satisfy their regulatory compliance requirements, make their IT and business processes more efficient and reduce management and technology costs in the process.
“Regulations such as PCI-DSS, SOX, HIPAA, FISMA, and The EU Privacy Directive and many other local and international regulatory requirements call for secure data collection, long-term storage and regular review of logs. This is to verify if proper IT controls are in place to protect the privacy of critical data, such as credit cardholder information, patient records or financial data. This shows how critical SIEM is with regards to the compliance,” says Lakshmanan.
IT organizations also will expect log management and intelligence technologies to provide more value to business activity monitoring and business intelligence. Though SIEM will continue to capture security-related data, its correlation engine can be re-appropriated to correlate business processes and monitor internal events related to performance, uptime, capability utilization and service-level management. We will see the combined solutions provide deeper insight into not just IT operations but also business processes. For example, we can monitor business processes from step A to Z and, if a step was missed, we'll see where and when.
In short, by integrating SIEM and log management, it is easy to see how companies can save by de-duplicating efforts and functionality. The functions of collecting, archiving, indexing and correlating log data can be collapsed. That will also lead to savings in the resources required and the maintenance of the tools.
It gets even more exciting when you can apply log-based activity data and security-event-inspired correlation to other business problems. Regulatory compliance, business activity monitoring and business intelligence are just the tip of the iceberg. Leading-edge customers are already using the tools to increase visibility and the security of composite Web 2.0 applications, cloud-based services and mobile devices. The key is to start with a central record of user and system activity and build an open architecture that lets different business users access the information to solve different business problems.
