SIEM solutions can be critical for compliance, security monitoring or IT optimization. But it also can be difficult to find the right SIEM product. The reason? First, a large number of products are available and vendors can make products sound roughly the same in core features such as correlation, reporting, collection, etc. Second, vendors seem too busy differentiating themselves with features that in many cases have little or nothing to do with core functionality.
SIEM solutions, in reality, are optimized for different use-cases and one size never fits all. The good news is that with the number of potential solutions to choose from, if you do your homework, you will find a product that will meet your requirements. What is the best way to select the right solution?
Step 1: Know your requirements. This first task is obvious, but cannot be overstated. Look at your requirements in depth and figure out your needs before heading to procurement. To use a car analogy – if you don't know the engine size you need, the trim package can easily become a critical buying criterion. You need to have a clear vision of your objective and the knowledge of what you need to get there. You should rank business drivers and requirements in order of importance, and during the vendor analysis do not get too excited by anything, no matter how powerful, if it's low on your list.
Keeping your objective in mind, consider these areas when drawing up requirements:
a. Collection – What data sources do you need to log? Do you need real-time collection? Do you need to collect all data or a subset?
b. Storage – Do you need to archive everything? How long do you need to store data? How long do you need to keep data online?
c. Analysis – How will you use your data once collected -- for legal forensics, detecting threats in real-time, isolating attacks or incidents or in audit scenarios?
d. Reporting – What sort of reports do you need? Do you want the ability to customize reports?
e. Other consideration: Do you need multi-user access? What level of access do you want to give users? Do you have specialized staff that can manage a more complex solution?
Step 2: Educate yourself. Once you have your requirements identified, educate yourself on the industry and available options. An educated buyer is the best buyer so, consult with your peers, do research and look at vendor websites. Although most vendors will claim they do everything equally well, a good clue to their overall focus is still their websites. For instance, if the focus of the website is on network defense and real-time intrusion detection, odds are the vendor comes from the security event management side. If the focus is on secure storage, compression, and compliance reports, it's likely the vendor comes from a security information management background. Since SIEM solutions are not created equal, it's wise to explore these core differences:
a. Optimized for SIM vs. SEM usecase
b. Appliance vs. software-based solution
c. Real-time vs. non real-time collection/analysis
d. Agent vs. agentless architecture
e. Data normalization vs. compression
f. Relational database vs. proprietary storage mechanism
All options have pros and cons. Take the time to understand the differences before you go into the procurement process.
Step 3: Understand the jargon. A couple of terms that SIEM vendors use to claim product superiority are EPS (events per second) and scalability. Both are defined differently by vendors and using them as a basis of comparison is often a case of apples to oranges. By understanding the jargon, you'll empower yourself to ask the right questions.
EPS – Many vendors use EPS synonymously with scalability and overall performance. Before you buy into this, ask yourself one question: What is the vendor definition of EPS? Is it number of events scanned/second, received/second, processed/second, or inserted in storage/second? Is it a real time count or a batch transfer count? What is the event size? Are events received in UDP or TCP mode? You will find the answer differs with each vendor. The reality is that an EPS measure is generally based on a small, non-typical normalized event. Also, understand that a vendor's chosen definition is likely the one where performance is the best.
With the lack of consensus on event size and no clear definition, EPS is therefore a terrible comparative measure. One vendor claiming 12,000 EPS is not necessarily faster than another claiming 10,000 EPS as they're generally measuring different things. EPS is worse for ascertaining true solution capability. Let's assume a system receives, processes and stores 10,000 windows events/sec with an average 1K size (a common Windows event size). In 24 hours 86 million events will be received, which will consume 8.6GB of storage at 90 percent compression. Even with heavy compression, many SIEM products will only be able to handle weeks or a few months of data with this load.
All in all, EPS is largely an empty measure until the unit of measure is standardized, and even then it will only be a small part of true capability
Scalability – Scalability should be thought of as multi-dimensional, encompassing:
Collection: Collection is a multi-step process. Receiving an event is not the only part of the process. Events must be processed and data committed to storage and these activities consume system resources. It is advisable to look at how vendors define scalability for all three activities.
Storage: Scalability is evident in not just storage size but also how easy it is to move data between on-line and off-line storage, retrieve and process records. And what happens to archiving when the system gets temporarily overloaded? Do the events get cached or lost?
Analysis/reporting: This important aspect of scalability is often ignored. A system might process 10 million events per minute but if it takes 10 hours to run a query you are probably not getting a scalable or for that matter a viable solution.
For best results, identify the dimensions of scalability that are critical to your success and dig deeper.
Step 4: Build your shortlist. By now, you know what you need and have a good understanding of the industry and available options. You can now approach vendors to start the procurement process. A good starting place is to look at product demos to create your shortlist and then move to an evaluation.
Look beyond the sizzle at the demo. For many vendors the main purpose of the demo is to use FUD (fear, uncertainty and doubt) or sizzle to make their features seem to be a must-have. The reality is that a lot of projects fail because prospects buy into this sizzle, but during implementation realize that they never use the cool stuff and struggle with the capability they really need. For instance, a lot of vendors claim “plug and play” during the demo. What they don't tell you, however, is that after installation, it is your responsibility to point your data sources to the collection engine, which can often be time-consuming and require manual intervention on each system. This is certainly not “plug and play” since real deployment work begins after the SIEM is installed.
Step 5: Conduct an evaluation. Without an evaluation you are dependent on the vendor to give you correct information. One of several things can happen during implementation if you don't know what you're getting:
- The solution is unable to scale to your requirements. The add-ons kill you on budget.
- Excessive false positives are generated. The analysis engine is too complicated to tune and you fail to detect a real threat.
- Once you gain experience with the product and tune it to meet evolving objectives, you find the architecture inflexible.
- Although the solution supports your current data sources and log volume, you are forced to make heavy chunks of investment to scale as your enterprise grows.
- Deployment is a nightmare and professional services are not in the budget.
- The things you really need are hard to use and the stuff you really don't need you never get to.
The best way to cut through vendor hype is to test products yourself. So, bring the short-listed products in-house and preferably use them on your network in limited deployment rather than a lab environment to get a sense of how the products work under real-life scenarios.
Step 6: Look at support, maintenance levels. Through the procurement process, you should have gotten a feel around the kind of support you should expect to get. Will you be struggling to make it work or helped through the deployment and management process? For instance, if you need a new report, will your vendor work with you to develop this? What if you need a custom correlation rule or if you have a new data source that the vendor does not support? Or, if you have a looming audit and need your vendor's guidance through the process? Ask the vendor these questions. A trusted provider will be willing to give you value-added services without breaking your budget -- this can often mean the difference between a good and a great project.
Step 7: Evaluate long-term value. Finally, evaluate licensing models for long-term value rather than initial price. A vendor might offer you a great price that fits your budget initially but what happens when your IT infrastructure grows? How will licensing scale when your log volume increases beyond solution capacity? Look also for hidden costs in terms of separate modules, compliance packs, storage, training and support. The last thing you need is unexpected costs that you never accounted for.
At the end of the day, if you understood your problems, requirements and drivers and did your due diligence, you should be on your way to a successful project with a product that is optimized for your business problem.
