Integrating with Splunk and ELK

Osquery is yet another powerful and effective tool in today’s security modernization concepts. Osquery has been driving Security Information Event Management (SIEM) for a couple of years with more intelligence and extended approach like never before. Osquery has been upgraded the standards of SIEM with its efficient log aggregation mechanism, which can be easily integrated with any other power source like Splunk and ELK.

Advantages of having an ideal SIEM solution:

  • SIEM provides a centralized and consolidated view of your data helping further to effective analyze the security standards of the stored information.
  • SIEM has the power packed ability to process both structured and unstructured data and can normalize it as per requirement and demands.
  • A wide variety of security use cases are addressed with SIEM, which can support any kind of security related operation.
  • An intelligent log processing can be achieved through SIEM
  • SIEM solutions are scalable and can support enormous data

Let us understand how Osquery can benefit your SIEM

Osquery enhances the SIEM solution with more integrity support and log aggregation tool integrations. Apart from this, Osquery can also help SIEM with:

  • Cross Platform Support: Osquery supports almost all the platforms and operating systems, thus increasing the capacity limits of SIEM and FIM concepts, promising more vigilance and security.
  • Cloud Based Security tool: Osquery completely supports and addresses the native demands of all cloud-based infrastructures, enabling more efficiency at cloud level providing optimum security.
  • Flexible Customization: Osquery extends the flexibility of customization of SIEM integrations, FIM solutions, Aggregation logs and many more security modules, which help in increasing the transparency of overall organization security standards.
  • Less Noise and More Research: Osquery enables security teams to research on a deep and rich set of endpoint data by prioritizing only the relevant data and removing the unnecessary data logs, which helps in focusing on vital elements. Osquery works on virtual platforms too, by easily deploying itself on modern environments. This helps various security organizations to conduct incident response investigations and hunting operations without installing external software.
  • Scope for Continuous Development: Osquery community is working tirelessly to improve the technology, which can provide a fair chance to all the security vendors to use Osquery without exploiting.

Why do you need an external SIEM Solution for Your Osquery?

Osquery is well designed to adapt in various platforms and environments irrespective of the existing data or data infrastructure. Each time the query engine runs on virtual tables, there are number of logs created with the respective data and Osquery does not implement the system of log forwarding internally.

So, based on one’s infrastructure demand and data inputs, one has to make the key decision on implementing the right log aggregation tool.

What are Aggregating logs?

Aggregating logs are generated by Osqueryd and by default, Osquery uses the filesystem logger plugin to streamline the logger data, which is not as effective in terms of security perspective. Hence various open source and commercial products, which are specially designed for processing log data are used here, to increase the scope of threat visibility.

Integrating Osquery SIEM with ELK or Splunk:

When we integrate the Osquery SIEM either with ELK or Splunk, the Osquery module collects and decodes the result logs written by Osqueryd in the JSON format and performs the following checks initially:

  • Sets the default paths to the log files (but don’t worry, you can override the defaults)
  • Makes sure each multiline log event gets sent as a single event
  • Uses ingest node to parse and process the log lines, shaping the data into a structure suitable for visualizing in Kibana

Once the initial prerequisites are met, the following conditions are satisfied:

  • Compatibility
  • Configuration
  • Variable Setting
  • Result Fileset Setting

We need to clearly understand that, Osquery SIEM solutions use Osquery, ELS/Splunk as follows:

  • Osquery
  • Data producer
  • can also be used to ship logs (log forwarder)
  • Splunk/ELK: Data Analysis/Aggregation

Threat Analysis with YARA 

YARA is a tool that aids in identifying malware and classifying malware samples by looking  for certain characteristics. YARA rules identify CVE; exploit kits, mobile malware and many others by using rules that specify regex patterns. 

Osquery offers YARA scanning with the help of two tables, yara_events and yara which are in beta phase currently. With the powerful FIM of Osquery and simple YARA configuration, YARA can verify rules when a file change event is fired. 

Yara table offers on demand scanning, given a set of rules and filepath Osquery can match the rules against the given file path. 

Deployment in Cloud with AWS logging: 

For those who want to store the logs in cloud, Osquery can log results to AWS kinesis streams and kinesis firehose. This would help in having a centralized storage for logs accumulated from various services and devices.

Configuring Osquery with AWS requires credentials, AWS config files, AWS access key ID, AWS secret access key, Profile from the EC2 Instance Metadata Service. 

Conclusion: We know that Osquery equips a wide variety of log forwarders and seeks vital attention as a perfect log analysis tool for better SIEM performance.

So, choosing the right tool to reduce the complexity of handling mundane log files in Osquery and processing them with more transparency is the point to be considered always.