Leveraging the Power of Snowflake

With massive amounts of telemetry data generated by cloud applications, having disparate data silos across an organization can create bottlenecks for security analytics to perform threat investigation and searching for IOCs. However, by leveraging new platforms, new products have transformed SecOps teams to be more data-driven. By applying an open data model and stitching all the data with normalized fields, SecOps teams can enable all downstream analytics with zero data engineering, zero operations, and full data democratization.

Using the power of Snowflake, new solutions aggregate relevant search indexes into a new index that stiches all relevant data sources into a new search index using an open data model, like OCSF. This enables security teams to perform faster lookup of IOCs (i.e., file hashes, users, entities) for threat investigators within a security data lake over 12+ month periods.

Unpacking the Benefits of an Open Data Model

To unpack the benefits of an open data model, we need to look at security analytics, which rely on highly correlated data to yield new insights into the data. With a unified schema that normalizes field names (e.g., SRC_IP) and field values (Success/Failure) across all the disparate sources, the data model empowers analysts to find answers faster without any of the data wrangling in legacy systems. Furthermore, the data model needs to be OCSF compliant, enabling customers to rapidly onboard new data sources that comply with this new format.

Achieving Performance Search Index

Elysium Analytics has built an aggregation pipeline inside Snowflake to roll up the data from relevant sources, enabling the power of Snowflake SOS to query over 12 plus months of data within seconds. Merely turning on Snowflake SOS for source IP will result in poor performance, hitting all views and tables across the entire corpus of the dataset. Instead, building an aggregated table with deduping heuristics is much more high-performance. The aggregation pipeline process indexes to deduplicate repetitive messages and recursive alerts, enriches the records with missing details (user account, internal IP, last owner, etc.), and provides the following benefits:

• Semantic data model to unify the data in terms of field names and values across the sources
• Aggregate index for long-range queries over 12+ month periods
• Deduping the data to remove noise with repetitive and recursive messages
• Fast performance with Snowflake SOS
• Enrichment of the data with missing details
• Rapid Adoption of New Data Platform Technologies

Most security products today require the security analysts to rehydrate data from cold storage to query historical data. This creates large delays in response times during threat investigations. Elysium Analytics has solved this issue by rapidly adopting new data platform technologies to speed up long forensic searches from days into seconds, enabling our customers to respond faster to emerging threats.

In conclusion, the advancements in cloud technologies have transformed the way SecOps teams operate, enabling them to perform faster searches for IOCs and speed up threat investigations by 10x. By leveraging Snowflake and adopting an open data model, Elysium Analytics has developed solutions that enable security teams to aggregate relevant search. Please contact Elysium Analytics to schedule a demo!

The post Accelerating SecOps by 10x: Faster IOC Searches across the (Data) Haystacks appeared first on Elysium Analytics.

The use of advanced analytics and security automation has improved over the past few years and can help the SOC achieve better results. As such, advanced security analytics tools, supplemented by data from semantic security data lakes, come into the picture when it comes to according observability tools greater scope to take advantage of data.

A Data-centric Approach to Security

Data storage and the capacity of data that security teams can leverage are undergoing a revolution. The ability to selectively apply the necessary data from all organizational data allows teams to create more advanced use cases, but only through the continued evolution of tools.

Splunk’s cloud platform with advanced analytics is used by organizations to achieve security outcomes both within the SOC and across teams. By combining low-level signals into a single alert, security teams can generate higher fidelity alerts without overtaxing human resources by generating faster detection times without overtaxing them.

Data and analytics can only be used to improve security outcomes if you use them. The use of orchestration and automation, and expansion of the scope of observability tool capabilities through add-ons comes in here.

In this scenario, it is anticipated that the evolution of security analytics will significantly facilitate better detection, orchestration and analytics.

Amplifying Splunk Value with Elysium:

To enable easier access to diverse security relevant data sets, Elysium follows a unique approach that leverages the existing Splunk platform. The Elysium Analytics Add-on for Splunk on Snowflake enables Splunk Enterprise and Splunk Cloud users to run SQL queries on a Snowflake security data lake directly from the Splunk search bar. The results are directly viewable from the Splunk UI itself, in an intuitive, visually compelling manner. By integrating in-app, SOC teams can also correlate across data on both Splunk and Snowflake, and build visualizations within the Splunk application.

The result? Versatile visualization with a unified visual experience, enriched data, and access to a diverse repository of hot data on a Snowflake data lake directly from Splunk!

In effect, the Splunk add-on allows for the following:

1. Data archiving: Data value is perishable, and hot data is the need of the hour. The Elysium add-on allows for easy data archiving and access to hot data
2. Query any data from snowflake and view in the Splunk interface with ease
3. Quick build dashboards or reports in Splunk interface with Snowflake data
4. One-touch access to additional Elysium Analytics data
5. Query data in Snowflake from Splunk in minutes

The benefits that ensue from a simple add-on are innumerable. With easy access to unlimited hot storage at low cost on Splunk, and compliance and integrity being ensured by Snowflake, you get access to better and more relevant data, in a better interface. Elysium’s open data model allows data to be stitched across sources for a unified analytics view. The operational and engineering overheads are near zero, with a completely hosted Snowflake security data lake.

Access to security relevant data is critical for any SOC team, and will continue to improve so that businesses can derive analytics from their data regardless of where it lives. Data enrichment, through sources like Elysium, enable SOCs with access to visually compelling data and enable them to provide a complete picture of security infrastructure.

The post Augment Splunk with Elysium Cloud appeared first on Elysium Analytics.

TL;DR

Digital transformation across business units has increased demand on cybersecurity teams who are still predominantly using legacy data platform technologies. The good news is that the very same modern data stacks and AI technologies that have made digital transformation possible for business units can also be leveraged for cybersecurity threat investigation and threat hunting use cases, removing the scalability, cost, and operational overhead challenges of legacy solutions.

Why is Cybersecurity Analytics Technology Adoption Lagging?

Most businesses today have adopted digital transformation strategies and modernized their business models, social culture, and technology stacks. On one hand, this resulted in significant productivity gains, but on the other hand, it increased demand on cybersecurity teams supporting these initiatives while protecting the organization from cyber risk. The reality is that as organizations adopt cloud technologies, enable remote workforces, and more, security must also change and adapt. If security measures don’t match the digital pace, organizations face cyber risks that can impact the success of digital initiatives. It would be reasonable to expect that the success of digital transformation would translate across the organization, including cybersecurity, allowing security initiatives to keep pace with the rest of the company.

Unfortunately, executives have focused less on making cybersecurity teams data-driven, which would help them match security measures to the digital pace. Cybersecurity teams are still predominantly using legacy data platform technologies which were built to process gigabytes of log data in a time when all organizations are generating terabytes of log data. (Even still, many need petabytes or even zettabytes of storage for long-term data retention.) Additionally, security teams are often not equipped with the data engineering skills required to wrangle the current volume of data. Both legacy technology and limited access to advanced data talent are impacting cybersecurity teams’ ability to secure their organizations.

The graph below illustrates the gap between digital transformation in a corporation’s business units and the cybersecurity team. At the beginning of 2020, there was a distinct upturn in digital transformation efforts across corporate America. However, cybersecurity teams often did not take part in this evolution, and today the gap in capabilities is wider than ever. This makes it challenging for security teams to keep up with the resulting increase in log data volume. Cybersecurity teams need to develop a data-driven culture to close this gap.

In an effort to maintain a secure environment while adopting new cloud technologies, enabling remote workforces, and more, organizations have added new security point solutions for new specific security use cases such as UEBA, EDR, NDR, etc. These new solutions capture high-volume log data across the corporate network, cloud implementations, and remote endpoints and store the data in their respective proprietary data stores. This results in an increase of data silos which are rarely integrated and prevent security analysts’ visibility across the corporate network and user activity, also complicating the threat analysis process. This leads to low cybersecurity productivity and efficacy.

Cybersecurity Analytics and Modern Data Stacks

The good news is that the very same modern data stacks and AI technologies that have made digital transformation possible and successful for business units can also be leveraged for cybersecurity use cases. Now, the same way e-commerce personalization and recommendation engines tap into big data’s massive processing power to draw insights from vast amounts of data, cybersecurity can leverage these modern platforms to gain actionable insights into corporate cybersecurity risk. For example, there is a growing ecosystem of Snowflake partners who have built cybersecurity solutions that leverage Snowflake’s leading data cloud for storage and compute.

This new approach to security analytics, building the solution on a cloud-native stack, removes the scalability, cost, and operational overhead challenges security teams are facing when operating legacy solutions. Snowflake, with its near-infinite elastic compute and usage-based billing model, allows users to access the compute they need when they need it, and at a far lower cost than before. Being fully managed, the expensive operational overhead burden security teams carry today is also eliminated.

What are the Data Challenges for Cyber Security Analytics?

The two key challenges are:

1.    Loading data to Snowflake and utilizing their compute is not enough in itself to create a security data lake. Collecting and loading data from multiple log sources is an important first step. But without applying a robust data model to the data, you will end up with several separate data tables—in effect data silos within the data store—that are challenging to use for threat investigations, threat hunting, and even monitoring. An open data model will unify, correlate, enrich, and transform event data from all log sources. You can read more about how to bring all your security data together with an open data model here.

2.    When you have your security event data in a security data lake and applied your data model, the next challenge is how to extract value from your data. Most data lake platforms, such as Snowflake, can be queried with a SQL query tool such as SQL Workbench. It is also often suggested that using BI Tools like Tableau or Looker is a viable way to build visualizations and dashboards. However, most security analysts in a SOC are not familiar with SQL and do not have the time or skill set to build advanced dashboards.

Enter Elysium Analytics

This is why Elysium Analytics developed a snowflake-native security analytics solution with a turn-key open data model, along with applications for KQL full-text search, out-of-the-box security dashboards, ML-based anomaly detection, alerting, and more. Security teams can now keep pace with corporate digital transformation and implement truly data-driven security analytics for threat investigations, threat hunting, and monitoring. With Snowflake’s data cloud and our open data model, we remove any data engineering requirements, allowing security analysts to focus on investigations and build their use cases. Additionally, our security data lake and open data model can be leveraged by data scientists for advanced use cases and custom applications with native integration of Jupyter Notebook and developer access to our ML pipeline.

Although the Elysium Analytics security data lake solution can be implemented as a “SIEM light” solution with strong alerting capabilities, larger organizations typically implement this solution as a capability that is complimentary to their primary SIEM. This allows them to continue the use of current investments in technology and workflow processes, while at the same time enabling the ability to capture all log data—even DNS logs, proxy logs, and EDR telemetry—and process and analyze every event for actionable insights.

With the rapid adoption of digital transformation and the technology supporting this modernization, it will only be a matter of time before corporations also look to leverage the latest cloud and data science developments in their cybersecurity efforts. Fortunately, there are cybersecurity data analytics vendors such as Elysium Analytics who recognize this, building solutions that come as turn-key solutions, essentially eliminating the need for data scientists and data engineers that also have the required cybersecurity experience.

Contact Elysium Analytics for more information and a demo on how you can implement a security data lake for threat investigations, threat hunting, and monitoring.

The post Cybersecurity Analytics and Digital Transformation appeared first on Elysium Analytics.

Today, leading companies are solving these problems by augmenting their SIEM and adopting a Snowflake security data lake as the single place for all cyber security analytics and compliance data. With Snowflake’s cloud-built, multi-clustered shared data architecture, you can efficiently store years of semi-structured log Insights and log data, and scale compute resources up or down, automatically or on the fly, to meet the needs of your security analysts. Storing all your log data in a security data lake gives you full visibility to investigate the timeline of an incident across the full breadth of your high-volume log sources, including firewalls, servers, network traffic, AWS, Azure, GCP, and SaaS applications. Being a fully managed, near-zero operations platform, Snowflake gives security analysts the opportunity to focus on performing security investigations rather than spending valuable time managing infrastructure.

By combining a best-practice security data lake platform with full-text search, any information—structured and unstructured—can be retrieved from billions of log lines. With the best aspects of Snowflake and the OpenSearch Dashboards interface, you have an easy-to-use and scalable search solution.

Download the data sheet to get detailed insights on how to increase ROI with Search on Snowflake

Implementing a security data lake for instant access to historical log data allows you to reduce SIEM costs. Send only data from high value log insights to your SIEM solution, and stream data from all logs to your security data lake for long term retention. This leads to significant savings on license fees, storage cost, and operational overhead, while also meeting compliance requirements.

How to Search Your Log Analytics Data

With the familiar OpenSearch dashboards, also known as Kibana, interface, you can interactively search and explore your data with a pre-defined index pattern by simply entering your search criteria with KQL standard query language in the Query Bar. With the pre-configured index pattern, you can interactively explore your data in Discover, analyze your data in charts, tables, gauges, tag clouds, and more in Visualize.

When submitting a search request, the histogram, Documents table, and Fields list are updated to reflect the search results. The total number of hits, or matching events, is shown in the toolbar. Sort the table by the values in any indexed field.

The Documents table shows the first 500 hits. By default, the hits are listed in reverse chronological order, with the newest documents shown first. You can reverse the sort order by clicking the Time column header. You can also sort the table by the values in any indexed field.

How to Load Your Log Data

Setting up data ingestion is often a time consuming and challenging task when you have multiple sources of log data from cloud and on-premises sources.

We have made the process of collecting, parsing, enriching and loading your data simple. For data you are already loading to your SIEM or log management solution, we will configure the data collection implementation enabling it to write logs directly to Elysium Analytics open data model on Snowflake while maintaining the data flow to your legacy applications in parallel with zero disruption to existing solution.

If you are not collecting log insights data for log analysis today, we will set you up and handle everything for you from end to end.

Download the data sheet to get detailed insights on how to increase ROI with Search on Snowflake

About Elysium Analytics

Elysium Analytics is a security data lake for modern enterprises with complex infrastructure who are challenged by data silos, data overload, and increasing cost of data retention and recognize the value of a security data lake with an open data model for threat investigation, hunting, and monitoring. The Elysium Analytics solution is a turnkey open data model on a security data lake with applications and tools that provides for faster investigations and more efficient threat hunts and monitoring at cloud scale with significantly lower cost. Unlike legacy solutions, our product leverages Snowflake for cloud platform benefits, eliminates data silos and provides full visibility of log data. Our solution integrates with existing SIEMs and security tools.

Get your free trial today

The post Log Insights and Log data Analysis Best Practices-Snowflake Security Data Lake vs SIEM appeared first on Elysium Analytics.

Most organizations are built with threat detection and investigation capabilities leveraging a ton of vendor solutions. According to a 2020 Ponemon Institute report, organizations deploy on average more than 45 security solutions and technologies. This approach leads to multiple vendor–specific data silos which often results in storing multiple copies of the same data and no correlation across data silos.

No legacy security vendor is able to cost-effectively store and analyze all the data required to detect threats and facilitate post-hoc investigations and remediation in today’s heterogeneous environments. A modern approach applies the power of second-generation cloud vendors, providing the scalability required, and a robust semantic data model for quicker insights with behavioral analytics.

The Elysium Analytics Open Data Model (ODM) brings together all security-related telemetry (event, user, network, endpoint, cloud, etc.) into a unified taxonomy that can help detect and understand threats more effectively than before. Unified views are used to create analytic models with richer context of user and entity behaviors across a disparate set of data sources. Furthermore, the Elysium Analytics ODM enables the downstream analytics for sharing and reuse of threat detection models, algorithms and analytics.

Essentially the ODM provides a library of data source-mapping and high-level abstractions (e.g., any new VMs/container creation across the enterprise). Also, it describes security telemetry data used to baseline users and entities in the event data. This baseline with risk-based scoring helps identify anomalous behaviors across different levels of abstraction of user and entity. This data engineering is accomplished through schemas, data structures, file formats, and configurations on the underlying Snowflake data platform where the collecting, storing and processing of security telemetry data takes place at scale.

Elysium defines relationships between the various security data types for joining log data with user, network and endpoint data in both relational and graph models.

• Key Features of the Elysium Data Model are
  • Entity and user relationships
  • Knowledge graph of security events
  • Pre-built, high-level views of security for threat investigation

Semantic Data Model

To provide a framework for effective cyber threat analytics, it is necessary to collect and analyze both the standard security event logs and alerts as well as the relevant contextual data.

In addition to the most common entities such as network, user and endpoint, we include other data points such as file and certificate.

In the diagram below, the raw event tells us that user “lsmith” successfully logged into a WebServices hosted server from the IP address 10.1.1.15. Based on the raw event only, we don’t know if this event is a possible threat or not. However, after injecting user and endpoint context, the enriched event indicates a potential threat with root access that requires further investigation since this is a new unusual behavior for the user “lsmith.”

Extensibility of Data Models

Our ODM can also be extended to accommodate custom attributes by embedding key-value pairs within the log/alert/context entries. We have a separate Enrichment Json for each log record where all the enrichment of logs (e.g., Geo enrichment, Threat-Intel enrichment and Asset Enrichments) are stored. The model itself is extensible to add more enrichments as needed.

 Mapping to Third-Party Data Models

Models are extensible to map to third-party data models (e.g., ArcSight CEF or any other vendors schema). Once the mapping layer is implemented, all the third-party vendor data is funneled through the Elysium ODM and applied to the downstream models and analytics without any modifications required.

Model Relationships

The relationships between the data model entities are illustrated below.

 Relational:

As you see in the above hierarchical layers, the original source data fidelity is preserved and all the transformations happen during the query time. The model gives full flexibility, extensibility and adaptability for working across all the disparate log sources. This was carefully designed by the engineers who worked with hundreds of security log sources. As we add more sources from the bottom, the downstream analytics won’t need any changes.

The example below shows the enrichment of the security data through the ODM schema using threat intelligence from a shared table on Snowflake.

Contextual Relational Enrichment: 

Elysium Knowledge Graphs

Let’s connect how Elysium Knowledge Graphs provide additional insights into user activity. For example, a scenario where an EC2 machine hosted on AWS by user “John” is accessed from a malicious IP. This will be logged as a “finding.” With the Graph view, an analyst can see the complete flow of all entities connected to this user and understand the severity of the “finding” as well as entities impacted through it.

Suppose the same user “John” is also using a service through some role and is connected to this service with a private IP which also generates a finding or an alert.

An analyst can see what service “John” was using, what “role” he was assigned and who the role was assigned by, as well as the service configuration. Furthermore, the analyst can observe all connections to any endpoint accessed by “John” and any subsequent connections from these endpoints to a second layer of entities. As these findings originate from the same user, both findings will be shown together in the below graph, presenting a complete picture to analysts for root cause analysis and resolution in just a few minutes.

Conclusion

Elysium ODM enables security analysts to uncover advanced threats and anomalies within enterprise networks. Through semantic data model and data mapping, security analysts can reduce attacker dwell time by discovering and assessing adversarial behavior faster and with fewer resources. Furthermore, the semantic data model fuses security data sources with other contextual data to generate an enterprise behavior graph, a unique visual environment for analyzing advanced adversarial behaviors across petabytes of data. Elysium users will also benefit from our kill chain-focused User and Entity Behavior Analytics (UEBA) which automatically uncover adversarial TTPs. The model enables machine learning-powered analytics to provide greater context and focus to hunts and combines with linked data techniques to present analysts with intuitive summaries and visualizations of threat actor behavior along the cyber kill chain.

The Elysium ODM enables organizations to:

Store one copy of the security telemetry data and apply unlimited analytics

• Full text search across the data
• Out-of-the-box analytics using the ODM
• Custom analytics to your desired specification
• Plug-in third-party vendor analytics

Leverage all your security telemetry data to establish the context needed to better detect threats

• Behavioral analytics for user, endpoint and network entity data
• Enrichment with threat intelligence data
• Avoid “lock-in” to a specific technology and gain needed analytic flexibility resulting from an ODM
• Security log Semantic Data Lake which enables faster and better contextual analytics to find unknowns

The post Bring All Your Security Data Together With Our Semantic Data Model appeared first on Elysium Analytics.

There are two significant problems with log analytics and security analytics solutions today: A storage cost problem and a data silo problem. Both are relating to how data is stored.

1. Data Storage Cost

Since the first recorded reference to “Big Data” by Michael Cox and David Ellsworth with the publication of “Application-controlled demand paging for out-of-core visualization” in 1997, dealing with rapidly growing data volumes in an organization has been a never-ending challenge.

Security data is no exception and analysts have always had the need to store large amounts of log data and the migration to cloud infrastructure as well as an increased need to collect a vast amount of data from endpoints and user activity created by the growth of a remote workforce has made it a challenge for legacy monitoring and security solutions keeping up. 

Current monitoring and security solutions have evolved over the years from simple log management and SIEM solutions with rule-based alerting and simple anomaly detection on a manageable volume of log data to advanced cloud analytics, threat hunting, and AI on logs and metrics from vast cloud infrastructure and several SaaS applications. This has led to a significant increase in data volume ingested by legacy log analytics applications. This increase in data volume has put significant budget pressure on teams who must often resort to reducing the data retention time, compromising the ability to interrogate historical data.

2. Data Silos

The number of security applications implemented in a SOC has increased as well. In addition to log management and SIEM, most organizations today have implemented solutions for UEBA, NTA, EDR, CASB, advanced threat detection, advanced analytics, and others. These solutions are addressing specific needs and are often characterized as “point solutions,” storing data in separate data silos which prevents data in different solutions to be correlated during analysis. As analysts realized that data in one solution would be valuable to analytics in another, SOAR products evolved to facilitate access to information across data silos. For example, SOAR gives SOC teams the ability to create playbooks that automate the collection of threat-related data from several sources used to automate responses to low-level threats. However, this is merely a “band-aid” on the problem which does not solve the problem fully.

So, you have multiple separate data silos, each with limited data retention, and you face steep increases in licensing and operational cost from the increase in data to be analyzed. These problems stem largely from the storage technology used in legacy solutions.

A Brief Review of Storage Technology Evolution

Data Storage technology has evolved over the past 30 years from basic filesystem storage to relational databases, data warehouses, index storage, Hadoop stack, columnar databases, and most recently, cloud data warehouses. Over the past few years, data lakes entered the market to consolidate multiple data sources into one repository. Hadoop was a major driver as organizations embraced big data by creating data lakes for various disciplines such as marketing, sales, R&D, etc. These data lakes were used primarily by data scientists who would load a huge amount of geological, demographical, and meteorological data to aid in oil exploration, market segmentation, and weather forecasting, respectively, for example.

There were also attempts to create Hadoop-based security data lakes by some forward-thinking enterprise security teams who recognized the storage and data silo problem, although these efforts were rarely successful for the following reasons:

• Significant complexity involved managing Hadoop and the required data operations skills
• High operational overhead managing Hadoop clusters
• Expensive hardware drove up cost
• Lack of a pure SQL interface for the store
• Proliferation of stores with different interfaces for Hive, HBase, Spark, etc.

What Is a Security Data Lake?

A security data lake aggregates all enterprise log and event data and, by storing all security data in one repository, analysts now have easy access to all data across all sources and across all applications. In general, data lakes are different from data warehouses as they aggregate all data “as-is” independent of format or source, structured and semi-structured, into a single repository. In the case of a security data lake, data can come from any device, entity, user, or network in the cloud or on-premises. With schema-on-read, there is no requirement to understand how data is related when it is ingested but rather relies on the end-users to define those relationships as they consume it. This makes a data lake very efficient in processing huge volumes of data cost-effectively. 

In the past, one disadvantage of a data lake was that it was usually not good for quick and easy analytical processing. If you needed to return queries very fast, as in a few seconds or less, the data lake didn’t give you that performance. Additionally, it could be difficult for less technical people to explore and ask questions of the data with the lack of simple query language support. This has now been addressed, and solved, with the emergence of the modern cloud data lake, like Snowflake, and applications like Elysium Analytics, running on the cloud data lake or Snowflake security data lake.

A key difference between a legacy data lake, like Hadoop, and a cloud data lake, like Snowflake, is that a cloud data lake utilizes low-cost cloud storage to provide unlimited scale both for storage and access whereas a Hadoop data lake has physical boundaries on how much it can store or expand. The modern cloud data lake provides not only truly unlimited low-cost storage, but also unlimited on-demand compute which, in the case of Snowflake, is billed by the second. Snowflake has also solved the search performance problem with their Search Optimization Service” bringing response times down by an order of magnitude compared to the earlier data lake solutions.

Benefits of Snowflake Security Data Lake

Snowflake Security Data Lake can be implemented with the following benefits:

• Easy collection of data from all sources
• Process for cleaning and enriching data in the pipeline to the store
• Access to the data using standard interfaces – SQL, REST API
• Metadata Catalog of the log data
• Low operational overhead

 This is why the team at Elysium Analytics ported our Hadoop-based security analytics solution to Snowflake.

While initially building our solution on Hadoop, we experienced first-hand the operational challenges customers faced in the effort of scaling a complex, on-premises architecture cost effectively. In evaluating options for moving our solution from an on-premises architecture to a cloud-scale architecture, we had to satisfy three high level requirements:

a. Cloud-Scale Compute — Access to “unlimited” processing power for deep machine learning and queries on-demand given the nature of the “ebb and flow” of a security operations center where massive compute capacity is required intermittently when there is a security event that needs to be investigated. Any on-premises or traditional PaaS architecture would not provide this elasticity.

b. Open Data Model — Security operations and threat detection is all about the data and you must be able to store both structured or semi-structured formats in a raw source data format for data analysis and machine learning. Structured data needs to be enriched and parsed to accommodate a wide variety of reporting and analytical needs but not normalized for flexible query capabilities. This also allows you to query multiple data stores and data types at once. Finally, the open data model allows analysts to change data schemas and allows for differing views into the underlying data, without the difficult process of re-indexing and re-loading the data.

c. Predictive Analysis — Reducing the reliance on rules and queries to detect threat behaviors increase the detection capabilities for outcomes that are challenging to detect via rules.

The Modern Data Lake – Offered as SaaS

At Elysium Analytics, we have developed a Snowflake-native solution that meets all the above criteria and provides search, analytics, and graph on one data store with a full API interface and zero operations. With Snowflake as our data platform, we have a solution that provides:

• Simple collection of data from all sources with a cloud based “connect and collect app.”
• Processing, cleaning, and enrichment of the data in the data store pipeline with a cloud-scale data pipeline running on snowflake.
• Access to the data with full-text search, standard SQL and REST API, KQL interfaces.
• A Catalog of log metadata data with our open data model.
• A zero operations overhead platform. 

With Elysium Analytics, you can integrate with any security device log source “out of the box” and apply a data model. You can enable all log data in the data lake for full text search and profile for all interesting data points with “out of the box” behavioral analytics and explainable machine learning.

On the front-end of the solution, we have pre-built analytics dashboards for most common sources and provide the ability to build your own dashboards with a few clicks. For visualization we are leveraging and bundling Kibana, Looker, and Jupyter Notebook. All these visualization tools are natively integrated.

Conclusion

Though it may seem obvious now that leveraging data lake technology in security analytics is the right way to go, the right technology is needed to significantly improve its viability in a production environment. With the availability of the highly efficient Snowflake platform, modern security solutions are finally able to break through barriers legacy solutions are limited by. We believe this is the time for security teams to embrace Snowflake Security Data Lake.

Contact Elysium Analytics experts for a free trial for a simple full-text search and visualization of your data on Snowflake.

The post Snowflake Security Data Lake – Utilize Low-Cost Cloud Storage to Provide Unlimited Scale appeared first on Elysium Analytics.

OpenSearch is an open-source software project launched in 2021 as a fork of the Elasticsearch and Kibana projects led by Amazon Web Services. OpenSearch includes a search engine daemon, OpenSearch, and frontend visualization and analytics called OpenSearch Dashboards. This initiative was a result of Elastic, the company behind the Elastic Stack, also known as the ELK Stack, moving to a dual licensing structure based on the Server-Side Public License (SSPL) and the Elastic License – neither of which has been recognized as an open-source license by the Open Source Initiative (OSI). This move to a proprietary dual license created a clear demand for an open-source alternative, laying the foundation for the OpenSearch project.

ElysiumSearch is utilizing the OpenSearch Dashboards visualization tool and user interface to provide a fully managed search and analytics suite that runs natively on Snowflake in the cloud of your choice.

What is Snowflake?

Snowflake is a fast, zero-operations data lake as a service that scales dynamically to give you the performance you need exactly when you need it. With thousands of customers that have come on board over a short time, it has proven to be the greatest data cloud platform on the market today providing hot data storage for unlimited retention at a cost of $23 per TB of compressed data. As result, organizations have moved their transactional data to Snowflake from log management platforms such as Elasticsearch and Splunk, or data lakes such as Hadoop, and can now consolidate all their data silos on a centralized data lake.

How can I search on my data in Snowflake?

To access your data in Snowflake, you have Snowflake Worksheets, a web interface for entering and submitting SQL queries or the Snowflake SQL API to develop custom applications and integrations to perform simple queries.

But what if you just want to do a full-text search on your data? SQL is not for everybody in the organization and relying on data scientists to run SQL queries for all the various departments with a need for access to the data is inefficient and time consuming. 

Our maintenance-free search solution helps you make the most of your data and focus on building your business with simple full text search with Kibana Query Language (KQL), a simple syntax for filtering your data using full text search or field-based search.  KQL can suggest field names, values, and operators as you type and is able to query nested fields and scripted fields. This simple search language makes your data available to stakeholders who may not be trained on SQL and, with Snowflake’s automated compute scaling and permission management, allows you to democratize your data seamlessly and with no performance degradation across the organization.

Ad-hoc search on data from thousands of data and data service providers

If you are a data consumer on Snowflake’s Data Marketplace, you can leverage ElysiumSearch to gain quick and easy ad-hoc access to 3rd party data through search rather than having to rely on complicated SQL queries. This is great news for business intelligence and analytics professionals, data scientists, and others who depend on data-driven decision-making live access to ready-to-search data from an ecosystem of business partners and customers, as well as potentially thousands of data and data service providers. With ElysiumSearch and Snowflake Data Marketplace, you can source data faster and more easily, reduce analytics costs, and monetize data.

Visualization tools

In addition to full-text search, ElysiumSearch supports visualization of your data with OpenSearch Dashboards, the successor to Kibana, as well as Looker which is bundled with the service. In addition to the numerous pre-built dashboards that come with ElysiumSearch, you can also build your own.

Zero risk, low-cost usage-based pricing

ElysiumSearch is billed on usage based on Snowflake credit consumption on the relevant data warehouses. This means that there are zero up-front cost and you only pay for what you use. 

Contact Elysium Analytics experts for a free trial for a simple full-text search and visualization of your data on Snowflake.

The post Elysium Analytics brings OpenSearch to Snowflake FAQs appeared first on Elysium Analytics.