Store and analyze high-volume logs efficiently with Flex Logs

The volume of logs that organizations collect from their systems is growing exponentially. These logs have increasingly diverse sources—from distributed infrastructure to data pipelines and APIs—and similarly varied forms and use cases. As a result, log data has become more and more difficult to manage. Organizations must reconcile conflicting needs for long-term retention, rapid access, and cost-effective storage. Meanwhile, existing solutions tend to create their own hurdles, from painstaking schema maintenance and complex querying to time-consuming re-indexing operations. All of this can lead to gaps in visibility and put key data out of reach when it is most urgently needed.

To address these problems, we're pleased to introduce Flex Logs. With Flex Logs, Datadog offers a solution for centralizing all of your logs within one platform, no matter the use case. Building on the flexibility offered by Logging Without Limits™, which decouples log ingest from storage—enabling Datadog customers to enrich, parse, and archive 100% of their logs while storing only what they choose to—Flex Logs decouples the costs of log storage from the costs of querying. It provides both short- and long-term log retention for a nominal monthly fee without sacrificing visibility, enabling streamlined correlation between all of your logs, metrics, and traces.

In this post, we'll show you how Flex Logs enables you to:

• Consolidate all of your logs—no matter the use case—in one platform
• Rein in costs without sacrificing visibility

Consolidate all of your logs—no matter the use case—in one platform

Because logs are collected from so many different kinds of sources and for such wide-ranging purposes, there can be no one-size-fits-all solution to managing them. Amid this complexity, organizations often face a dilemma: to index or to archive. Indexing makes your data rapidly searchable, so it's essential for logs you query frequently and use for real-time monitoring and alerting, such as application logs. But indexing can be resource-intensive and costly, so indexing all of your logs is impractical at scale. Archiving tends to be a better solution for logs you need to keep long-term, for compliance and other purposes, but which you query infrequently and without much urgency. If and when you do need to search them, these types of logs can be retrieved using Datadog Log Rehydration™, for example.

But in many cases, neither indexing nor archiving is a perfect solution. Logs of network activity, security events, and business transactions, for example, are often generated in enormous volumes. Compared to application logs, these types of logs are seldom queried—often only months after they are generated. At that point, however, querying them is often a matter of urgency (during investigations of potential security breaches or fraudulent activity, for example).

By decoupling storage from compute costs, Flex Logs provides a solution for long-term retention of logs without impeding query performance. With Datadog Log Management, you can now:

• Use Standard Indexing for logs you tend to query frequently within a short time span (i.e., a few days), such as application logs
• Use Flex for logs you collect in high volumes, retain in the relatively long term (i.e., months or years), and sometimes urgently need to query, such as security, transaction, and network logs
• Use Flex Frozen for logs you need to retain for extended periods (up to seven years) in order to meet regulatory requirements or for auditing purposes, and want to be able to query on the fly without rehydration or context switching

Take, for example, an e-commerce company that makes millions of sales a day. Visibility into the services that handle the checkout process is indispensable to many different teams within this organization. Datadog Log Management enables these teams to cost-effectively centralize all their log data in one platform by tailoring their log indexing and retention policies to their needs—eliminating the overhead of managing external storage solutions as well as the need for context switching.

For example, in order to rapidly query and alert on any failed checkouts, engineering teams keep these logs in the Standard Tier. In case of an incident causing a spike in failed checkouts, teams can respond immediately and query as much as they need to quickly identify the root cause and prevent revenue loss.

Meanwhile, operations and security teams need to retain those same logs longer-term for forensic investigation purposes. Let's say there's an uptick in payment errors indicating potentially fraudulent activity. The errors are coming from a specific user account with a transaction history spanning several months. By keeping their transaction logs in the Flex Tier for up to 15 months, these teams are able to store these logs at cost-effective rates without compromising their ability to quickly investigate incidents like this one.

What's more, in order to comply with regulatory requirements, the organization must retain these same logs for up to seven years. By enabling Flex Frozen, teams can ensure compliance and keep these logs fully searchable in Datadog, without any need for rehydration or external storage, via Archive Search.

Or, consider a video streaming platform that generates terabytes of CDN logs every day. Despite the volume of these logs, network analysts may need to query them only a few times a day. By keeping them in Flex Tier these teams can instantly query their network logs without incurring the typical costs of indexing such massive volumes of data.

Lastly, consider a financial services organization that needs to keep track of employee actions in its cloud environment for auditing purposes. By storing its user activity logs from CloudTrail in both the Flex and Frozen Tiers, this organization ensures a fast response to regulatory requests, without requiring context switching between different tools. Since these logs won't be needed for DevOps incidents, Standard Indexing can be skipped for improved cost control.

Rein in costs without sacrificing visibility

Flex Logs provides log retention at a commodity storage price point—$0.05 per million events per month—without impeding querying. It allows you to allocate a fixed level of compute capacity to individual teams within your organization according to their needs at a fixed monthly rate. With Flex Tier storage, you can retain logs for three, six, or fifteen months. With Frozen Tier, you can extend that retention up to seven years while maintaining full searchability—ideal for organizations with compliance, auditing, or forensic investigation needs. And because Flex Frozen is fully managed by Datadog, it eliminates the overhead of maintaining external object storage and switching between tools.

Flex Logs enables you to rapidly search any and all of your logs in one place without ballooning storage costs. The Log Explorer allows you to toggle between searching only standard-tier, fully indexed logs (e.g., frequently queried data such as application logs) and expanding your search to include Flex Logs, while Frozen Tier can be directly searched in Datadog via Archive Search. As a result, Flex Logs allows you to avoid the logistical overhead of maintaining additional tools, writing complex query statements, or performing targeted re-indexing operations. It also allows for smooth correlation with your other data in Datadog, including metrics, traces, and more.

Maximize the value of your logs

Flex Logs provides organizations with a log management solution whose costs won't multiply when their storage volumes do. It enables comprehensive observability at scale, giving organizations granular control over fast-growing volumes of logs and ensuring teams quick, consistent access to the data they need.

Datadog users can learn more in our docs to get started with Flex Logs today. Sign up for the preview of Flex Frozen and Archive Search to get early access.

If you're brand new to Datadog, you can sign up for a 14-day free trial.