The Cost of Log Management and Machine Data Solutions
When we talk to implementors of these systems, they have concerns over the ever increasing costs for initial implementation and on-going lifecycle. There is always an upward pressure on data ingest rates as new use-cases are identified and more data needs to be ingested. The costs arise not only from vendor licence fees, but also the infrastructure and storage costs for processing and storing the data. If your company has a multi-year data retention policy then the storage costs alone can be substantial.
Cribl LogStream can help offset increasing licence, infrastructure and storage costs by trimming data before it's ingested. Many 'logs' have fields that are either repeated from one log entry to another and so don't add value, or key-value pairs are always null (also no value). If such data is stripped before ingest then you do not have to licence for it, build infrastructure to process it, or provision storage to keep it for a lengthy retention period. It is usually quite easy to trim 25~30% of such data.
Another factor to consider is the carbon footprint of the solution, whether it be on-prem or in the cloud. Given that most organisations have (or are in the process of) setting carbon targets, the carbon footprint of the IT infrastructure and long term storage of large volumes of data should be considered.
We have developed a calculator to help provide some insight into the savings that could be made. It's based on publicly available Splunk and Cribl price data, with links to background supporting information. Of course, you may have negotiated custom pricing for your implementation so the pricing factors can be adjusted. We would welcome your feedback as to whether this calculator is useful. Or, if you have any questions on its usage, we would be pleased to discuss.
Filter, Transform, and Route Log Data
Cribl LogStream processes log data before you pay to analyze it. LogStream helps you discern which data you need to send to an analytics tool to analyze now; which logs can be aggregated into metrics; which data should be stored and analyzed later if needed; and which elements of data should be dropped altogether. LogStream allows you to implement an observability pipeline which helps you parse, restructure, and enrich data in flight. Get the right data, where you want, in the formats you need.
- Cribl LogStream, designed by ex-Splunkers, optimized for Splunk
- Reduce data volumes to manage licence costs and/or avoid upgrades
- Route data to, and replay from, low cost S3 Storage
- Clone and route data to multiple destinations
- Transform, enrich and secure data before ingest