Natural Intelligence gains pinpoint control over cloud data costs and anomaly detection

The hurdles: navigating complex cloud data cost management

Natural Intelligence faced several challenges in managing and understanding their cloud expenditures, particularly concerning their data services:

Tracking and allocating Snowflake data warehouse costs

Their Snowflake data warehouse, crucial for storing all customer-related data and events, powers numerous internal and external data applications used by BI, Sales, and Product teams. Over time, Snowflake's costs escalated significantly, withconsiderable variances across departments. The company urgently needed a detailed cost breakdown to identify specific cost drivers and manage these expenses more effectively.

Achieving real-time cost anomaly detection and alerting

Natural Intelligence aimed to implement a real-time anomaly alerting system for their cloud costs. However, their existing FinOps and observability tools were generating excessive, often irrelevant, alerts due to generic algorithms. They sought a more sophisticated anomaly detection system capable of aligning with their organizational structure and delivering pertinent, timely alerts to the correct stakeholders.

Monitoring Amazon EMR job costs

With multiple Amazon EMR clusters running for data processing, ETL, and machine learning training jobs initiated by data science, DevOps, and engineering teams, the volume of daily jobs had surged. Simple total billing reports were no longer sufficient; they required deep visibility into the cost breakdown of individual jobs and a method to track the cost of failed jobs.

Overcoming legacy platform limitations

The company had been using a legacy Cloud Cost Management platform that provided only limited, high-level visibility into costs, primarily at the infrastructure level, hindering detailed analysis.

The breakthrough: Pelanor's tailored solution

To address these challenges, Pelanor collaborated closely with Natural Intelligence’s DevOps and FinOps teams, delivering immediate value through a rapid Proof-of-Value (PoV). The key steps included:

Seamless platform and service integrations

The DevOps team quickly configured billing data imports from AWS and installed Pelanor’s advanced, dedicated sensors for Snowflake and EMR.

Strategic dimension configuration

Ensure accurate cost categorization, the FinOps team defined department-based dimension groups within Pelanor, including BI, Data Infrastructure, and Operations.

Intelligent anomaly monitor configuration

The FinOps team set up sophisticated anomaly detection monitors in Pelanor. This involved specifying precise filter conditions, trigger thresholds, and notification rules to guarantee that alerts reached the appropriate stakeholders efficiently.

The transformation: measurable results and empowered FinOps

The implementation of Pelanor's platform brought about significant improvements and tangible results for Natural Intelligence:

Full Visibility into Snowflake & EMR costs at the query/job level

After integrating their AWS and Snowflake accounts into Pelanor, Natural Intelligence gained unprecedented insight into the costs associated with each specific query and job. They established comprehensive reports and dashboards to analyze:

• Costs by individual database (Snowflake) or cluster (EMR).
• Costs per specific query (Snowflake) or individual job (EMR).
• Costs attributed to the user executing the query or job. This granular data empowered them to swiftly investigate and pinpoint the root causes of cost increases, such as inefficient, long-running queries and costly failed data processing jobs.

Customized anomaly monitors with granular, relevant notifications

Natural Intelligence leveraged Pelanor’s anomaly monitors to define highly tailored alerting. They could filter out specific usage types, billing components, and services that did not require monitoring, and set precise dollar and percentage-based thresholds to trigger alerts. Furthermore, they configured distinct recipients for each monitor, ensuring alerts were consistently delivered to the relevant department stakeholders. Consequently, Natural Intelligence dramatically reduced FinOps alert noise, received fewer but more meaningful daily alerts, and ensured that notifications were perfectly aligned with their organizational structure.

The bottom line: mastering cloud data service costs

By implementing Pelanor’s FinOps platform, Natural Intelligence achieved significantly deeper visibility into their cloud data services, enabling far more effective cost tracking and management. Their tailored anomaly detection system markedly enhanced operational efficiency by delivering relevant, actionable alerts while minimizing unnecessary noise. The enhanced insights and customizable reports empowered the company to identify substantial cost-saving opportunities, align cloud expenditures meticulously with business objectives, and drive data-informed decisions across all departments.

This case study clearly demonstrates how an adaptive and intelligent approach to cloud cost management, as provided by Pelanor, can significantly enhance FinOps processes for data-driven organizations operating in complex modern cloud environments.