SaaS Observability: Cost vs. Value & Open Source Alternatives
A significant theme in the discussion is the high cost associated with SaaS observability solutions like Datadog, contrasted with the potential cost savings and increased flexibility offered by open-source alternatives. Users pointed out that substantial bills, even in the "five figures," are common, and that companies might be overspending even on these services.
- "We recently did the same, and our Datadog bill was only five figures. We're finding the new stack to not be a poor man's anything, but more flexible, complete and manageable than yet another SaaS." - delichon
- "Crazy crazy they spent so much on observability. Even with DataDog they could've optimized that spend." - asnyder
- "What problems does Datadog solve that you can't solve with cheaper solutions?" - cloudking
- "Does anyone have such an experience with Datadog? A few million wasn't enough to get them to talk about anything, always paid list price and there was no negotiating either when they restructured their pricing." - aeyes
The Appeal of Open Source for Observability
Many users expressed a strong preference for open-source solutions such as Grafana, Prometheus, and ClickHouse (or VictoriaMetrics/VictoriaLogs) for observability. They argue that these stacks are not only more cost-effective but also more flexible and complete than proprietary SaaS offerings. The learning curve associated with these tools is seen as a worthwhile investment, leading to better control and outcomes.
- "With just a little extra learning curve observability is a domain where open source trounces proprietary, and not just if you don't have money to set on fire." - delichon
- "There's also https://openobserve.ai, while not as stable as Grafana/Prometheus/Clickhouse, feels a bit easier to setup and manage. Though has a bit of ways to go, does the basics and more without issue." - asnyder
- "It is not hard to spin up Grafana and VictoriaMetrics (and now VictoriaLogs) and keep them running. It is not hard to build a Grafana dashboard that correlates data across both metrics and logs sources, and alerting functionality is pretty good now." - pphysch
"Money-is-no-object" Culture and its Impact on Engineering Decisions
A recurring concern is how a "money-is-no-object" attitude towards purchasing SaaS solutions can negatively influence engineers' decision-making regarding infrastructure and architecture. This approach can lead to inflated costs, especially for startups, and can also mask underlying inefficiencies or a lack of fundamental engineering skills. The tendency to default to expensive SaaS without proper cost-benefit analysis is seen as a potential trap.
- "I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture." - ljm
- "Coinbase might have been fine blowing 65 mil but take that approach to a new startup and you could trivially eat up a significant amount of runway with it." - ljm
- "I saw this a lot at a previous company. Being able to just "have more Lambdas scale up to handle it" got some very mediocre engineers past challenges they encountered. But it did so at the cost of wasting VAST amounts of money and saddling themselves with tech debt that completely hobbled the company's ability to scale." - viccis
- "I consider cost consideration and profiling to be an essential skill that any engineer working in cloud style environments should have, but it's especially important that a staff engineer or person in a similar position have this skill set and be ready to grill people who come up with wasteful solutions." - viccis
The Trade-off Between SaaS Cost and Engineering Time
The discussion touches upon the economic argument that while SaaS solutions are expensive, they can sometimes be more cost-effective than the "invisible cost of burning engineer time" on rebuilding foundational components. However, a counterpoint is raised that the focus should be on migrating to more cost-effective, manageable solutions rather than accepting prohibitively high SaaS bills.
- "the visible cost of burning runway on a bill is very often far less than the invisible cost of burning engineer time rebuilding undifferentiated heavy lifting rather than working on product/customer needs" - swyx
- "People say this but I wonder about this from time to time. I don't think anyone is asking to rebuild datadog from scratch for your company but surely it's worth it to migrate to something not as expensive even if it takes a bit of elbow grease." - 9283409232
Questionable Justifications for High SaaS Spending
Several comments scrutinize the justifications often presented for high expenditure on SaaS observability tools. Specifically, arguments that tie the SaaS cost directly to revenue saved by preventing outages are met with skepticism, especially when they fail to consider cheaper alternatives or the nuanced reality of revenue loss during downtime.
- "Those are some pretty heroic assumptions. In particular, they assume the only options are Datadog or nothing, when there are far cheaper alternatives like the Prometheus/Grafana/Clickhouse stack mentioned in the article itself." - decimalenough
- "Another assumption that bothers me here is that the $9M in revenue would be completely lost during an outage. I imagine many customers would simply wait until the outage was resolved before performing their intended transactions, meaning far less than $9M would be lost." - passivepinetree
- "am i misunderstanding, or is the author saying it's better to spend $10m than $9m?" - GuinansEyebrows
Datadog's Billing Practices and Contract Negotiation
Specific criticisms were leveled against Datadog's billing practices, with one user suggesting that their default configurations, particularly for on-demand instances, can lead to significantly higher charges than necessary. The difficulty in negotiating contracts, even for substantial amounts, was also highlighted.
- "asnyder: DataDog does lots of bad things with billing where by default, especially with on-demand instances you get charged significantly more than you should as they have (had?) pretty deficient counting towards instance hours and instances."
- "aeyes: Does anyone have such an experience with Datadog? A few million wasn't enough to get them to talk about anything, always paid list price and there was no negotiating either when they restructured their pricing."
The Scale and Startup Context of SaaS Spend
It was noted that while companies like Coinbase might have the scale to warrant significant observability bills, this approach is not sustainable for most startups. The ability to manage and optimize cloud spend and tool adoption is crucial for the runway of new companies.
- "Most startups are not going to have anywhere near the scale to generate anything approaching this bill." - JohnMakin
- "Unless you're in the business of deploying and maintaining production-ready datacenters at scale, it very literally isn't [burning money, referring to cloud spend]." - JohnMakin