Open-Source is the Way Out of a Tech Downturn

It is no secret that 2022 was a brutal year for the tech industry. A combination of multiple factors has been a vicious one-two punch for the destruction of shareholder value. These include historic increases in the federal funds rate, margin compression, and companies over-hiring at the peak of the tech job market in 2021 with the expectation of continued pandemic demand.

While the market might stabilize itself beyond this year, the underlying inflationary dynamic is unlikely to change.

First, the labor market has dealt with an abundance of retirements in 2021 creating a supply/demand gap that will take years to close. Secondly, our natural resources sector is not only undersupplied in oil, but also basic materials like copper, zinc, and lithium. Lastly, there has been a secular shift to a more generous fiscal policy to meet populist political demands here and abroad.

The next decade simply will likely not mirror the last in terms of steady-state inflation.

The future of startup tech companies

So, what does this mean for your typical tech startup in 2023 and beyond?

• Continued wage pressure: Talent will continue to be scarce and begin to flock to non-tech opportunities in other industries, like finance, that benefit in inflationary environments.
• Higher financing hurdles: As venture capitalists (VC) exit multiples become sustainably lower, their investments will recalibrate tighter to guarantee a decent internal return rate (IRR) on each deal made.
• Higher cost of goods sold (COGS) as cloud providers pass on costs: Inverting the debatable old wisdom that the commoditized side of the cloud "always gets cheaper", but also exacerbated by software vendors exploiting pricing power to prop up their bottom lines

The silver lining of all of this is necessity is the mother of invention. We have seen it happen before in the tech industry when companies start to challenge the conventional wisdom that is more often than not foolish and occasionally creates incredible value as a result.

Post-dot-com and the Rebuilding of Amazon

The most obvious example of this is Amazon during the dot-com era. Considering where Amazon is now, it is crazy to look back in time and see how close they were to becoming bankrupt when the dot-com bubble burst.

With a major slowdown in demand, which increased their burn rate, and their inability to raise capital in the public equity market and maxed-out debt markets, the company nearly ran out of cash.

To say times were hard for Amazon would have been an understatement. Their stock was down nearly 90%, and the company could not make a profit. Drilling down into the numbers, Amazon's biggest COGS line-item was its data center, which was primarily due to the cost of licensing Sun server software. Amazon quickly realized the best way to right-size their spending was to swap out Sun for Linux, then a largely unproven but promising open-source operating system.

At the time this was a radical decision because the business consensus said Sun was best-in-breed (no one got fired for choosing Sun). However, a technical comparison would have shown that they were perfectly substitutable with some upfront engineering effort. Not only did Amazon succeed in the switchover, but the investment in running Linux at scale also ended up seeding what is now Amazon Web Services (AWS), arguably the best business in the world.

Coatue Ventures chairman Dan Rose breaks down the above in a 2020 Twitter thread.

Similarly, Google bucked the trend of relying on IBM and Sun for their infrastructure and instead focused on running large-scale, horizontally-distributed computing on commodity hardware, which has been the strategic bedrock for seminal technologies like Borg, Google File System, MapReduce, BigTable (Amazon also contributed here with dynamo), Spanner, etc.

All this innovation was ultimately born out of the avoidance of cost pressures, and all required rejecting conventional business wisdom they would have been advised upon by the likes of McKinsey or Gartner.

Companies Overspent in 2022

In the dot-com era, pricing power was heavily concentrated among big legacy players and ultimately easy to identify and avoid (although I’m sure that didn’t seem true at the time).

While there are blatantly obvious megascale culprits for cost overruns now (looking at you AWS), the reality is the situation has become far more complex. The omnipresence of SaaS has created a zoo of mini-monopolies each who will increasingly desperately overbill to continue to show growth to the public markets.

One of the more interesting examples here is found in the “modern data stack,” which if you were to go with best-of-breed SaaS would amount to something like $1M/yr+ in SaaS spend, broken down something like:

• Looker - $100,000
• Fivetran - $100,000 - $200,000
• Snowflake - $100,000 to $1.5 million depending on usage
• Astronomer - $25,000 for small use cases
• AWS Sagemaker/Databricks - $100,000 to $1.5 million (most likely on the higher end here.)

If you move to more secondary applications for data observability, data cataloging, and data governance, each could involve contracts upwards of $75,000 or more, depending on the amount of data your organization works with daily.

The truth is there is a similar open-source switch for the data stack, moving onto something like:

• Superset
• Airbyte
• Clickhouse
• Dagster
• Kubeflow / MLFlow
• Datahub

This exact open-source stack can solve a majority of the use cases that the average SaaS stack does, and it is quite cheaper. But, why are companies still hesitant to switch over to open-source?

The switch to open-source tooling in 2023

The most obvious friction to swapping from a standard data SaaS tool to OSS is operational cost. In theory, replacing a 100k Fivetran bill with a cheaper self-hosted Airbyte setup is nice, but if you end up needing to hire a Site Reliability Engineer (SRE) to oversee it at around $150k/yr, you end up wasting more capital.

The math on this changes in two common ways.

• You start to deploy OSS in bundles (e.g., Superset, Airbyte, Airflow all simultaneously,) rather than a la carte, allowing you to drive operational economy of scale.
• The infrastructure underneath matures to the point that the operational burden no longer requires dedicated operational resources.

Previously, organizations steered clear of solving this and just kept paying for their SaaS data stack. Honestly, I can't blame them, since it's nontrivial to solve and it is not a core mandate of their business (part of what we are building at Plural is aimed at solving this exact problem that organizations constantly face).

That said, cloud infrastructure has matured in the last few years and has become much more powerful. The likes of Kubernetes and infrastructure as code tools such as Terraform have decreased the burden involved in self-hosting OSS. We are still in the early innings of realizing the full potential of these revolutions in making distributed systems management nearly painless. Still, there is finally a path to that goal.

The other barrier is similar to the Sun/Linux decision during the dot-com bubble. Open-source almost universally lacks the fit and finish of its commercial peers. This is primarily due to the lack of capital available to open-source projects, which limits their ability to hire designers and product managers to manicure a product to the standards most would expect. It can become even more severe when key features are missing or bugs get left unaddressed.

A fortunate accident of the previous boom is that open-source has become an extremely hot, well-funded asset among VCs. Products like Airbyte and Dagster provide slick user experiences and I expect a new generation of well-capitalized open-source businesses to follow in their path and close that gap.

Surviving the tech downturn

No two companies will be the same in how they manage the next few years, but I do believe some of the more risky, out-of-the-box strategies like embracing a more OSS-first approach to building your tech stack are ultimately going to be winners.

The cost savings are simply too immense to ignore, and the downsides are technically resolvable if there’s the motivation (like sheer necessity) to invest in it. At the very least, we’re motivated to tackle the problem, and hopefully, more join us.