May 7, 2026 —Jonathan Almanza

USD 1,600 a month for a pipeline that ran once a day

A client was paying for Redshift, Glue and a row of Lambdas to serve 30 GB queried once a day. We switched almost everything off, made PostgreSQL the orchestrator, and the bill dropped 91%.

USD 1,600 a month for a pipeline that ran once a day

A large client had a problem costing them USD 1,600 a month. They asked us to review their infrastructure and run a full audit. That's where it started.

The problem we inherited

  • A pipeline that took 1.5 to 2 hours to make data ready for Power BI
  • Queries running 8 to 10 minutes
  • 30 GB of data growing without order
  • Several Lambdas with no CI/CD, hardcoded credentials, and no failure alerts
  • A frustrated team, because "the data is always late"
APIsSP / Ads Lambdas×N no CI/CD Glue ETLall in memory Redshift30 GB Power BI1 query/day ■ BEFORE — the inherited architecture USD 1,600/mes 1.5–2 h pipeline 8–10 min queries 30 GB ×3 Data duplicated — even triplicated — across the database, S3 and intermediate processes. No state tracking: a failure surfaced only when the report didn't arrive.

What we found

  • Redshift holding just 30 GB, queried once a day from Power BI
  • Glue ETL loading everything into memory to do a merge
  • Data duplicated — even triplicated across the database, S3 and intermediate processes
  • No state tracking: when a process failed, nobody knew until the report didn't arrive

The decision

Simplify the architecture completely: switch off Redshift and Glue, and center the whole flow on PostgreSQL over EC2.

■ AFTER — one orchestrator, zero clutter APIsSP / Ads PostgreSQL / EC2central orchestrator · idempotency · retries Materialized viewsfor the dashboards Power BI1 query/day S3raw zone Lambdaa single trigger USD 130/mes pipeline in minutes 1–5 s queries 5 GB -91%

What we built

  • PostgreSQL as the central orchestrator
  • A report_status table with idempotency keys
  • A single Lambda as the S3 trigger
  • Automatic schema and table generation
  • Materialized views for the dashboards
  • Automatic retries and controlled reprocessing
  • CI/CD for the orchestrator

The result

Before After
USD 1,600/mo USD 130/mo
2-hour pipeline minutes
8–10 min queries 1–5 seconds
30 GB 5 GB
Unordered growth predictable architecture

The takeaway

You don't need an "enterprise" architecture for everything. Sometimes the best solution isn't adding more services: it's understanding the problem and designing something simpler, cheaper and more efficient. The right tool isn't the most expensive one — it's the one that actually solves your need.

Have you seen over-engineered architectures for simple problems?

Back to all posts