Staff Data Engineer, Data Platform (Lakehouse & Streaming)
- Seniority
- Staff Principal
- Location
- United States
- Posted
- 19d ago
pythonjavaawsdockerterraformlambdadatabrickssparkairflowdbtllmci/cd
<p class="font-claude-response-body break-words whitespace-normal leading-[1.7]">If you want to own the data infrastructure that powers real-time pricing decisions for thousands of hotels worldwide — and you're energised by the idea of doing it in an engineering culture where AI isn't a future ambition but how work actually gets done today — this is the role. You'll be the technical authority on Duetto's data lakehouse, driving everything from pipeline architecture and data quality to the shift from batch to near-real-time streaming.</p> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"><strong>What Makes Us Different?</strong></p> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]">Duetto is the hospitality industry's leading revenue management platform, founded in 2012 by former Wynn Resorts executives who knew the industry needed better technology. We built the world's first Revenue & Profit Operating System — a suite of tools (GameChanger, ScoreBoard, BlockBuster, Advance and more) that goes beyond room pricing to give hotels, resorts and casinos a complete picture of their revenue and profitability. Trusted by clients ranging from independent boutique hotels to global chains, we've been named the #1 Revenue Management Software by HotelTechAwards four years running and the #1 Best Place to Work in Hotel Tech in 2025. Backed by GrowthCurve Capital since 2024, we're accelerating our investment in AI — and we're genuinely passionate about the industry we serve. We build products we're proud of, for customers we care about.</p> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"><strong>What You'll Be Doing</strong></p> <ul class="[li_&]:mb-0 [li_&]:mt-1 [li_&]:gap-1 [&:not(:last-child)_ul]:pb-1 [&:not(:last-child)_ol]:pb-1 list-disc flex flex-col gap-1 pl-8 mb-3"> <li class="whitespace-normal break-words pl-2">You'll own the design, performance, and reliability of Duetto's data lakehouse — evolving the Python/PySpark pipeline framework across a bronze → silver → gold architecture on AWS, including Glue jobs, Iceberg MERGE operations, schema evolution, and partitioning strategies.</li> <li class="whitespace-normal break-words pl-2">You'll architect the shift from batch to near-real-time streaming, building SQS-driven stream pipelines with Iceberg sinks and expanding ingestion, normalisation, and analytics layers across the full lakehouse.</li> <li class="whitespace-normal break-words pl-2">You'll drive data quality and governance at scale — extending the Great Expectations framework, leading adoption of data contracts to formalise schemas between producers and consumers, and owning the Athena SQL layer that analysts and product teams depend on.</li> <li class="whitespace-normal break-words pl-2">You'll strengthen observability and reliability through Datadog, Sentry, and Sumo Logic, while optimising Glue job performance — worker sizing, DPU allocation, Spark tuning, and cost management.</li> <li class="whitespace-normal break-words pl-2">You'll build and maintain shared internal Python libraries published to JFrog, and drive improvements to GitHub Actions, Docker-based testing, and CI/CD deployment workflows.</li> <li class="whitespace-normal break-words pl-2">You'll work AI-first every day — using Claude Code and MCP tools in your regular workflow, and contributing to AI-assisted pipeline generation, schema inference, and automated data quality alongside a custom multi-agent system with 17 specialised agents.</li> </ul> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"><strong>What We're Looking For</strong></p> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"><em>You may be a good fit if you have:</em></p> <ul class="[li_&]:mb-0 [li_&]:mt-1 [li_&]:gap-1 [&:not(:last-child)_ul]:pb-1 [&:not(:last-child)_ol]:pb-1 list-disc flex flex-col gap-1 pl-8 mb-3"> <li class="whitespace-normal break-words pl-2">7+ years building production data systems in Python</li> <li class="whitespace-normal break-words pl-2">Deep expertise in PySpark and distributed data processing — Glue, EMR, or Databricks</li> <li class="whitespace-normal break-words pl-2">Strong experience with lakehouse architectures: Iceberg, Delta Lake, or Hudi on S3</li> <li class="whitespace-normal break-words pl-2">Production experience with Airflow or a comparable workflow orchestrator</li> <li class="whitespace-normal break-words pl-2">Solid AWS production experience across S3, Glue, Athena, Lambda, and SQS</li> <li class="whitespace-normal break-words pl-2">A track record of improving data quality, governance, and pipeline reliability at scale</li> </ul> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"><em>Strong candidates may also have:</em></p> <ul class="[li_&]:mb-0 [li_&]:mt-1 [li_&]:gap-1 [&:not(:last-child)_ul]:pb-1 [&:not(:last-child)_ol]:pb-1 list-disc flex flex-col gap-1 pl-8 mb-3"> <li class="whitespace-normal break-words pl-2">Working knowledge of Java for reading upstream systems</li> <li class="whitespace-normal break-words pl-2">Experience with Trino or Presto for interactive SQL analytics at scale</li> <li class="whitespace-normal break-words pl-2">Experience with dbt for data transformation and modelling</li> <li class="whitespace-normal break-words pl-2">Familiarity with Great Expectations or similar data quality frameworks</li> <li class="whitespace-normal break-words pl-2">Genuine interest in AI-assisted development and LLM-based tooling</li> <li class="whitespace-normal break-words pl-2">Familiarity with hospitality data — reservations, rates, inventory, demand signals</li> </ul> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"><strong>Why Duetto?</strong></p> <ul class="[li_&]:mb-0 [li_&]:mt-1 [li_&]:gap-1 [&:not(:last-