Lead Compute Platform Engineer

About this role

<p><strong>WHO WE ARE&nbsp;</strong></p> <p>Zeta Global (NYSE: ZETA) is the AI-Powered Marketing Cloud that leverages advanced artificial intelligence (AI) and trillions of consumer signals to make it easier for marketers to acquire, grow, and retain customers more efficiently. Through the Zeta Marketing Platform (ZMP), our vision is to make sophisticated marketing simple by unifying identity, intelligence, and omnichannel activation into a single platform – powered by one of the industry’s largest proprietary databases and AI. Our enterprise customers across multiple verticals are empowered to personalize experiences with consumers at an individual level across every channel, delivering better results for marketing programs. Zeta was founded in 2007 by David A. Steinberg and John Sculley and is headquartered in New York City with offices around the world. To learn more, go to <a href="https://www.zetaglobal.com">www.zetaglobal.com</a>.</p> <p><strong><span data-contrast="auto">The Role</span></strong><span data-ccp-props="{}">&nbsp;</span></p> <p><span data-contrast="auto">As a Senior Compute Platform Engineer on the Data Platform team, you will be part of the group responsible for building and running the data infrastructure that powers Zeta's platform, including Zeta’s&nbsp;lakehouse&nbsp;platform. The Data Platform team delivers reliable, cost-efficient data processing that scales with growth — so engineers and analysts can focus on business problems instead of wrestling with infrastructure.</span><span data-ccp-props="{}">&nbsp;</span></p> <p><span data-contrast="auto">The Data Platform team&nbsp;empowers data scientists and data engineers through a Scala data warehouse framework designed for data modeling and the creation of type-safe and easily testable batch jobs. The team also orchestrates a compute environment that simplifies the execution of Spark jobs,&nbsp;facilitating&nbsp;the analysis of terabytes of data with ease and cost-effectiveness. The runtime environment is based on Hadoop clusters on EC2 spot instances, dynamically scaling up as&nbsp;required, ensuring high resource&nbsp;utilization&nbsp;and stability.</span><span data-ccp-props="{}">&nbsp;</span></p> <p><span data-contrast="auto">Our team&nbsp;is responsible for&nbsp;the development and deployment of the company's next-generation technology, closely collaborating with the company's other branches in&nbsp;San Francisco&nbsp;and Berlin.&nbsp;You will help grow the team through mentoring, code reviews, and knowledge sharing so others can work effectively with our stack.</span><span data-ccp-props="{}">&nbsp;</span></p> <p><strong><span data-contrast="auto">Essential Responsibilities:</span></strong><span data-ccp-props="{}">&nbsp;</span></p> <p><span data-contrast="auto">As a Senior Compute Platform Engineer, your responsibilities&nbsp;will include:</span><span data-ccp-props="{}">&nbsp;</span></p> <ul> <li data-leveltext="" data-font="Symbol" data-listid="4" data-list-defn-props="{"335552541":1,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" data-aria-posinset="1" data-aria-level="1"><span data-contrast="auto">Operating and implementing new features in the framework and the services managed by the team</span><span data-ccp-props="{}">&nbsp;</span></li> </ul> <ul> <li data-leveltext="" data-font="Symbol" data-listid="4" data-list-defn-props="{"335552541":1,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" data-aria-posinset="2" data-aria-level="1"><span data-contrast="auto">Engaging with data scientists and&nbsp;identifying&nbsp;ways of improving their experience and productivity when using our framework and services</span><span data-ccp-props="{}">&nbsp;</span></li> </ul> <ul> <li data-leveltext="" data-font="Symbol" data-listid="4" data-list-defn-props="{"335552541":1,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" data-aria-posinset="3" data-aria-level="1"><span data-contrast="auto">Identifying&nbsp;and implementing computational improvements</span><span data-ccp-props="{}">&nbsp;</span></li> </ul> <ul> <li data-leveltext="" data-font="Symbol" data-listid="4" data-list-defn-props="{"335552541":1,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" data-aria-posinset="4" data-aria-level="1"><span data-contrast="auto">Making improvements and/or fixing bugs by&nbsp;submitting&nbsp;patches to&nbsp;open-source&nbsp;projects in our stack, such as Spark,&nbsp;Iceberg,&nbsp;Hadoop, Airflow, and many others</span><span data-ccp-props="{}">&nbsp;</span></li> </ul> <ul> <li data-leveltext="" data-font="Symbol" data-listid="4" data-list-defn-props="{"335552541":1,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" data-aria-posinset="5" data-aria-level="1"><span data-contrast="auto">Acting as an&nbsp;expert on Spark and Scala</span><span data-ccp-props="{}">&nbsp;</span></li> </ul> <ul> <li data-leveltext="" data-font="Symbol" data-listid="4" data-list-defn-props="{"335552541":1,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" data-aria-posinset="6" data-aria-level="1"><span data-contrast="auto">Mentoring engineers (including through pairing, design discussions, and knowledge-sharing sessions) and helping raise the bar on quality, reliability, and good practices across the team</span><span data-ccp-props="{}">&nbsp;</span></li> </ul> <p><span data-ccp-props="{"335559685":720}">&nbsp;</span></p> <p><strong><span data-contrast="auto">Desired Characteristics:</span></strong><span data-ccp-pro