Technical Director, Data Engineering
- Seniority
- Director Plus
- Location
- Vancouver, British Columbia, Canada
- Posted
- 6d ago
typescriptpythongojavapostgresmongodbelasticsearchdynamodbawskubernetesserverlessspark
<p><span data-olk-copy-source="MessageBody">Diligent is looking for a Technical Director of Data Engineering to help design and build the next generation of data and AI infrastructure powering our products and internal platforms.</span></p> <p>This is not a traditional management role. We are looking for an experienced builder; someone who still enjoys writing code, debugging distributed systems, evaluating frameworks, and getting hands-on with architecture and implementation.</p> <p>You will work across large-scale ingestion pipelines, search systems, AI/LLM infrastructure, event-driven architectures, vector databases, analytics platforms, and real-time data processing. You should be equally comfortable discussing high-level architecture with senior leadership and diving into a failing Kubernetes pod or optimizing a Spark job.</p> <p>The ideal candidate has strong opinions informed by real-world experience, understands tradeoffs deeply, and can move quickly without creating unnecessary complexity.</p> <h3>What You’ll Do</h3> <ul> <li> <p>Design and build scalable data platforms and distributed processing systems</p> </li> <li> <p>Develop modern ingestion, transformation, and retrieval pipelines for structured and unstructured data</p> </li> <li> <p>Build systems supporting AI/LLM applications, semantic search, RAG pipelines, vector search, and agentic workflows</p> </li> <li> <p>Work hands-on with engineering teams to implement production-grade solutions rather than producing slideware</p> </li> <li> <p>Evaluate and standardize frameworks, tooling, and infrastructure patterns across teams</p> </li> <li> <p>Improve performance, reliability, observability, and cost efficiency of data systems</p> </li> <li> <p>Partner with product and platform engineering teams to accelerate delivery of AI-native capabilities</p> </li> <li> <p>Drive pragmatic engineering decisions balancing speed, maintainability, and operational simplicity</p> </li> <li> <p>Mentor engineers technically through design reviews, architecture guidance, and pair debugging</p> </li> <li> <p>Help establish engineering standards around CI/CD, testing, data quality, monitoring, and operational excellence</p> </li> </ul> <h3>What We’re Looking For</h3> <h3>Strong Hands-On Engineering Experience</h3> <p>Candidates should have significant real-world experience building and operating production systems using many of the following:</p> <h4>Data & Distributed Systems</h4> <ul> <li> <p>Airflow</p> </li> <li> <p>Elasticsearch / OpenSearch</p> </li> <li> <p>Vector databases and semantic retrieval systems</p> </li> <li> <p>MongoDB, PostgreSQL, DynamoDB, or similar platforms</p> </li> </ul> <h4>Cloud & Infrastructure</h4> <ul> <li> <p>AWS</p> </li> <li> <p>AWS CDK</p> </li> <li> <p>Serverless architectures</p> </li> <li> <p>Distributed observability and monitoring stacks</p> </li> </ul> <h4>AI / Search / Modern Data Applications</h4> <ul> <li> <p>LLM integration patterns</p> </li> <li> <p>RAG architectures</p> </li> <li> <p>Embeddings and vector search</p> </li> <li> <p>MCP servers and AI orchestration frameworks</p> </li> <li> <p>LangChain, LlamaIndex, DSPy, or similar ecosystems</p> </li> <li> <p>AI evaluation, tracing, and observability tooling</p> </li> <li> <p>Search relevance and ranking systems</p> </li> </ul> <h4>Backend Engineering</h4> <ul> <li> <p>Python strongly preferred</p> </li> <li> <p>Experience with Java, Go, or TypeScript is a plus</p> </li> <li> <p>API design and distributed service architectures</p> </li> <li> <p>Event-driven and asynchronous systems</p> </li> </ul> <h2>The Right Candidate</h2> <ul> <li> <p>Still enjoys building and debugging systems directly</p> </li> <li> <p>Has strong technical depth, not just architectural vocabulary</p> </li> <li> <p>Comfortable operating in ambiguity and fast-moving environments</p> </li> <li> <p>Understands how to simplify systems instead of endlessly abstracting them</p> </li> <li> <p>Has experience modernizing legacy platforms and evolving architectures incrementally</p> </li> <li> <p>Can distinguish between engineering fundamentals and hype cycles</p> </li> <li> <p>Values shipping working systems over theoretical perfection</p> </li> </ul> <h3>What Success Looks Like</h3> <ul> <li> <p>Engineering teams can move faster because the underlying platforms are reliable and scalable</p> </li> <li> <p>AI and data systems become production-grade rather than experimental prototypes</p> </li> <li> <p>Infrastructure costs and operational complexity are reduced through better architecture</p> </li> <li> <p>Search, ingestion, and retrieval systems improve significantly in performance and relevance</p> </li> <li> <p>Teams adopt consistent, maintainable technical patterns without excessive bureaucracy</p> </li> </ul> <h3>Nice to Have</h3> <ul> <li> <p>Experience with large-scale news, document, or regulatory data pipelines</p> </li> <li> <p>Experience with search relevancy tuning and semantic retrieval</p> </li> <li> <p>Exposure to governance, compliance, risk, or enterprise SaaS platforms</p> </li> <li> <p>Familiarity with modern AI engineering workflows and agentic systems</p> </li> <li> <p>Experience supporting both startup-speed execution and enterprise-scale operations</p> </li> </ul> <div> </div><div class="content-pay-transparency"><div class="pay-input"><div class="title">Pay Range</div><div class="pay-range"><span>$210,000</span><span class="divider">—</span><span>$250,000 CAD</span></div></div></div><div class="content-conclusion"><p> </p> <p><strong>About Us</strong></p> <p><span data-olk-copy-source="MessageBody">Diligent is the AI leader in governance, risk and compliance (GRC) SaaS solutions, helping more than 1 million users and 700,000 board members to clarify risk and elevate governance. The Diligent One Platform gives practitioners, the C-Suite and the board a consolidated view of their entire GRC practice so they can more effectively manage risk, build grea