Senior Principal Site Reliability Engineer

About the company

Bybit is a major cryptocurrency exchange that is building MyBank, a retail-banking service meant to blend crypto trading with traditional financial services.

About this role

<div class="content-intro"><p><strong>About Us</strong></p> <div data-page-id="HladdPLXIoNz4nxkqkCuk4rAs8v" data-lark-html-role="root" data-docx-has-block-data="false"> <div class="ace-line ace-line old-record-id-XCikdRR2xoZIjdxtCKcuOflrs8d">Established in 2018, Bybit is one of the world’s leading cryptocurrency exchanges and digital financial platforms, serving over 80 million users across more than 200 countries and regions. Powered by world-class technology and a user-first mindset, Bybit delivers a seamless ecosystem across trading, payments, wealth management, custody, institutional services, and Web3 — connecting users to the future of digital finance.</div> <div class="ace-line ace-line old-record-id-IF06dI0MPoBwYfxbZ4zui13HsoC">&nbsp;</div> <div class="ace-line ace-line old-record-id-X5EbdOwvMoYuOmxQBbuueUEDsDd">Our core values define how we build. We listen, care and improve to create products and experiences that put users first. Backed by a global team of ambitious builders, problem-solvers, and innovators, we foster a high-performance and fast-moving environment where talent is empowered to drive real impact at the global scale. Supported by 24/7 multilingual customer service and a strong commitment to innovation, we are shaping the future of finance through technology, collaboration, and bold execution.</div> <div class="ace-line ace-line old-record-id-LtVud2dO7oglqJxxZoOucqjJsPc">&nbsp;</div> <div class="ace-line ace-line old-record-id-RMHkdm4bPo66l3xMVO8ui1o8sle">Today, Bybit is recognized as one of the most trusted and transparent platforms in the digital asset industry, continuing to expand its global presence while building the infrastructure for the next generation of financial services.</div> </div></div><div data-zone-id="0" data-line-index="0" data-line="true">Core Responsibilities</div> <div data-zone-id="0" data-line-index="1" data-line="true">&nbsp;</div> <div data-zone-id="0" data-line-index="2" data-line="true"><strong>Chaos Engineering Platform Architecture &amp; Development (50%)</strong></div> <div data-zone-id="0" data-line-index="3" data-line="true">&nbsp;</div> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Design and build an enterprise-grade chaos engineering platform supporting multi-cluster (K8s + EC2 hybrid), multi-region, and multi-environment (testnet/mainnet) deployments</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Core capability development:</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Fault Injection Engine: Pod-level / Node-level / AZ-level fault simulation, network latency / packet loss / partition, dependency timeout / error injection</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Production Safety Assurance: Blast radius control, one-click Kill Switch, automatic rollback, real-time impact monitoring</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Traffic Isolation: Experiment traffic tagging and isolation to ensure fault injection does not impact real users</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Fault Isolation: Precise impact scoping at service / cluster / AZ granularity</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Design experiment orchestration capabilities supporting complex fault scenario composition (e.g., simultaneous network latency + downstream timeout + cache invalidation)</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Deep integration with existing monitoring, alerting, and SLO systems to achieve an automated closed loop: inject fault → observe impact → determine pass/fail</li> </ul> <div data-zone-id="0" data-line-index="12" data-line="true">&nbsp;</div> <div data-zone-id="0" data-line-index="13" data-line="true"><strong>Production Resilience Validation Framework (30%)</strong></div> <div data-zone-id="0" data-line-index="14" data-line="true">&nbsp;</div> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Define safety standards and approval workflows for mainnet fault injection</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Design and drive routine chaos experiments:</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Daily patrol-level experiments: Low-risk experiments executed automatically on a daily/weekly basis</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Periodic validation experiments: Monthly/quarterly resilience verification of critical paths</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Large-scale drills: Cross-AZ / cross-region disaster recovery failover validation</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Establish a resilience scoring system to quantify system health based on experiment results</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Deliver improvement recommendations and drive business teams to remediate identified weaknesses</li> </ul> <div data-zone-id="0" data-line-index="22" data-line="true">&nbsp;</div> <div data-zone-id="0" data-line-index="23" data-line="true">3. Technology Selection &amp; Team Enablement (20%)</div> <div data-zone-id="0" data-line-index="24" data-line="true">&nbsp;</div> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Evaluate and select the technology foundation (Chaos Mesh / Litmus / custom components — hybrid strategy)</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Develop chaos engineering best practices and playbooks to enable SRE teams and application developers</li> </ul> <ul> <li class="temp-li bullet1" data-line="true" data-list="bullet1">Mentor and grow the team (2–3 engineers) in chaos engineering capabilities</li> </ul> <ul