Lead Site Reliability Engineer - Imunify Reliability Platform (remote-only)

About this role

The problem you'd own Imunify360 is a multi-layer Linux server security suite — WAF, IDS/IPS, malware scanning and cleanup, proactive defence, patch management, reputation — running as an agent on hundreds of thousands of customer servers, backed by a cloud estate of scanning, correlation and signature-delivery services on our own bare metal. Roughly 70 components currently ship without a defined service level indicator. Some are internal services we can scrape. Many are agent-side subsystems running on machines we do not own, reporting through a heartbeat we designed for something else. There is monitoring, and there are dashboards, and there is no coherent answer to the question "is this component doing its job right now, and how would we know if it stopped?". We know the cost of that gap precisely, because we recently paid it: a security control was silently disabled across a large fraction of the fleet for 61 days. Every dashboard was green. The telemetry reported a ruleset version but not whether the control that consumed it was switched on, so a configuration change was indistinguishable from a broken updater. Three independent safety mechanisms existed and all three were gated behind the same condition that caused the failure. Your job is to make that class of failure detectable in hours instead of months, across the whole product line, and to build the system that keeps it detectable as the product changes. This is a greenfield charter inside a brownfield estate. You are not inheriting an SRE team, an SLO framework or a paging culture. You are defining them, with the engineering leads, and then making them stick. What you'll do 1. Define what "working" means for ~70 components Run SLI definition with squad leads and senior engineers. You facilitate and hold the standard; the owning squad signs the SLI. Build the taxonomy this product actually needs, which is broader than availability and latency: Service SLIs — availability, latency, error rate for cloud-side services. Fleet SLIs — heartbeat reachability, version and configuration convergence across the installed base. Control-efficacy SLIs — the differentiator. What fraction of protected units have the control effectively enabled and current , not merely installed. Ruleset generation drift, signature age, scan coverage, enforcement-mode distribution. Delivery SLIs — artifact publish success, rule-to-fleet lead time, hotfix time-to-convergence. Pipeline SLIs — ingest lag, verdict latency, queue age, backlog burn. Enforce one non-negotiable design rule: an SLI must be measurable from outside the gate of the thing it measures. If the control being off also switches off the signal that would tell you it is off, the SLI is invalid. This is the lesson of the incident above and it is the reason this role exists. Attach an SLO, an error budget and an owning squad to each. Tiering is expected — not every component earns a 99.9% target or a pager. 2. Build the collection system Design and build the pipeline that gets these indicators off the fleet and into a queryable store: push-based, sampled, privacy-constrained, and with a cardinality budget you set and defend. Extend agent-side and service-side instrumentation where the signal does not exist yet, in Python, Go and Rust, working with the owning squads. Consolidate the current sprawl of dashboards, ad-hoc queries and reporting paths into a defensible set of instruments, and retire what does not earn its keep. 3. Build alerting and alert management Symptom-based, SLO-anchored alerting with multi-window burn-rate semantics. Not threshold soup. A three-tier taxonomy — page / ticket / dashboard — with an explicit rule for what is allowed to page a human at 03:00. Every alert ships with an owner, a runbook and a documented failure mode, or it does not ship. Alert hygiene as a standing practice: quarterly review, deletion counted as a win, actionable-rate tracked. A persistent inability to perform a security-relevant refresh should page. It currently logs a warning. 4. Build escalation Component → owning squad ownership map, kept current, machine-readable, and wired into routing so an alert reaches the right seven people rather than a shared channel. Severity matrix, acknowledgement SLAs, follow-the-sun rota design across UTC−5 … UTC+8, and clean handoff protocol. Incident command practice and blameless postmortems within 24 hours. We already do postmortems and do them honestly, including publicly retracting our own wrong findings; you raise the floor on the mechanical parts — timelines, ownership, action-item follow-through. Design the escalation system so that squads carry their own pagers. You build and operate the platform and coach on the practice; you are not the buffer that absorbs everyone else's alerts. Requirements What you'll bring Required (Must-haves): Substantial production-engineering or SRE experience, including at least one environment where you defined the SLO framework rather than inherited it. We will ask you to walk through SLIs you personally wrote and how you negotiated them with resistant teams. Strong Python. Comfortable reading and modifying Go or Rust — our agents are written in them and instrumentation lands there. Deep practical grip on time-series and event telemetry at scale: Prometheus/OpenMetrics, Grafana, an Alertmanager-class routing layer, and a columnar store for high-cardinality fleet data (ClickHouse or equivalent). Distributed systems debugging on bare metal and long-lived hosts. Most of this estate is not Kubernetes, and the reflexes that assume an orchestrator will not transfer cleanly. Configuration management and CI at production scale — Ansible, GitLab CI, Jenkins or close equivalents. The judgement to design measurement for machines you do not own and cannot scrape: push telemetry, sampling, clock skew, partial reporting, and the privacy constraints that come with running on a customer's server. Written communication that holds up async. This rol