Senior Data Engineer

About this role

<p><strong>About SecurityScorecard:</strong></p> <p><span style="font-weight: 400;">SecurityScorecard is the global leader in cybersecurity ratings, with over 12 million companies continuously rated, operating in 64 countries. Founded in 2013 by security and risk experts Dr. Alex Yampolskiy and Sam Kassoumeh and funded by world-class investors, SecurityScorecard’s patented rating technology is used by over 25,000 organizations for self-monitoring, third-party risk management, board reporting, and cyber insurance underwriting; making all organizations more resilient by allowing them to easily find and fix cybersecurity risks across their digital footprint.&nbsp;</span></p> <p><span style="font-weight: 400;">Headquartered in New York City, our culture has been recognized by Inc Magazine as a "Best Workplace,” by Crain’s NY as a "Best Places to Work in NYC," and as one of the 10 hottest SaaS startups in New York for two years in a row. Most recently, SecurityScorecard was named to Fast Company’s annual list of the</span><a href="https://securityscorecard.com/blog/blog-top-innovators-of-the-year/"><span style="font-weight: 400;"> </span><span style="font-weight: 400;">World’s Most Innovative Companies for 2023</span></a><span style="font-weight: 400;"> and to the Achievers 50 Most Engaged Workplaces in 2023 award recognizing “forward-thinking employers for their unwavering commitment to employee engagement.”&nbsp; SecurityScorecard is proud to be funded by world-class investors including Silver Lake Waterman, Moody’s, Sequoia Capital, GV and Riverwood Capital.</span></p> <p><strong>About the Team:<br><br></strong>Be part of an exciting and growing Technology team focused on product development. Our team emphasis is collaboration and shared responsibilities, you will be hands-on involved in the entire Software Development Life Cycle. Our team is growing and so will you! Check out our tech blog below and let us know if you are our next star.<br><br><a href="https://techblog.securityscorecard.com/">https://techblog.securityscorecard.com/</a></p> <p><strong><br>About the Role:<br></strong></p> <p><strong>What you will do</strong></p> <p>We are looking for a Senior Data Engineer who thrives on building commercial products and plays a key role in delivering reliable, scalable, and maintainable applications. You will own projects end-to-end, collaborating with Product Managers to define requirements, implement solutions, and ensure ongoing success post-deployment. You'll focus on simplicity while contributing technical insights to enhance the system's performance, stability, and user experience.</p> <ul> <li>Participate in all stages of the software development life cycle, from design and development to deployment and maintenance.</li> <li>Contribute to the product and technical roadmap by providing data-driven insights and articulating trade-offs between technical options and business goals.</li> <li>Support and mentor junior engineers on the team through code reviews, pairing, and technical guidance.</li> <li>Follow clean code principles, using best practices to write secure, testable, and well-documented software.</li> <li>Work closely with a global team, including engineers and product managers, to build a stable, high-performance platform.</li> <li>Implement comprehensive testing and performance tuning to ensure system stability and uptime.</li> </ul> <p><strong><br>Responsibilities:</strong></p> <ul> <li>Translate requirements, estimate effort, and address or escalate blockers as needed.</li> <li>Write clean, maintainable code that adheres to best practices in readability, design patterns, reusability, and testing.</li> <li>Own end-to-end projects, understanding and contributing to all aspects (infrastructure, application tiers, and data tiers).</li> <li>Continuously monitor performance metrics and recommend improvements or refactors.</li> </ul> <p><strong><br>Our Tech Stack:</strong></p> <ul> <li>Scala, Apache Spark, Python</li> <li>Airflow</li> <li>ScyllaDB, PostgreSQL, ClickHouse, Hive, Redis, Kafka, Kafka Connect</li> <li>Docker, Jenkins, Terraform, AWS</li> </ul> <p><strong><br>Required Qualifications:</strong></p> <ul> <li>5+ years of experience working in the Data Engineering space, and with building and maintaining big data pipelines&nbsp;</li> <li>3+ years of experience working in agile environments (ideally Scrum), collaborating with cross-functional teams (engineering, design, product).</li> <li>Proficient in Spark</li> <li>Experience with strongly-typed languages (Java / Scala preferred)</li> <li>Experience designing, building, and maintaining RESTful APIs and integrating with external services.</li> <li>Participate in code reviews to ensure best practices, maintainability, and continuous improvement of the codebase.</li> <li>Ability to write and maintain unit and integration tests based on acceptance criteria, ensuring code quality and reliability.</li> <li>Proficiency with version control tools, particularly Git, for collaborative development and code management.</li> </ul> <p><strong><br>Preferred Qualifications:</strong></p> <ul> <li>Strong understanding of building scalable and high-performance back-end systems, optimizing for low-latency and high-throughput.</li> <li>Worked with a variety of data (structured/unstructured), data formats (flat files, XML, JSON, relational, parquet).</li> <li>Experience in functional programming languages (Scala preferred)</li> <li>Experience in Cyber Security</li> </ul> <p><strong><br>Communication and Collaboration:</strong></p> <ul> <li>Strong attention to documentation and maintaining standards across projects.</li> <li>Ability to present and defend technical decisions with confidence.</li> <li>Skilled at estimating project complexity based on early documentation and discussions.</li> <li>Excellent written and verbal communication skills, with the ability to articulate technical challenges and solutions clearly.</li> </ul> <p><span data-she