Web Scraping Engineer - Global Team (India)

About this role

<p><img style="max-width: 100%;" src="https://images.squarespace-cdn.com/content/v1/61a5868fbaa6ba287e749ca1/2c2722b4-1d79-4c06-a4de-896076e98fcc/SHOT_06_INVESTOR_DATA_MM_220322_122.jpg" alt="" width="800"></p> <p><strong>About YipitData:</strong></p> <p>YipitData is the leading market research and analytics firm for the disruptive economy and recently raised up to $475M from The Carlyle Group at a valuation over $1B.</p> <p>We analyze billions of alternative data points every day to provide accurate, detailed insights on ridesharing, e-commerce marketplaces, payments, and more. Our on-demand insights team uses proprietary technology to identify, license, clean, and analyze the data many of the world’s largest investment funds and corporations depend on.</p> <p>For three years and counting, we have been recognized as one of<strong><em> </em></strong><a href="https://www.inc.com/profile/yipitdata"><strong><em>Inc’s Best Workplaces</em></strong></a>. We are a fast-growing technology company backed by The Carlyle Group and Norwest Venture Partners. Our offices are located in NYC, Austin, Miami, Denver, Mountain View, Seattle<strong>, </strong>Hong Kong, Shanghai, Beijing, Guangzhou, and Singapore. We cultivate a people-centric culture focused on mastery, ownership, and transparency.</p> <p><strong>Why You Should Apply NOW:</strong></p> <ul> <li>High Impact: Your work will directly influence key reports and strategic decisions across multiple business units.</li> <li>Exciting Challenges: Tackle the design of resilient web scrapers, navigate dynamic website structures, and optimize large-scale data extraction.</li> <li>Growth Opportunities: As an early member of our expanding Web Scraping Engineering team, you will have significant input on our strategies, processes, and team culture.</li> </ul> <p><strong>About The Role:</strong></p> <p>We are seeking a <strong>Web Scraping Engineer </strong>to join our growing engineering team. In this hands-on role, you’ll take ownership of designing, building, and maintaining robust web scrapers that power critical reports and customer experiences across our organization. You will work on complex, high-impact scraping challenges and collaborate closely with cross-functional teams to ensure our data ingestion processes are resilient, efficient, and scalable, while delivering high-quality data to our products and stakeholders.</p> <p><strong>As Our Web Scraping Engineer You Will:</strong></p> <p>Refactor and Maintain Web Scrapers</p> <ul> <li>Overhaul existing scraping scripts to improve reliability, maintainability, and efficiency.</li> <li>Implement best coding practices (clean code, modular architecture, code reviews, etc.) to ensure quality and sustainability.</li> </ul> <p>Implement Advanced Scraping Techniques</p> <ul> <li>Utilize sophisticated fingerprinting methods (cookies, headers, user-agent rotation, proxies) to avoid detection and blocking.</li> <li>Handle dynamic content, navigate complex DOM structures, and manage session/cookie lifecycles effectively.</li> </ul> <p>Collaborate with Cross-Functional Teams</p> <ul> <li>Work closely with analysts and other stakeholders to gather requirements, align on targets, and ensure data quality.</li> <li>Provide support, documentation, and best practices to internal stakeholders to ensure effective use of our web scraped data in critical reporting workflows.</li> </ul> <p>Monitor and Troubleshoot</p> <ul> <li>Develop robust monitoring solutions, alerting frameworks&nbsp; to quickly identify and address failures.</li> <li>Continuously evaluate scraper performance, proactively diagnosing bottlenecks and scaling issues.</li> </ul> <p>Drive Continuous Improvement</p> <ul> <li>Propose new tooling, methodologies, and technologies to enhance our scraping capabilities and processes.</li> <li>Stay up to date with industry trends, evolving bot-detection tactics, and novel approaches to web data extraction.</li> </ul> <p><strong>This is a fully-remote opportunity based in India. Standard work hours are from 11am to 8pm IST, but there is flexibility here.</strong></p> <p><strong>You Are Likely To Succeed If:</strong></p> <ul> <li>Effective communication in English with both technical and non-technical stakeholders.</li> <li>3+ years of experience with web scraping frameworks (e.g., Selenium, <strong>Playwright, or Puppeteer</strong>).</li> <li>Strong understanding of HTTP, RESTful APIs, HTML parsing, browser rendering, and TLS/SSL mechanics.</li> <li>Expertise in advanced <strong>fingerprinting </strong>and evasion strategies (e.g., browser fingerprint spoofing, request signature manipulation).</li> <li>Deep experience managing <strong>cookies, headers, session states, and proxy rotations</strong>, including the deployment of both residential and data center proxies.</li> <li>Experience with logging, metrics, and alerting to ensure high availability.</li> <li>Troubleshooting skills to optimize scraper performance for efficiency, reliability, and scalability.</li> </ul> <p><strong>What We Offer:</strong></p> <p><strong>Our compensation package includes comprehensive benefits, perks, and a competitive salary:&nbsp;</strong></p> <ul> <li>We care about your personal life and we mean it. We offer vacation time, parental leave, team events, learning reimbursement, and more!</li> <li>Your growth at YipitData is determined by the impact that you are making, not by tenure, unnecessary facetime, or office politics. Everyone at YipitData is empowered to learn, self-improve, and master their skills in an environment focused on ownership, respect, and trust.</li> </ul> <p><em>We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, marital status, disability, gender, gender identity or expression, or veteran status. We are proud to be an equal-opportunity employer.</em></p> <p><a href="https://www.yipitdata.com/careers/job-applicant-privacy-noti