Job Description

Replit is the agentic software creation platform that enables anyone to build applications using natural language. With millions of users worldwide, Replit is democratizing software development by removing traditional barriers to application creation.About the roleThe Trust & Safety team is the front line defending Replit's platform from exploitation. We detect and shut down phishing deployments, prevent cryptomining on free-tier infrastructure, stop LLM token farming, and keep bad actors from weaponizing the platform against our users. This is adversarial work: attackers adapt constantly, and we build the detection systems, heuristics, and automated responses that stay ahead of them.What makes this role unique is the AI-native nature of Replit's platform. You'll work on problems that barely exist elsewhere: building guardrails for AI-generated code, detecting prompt injection attacks at scale, and using LLMs as a defensive tool against abuse. If you want hands-on experience applying AI to security problems, this is one of the few places you can do it in production with real attackers. You'll own problems end-to-end, from identifying emerging abuse patterns to shipping the systems that stop them at scale.In this role you will…Design and implement LLM guardrails that detect abuse scenarios in AI-generated code and agent interactionsBuild AI-powered detection systems that use LLMs to identify malicious patterns, classify threats, and automate response decisionsBuild and operate abuse detection systems that identify phishing, cryptomining, account takeover, and financial fraud across millions of daily user actionsDesign automated response mechanisms that enforce platform policies without manual interventionOwn the full abuse response lifecycle: detection, investigation, enforcement, and handling appeals alongside Support and LegalAnalyze attack patterns using BigQuery and Hex, turning investigation findings into new detection rulesMaintain and extend internal detection tools (Slurper, Netwatch) that continuously monitor user activityIntegrate and tune security scanners (SAST, SCA) in CI pipelines with tight performance SLAsTrack abuse trends, measure detection effectiveness, and adapt defenses as attack patterns evolveRequired skills and experience:4+ years of experience in security engineering, anti-abuse, trust & safety, or fraud detectionStrong programming skills in Python and/or TypeScript for building detection systems and automationExperience with SQL and data analysis at scale (BigQuery, Snowflake, or similar)Experience building or fine-tuning ML/LLM-based classifiers for security or abuse detectionFamiliarity with prompt injection, jailbreaking, and other LLM-specific attack vectorsAbility to investigate complex abuse patterns and translate findings into automated defensesFamiliarity with common attack patterns: phishing infrastructure, account takeover, credential stuffing, resource abuseClear communication skills for working across Security, Support, Legal, and Engineering teams.Nice to have:Experience at a platform company dealing with user-generated content or compute abuse (hosting providers, cloud platforms, developer tools)Background in fraud detection, payment abuse, or financial crimeFamiliarity with device fingerprinting, IP reputation, and email validation servicesExperience with CI/CD security tooling (SAST, SCA, Dependabot, Snyk)Knowledge of container security, Linux internals, or cloud infrastructure (GCP preferred)Prior work with abuse reporting pipelines, trust & safety tooling, or content moderation systemsTools + Tech Stack for this roleLanguages: Python, TypeScript, Go, SQLData: BigQuery, HexDetection tools: Slurper, Netwatch, Stytch (device fingerprint); ClearOut (email reputation)CI/CD Security: Dependabot, Snyk, SAST/SCA scannersInfrastructure: GCP, KubernetesCollaboration: Linear, Slack, Zendesk (for abuse reports)This role may not be a fit ifYou prefer deep security research over building operational detection systemsYou want to focus on vulnerability management, pentesting, or bug bounty triage (that's our Security team)You're looking for a role with predictable, well-defined problems rather than constantly adapting to adversarial behaviorYou prefer working in isolation rather than partnering closely with Support, Legal, and cross-functional teamsYou're uncomfortable making enforcement decisions that affect real usersThis is a full-time role that can be held from our Foster City, CA office. The role has an in-office requirement of Monday, Wednesday, and Friday. Full-Time Employee Benefits Include:💰 Competitive Salary & Equity💹 401(k) Program with a 4% match (US Only)⚕️ Health, Dental, Vision and Life Insurance🩼 Short Term and Long Term Disability🚼 Paid Parental, Medical, Caregiver Leave🏝 Flexible Time Off (FTO) + Holidays🚗 Commuter Benefits (In-Office Only)📱 Monthly Wellness Stipend🧑‍💻 Autonomous Work Environment🖥 In Office Set-Up Reimbursement (In-Office Only)🚀 Quarterly Team Gatherings☕ In Office Amenities (In-Office Only)Want to learn more about what we are up to?Meet the Replit AgentReplit: Make an app for thatReplit BlogAmjad TED TalkInterviewing + Culture at ReplitOperating PrinciplesReasons not to work at ReplitTo achieve our mission of making programming more accessible around the world, we need our team to be representative of the world. We welcome your unique perspective and experiences in shaping this product. We encourage people from all kinds of backgrounds to apply, including and especially candidates from underrepresented and non-traditional backgrounds.

Job Tags

Similar Jobs