Staff Software Engineer, Inference

CoreWeave is The Essential Cloud for AI™. Built for pioneers by pioneers, CoreWeave delivers a platform of technology, tools, and teams that enables innovators to build and scale AI with confidence. Trusted by leading AI labs, startups, and global enterprises, CoreWeave combines superior infrastructure performance with deep technical expertise to accelerate breakthroughs and turn compute into capability. Founded in 2017, CoreWeave became a publicly traded company (Nasdaq: CRWV) in March 2025. Learn more at www.coreweave.com.

About the Role

As a Staff Software Engineer (IC5) on the Inference team, you will operate as a technical leader across multiple teams and services, driving architecture, performance, and reliability for CoreWeave’s Kubernetes-native inference platform.

You will define and lead complex, cross-cutting initiatives that directly impact latency, throughput, cost efficiency, and system reliability at scale. This role requires deep expertise in distributed systems and AI inference, along with the ability to influence engineering direction across the organization.

What You’ll Do

Technical Leadership & Architecture

• Define and drive multi-service, cross-team architecture for CoreWeave’s inference platform.
• Lead complex design initiatives spanning request routing, adaptive scheduling, GPU resource management, and cost-per-token optimization.
• Set technical direction for low-latency, high-throughput inference systems operating under strict P99 SLAs.

Performance & Optimization

• Lead development of advanced inference optimizations, including:
• micro-batching and dynamic scheduling
• speculative decoding
• KV-cache reuse and memory optimization
• Drive measurable improvements in P95/P99 latency, throughput, and GPU utilization across the platform.
• Establish performance benchmarking frameworks and guide teams in data-driven optimization strategies.

Reliability & Systems Excellence

• Own system-wide SLIs/SLOs and ensure reliability improvements across releases.
• Lead efforts in capacity planning, autoscaling strategies, traffic management, and failure mitigation.
• Drive post-incident analysis and ensure systemic improvements across teams.

Cross-Team Influence

• Partner with Product, Orchestration, Networking, Storage, and Hardware teams to deliver a cohesive inference platform.
• Influence technical decisions across org boundaries and ensure alignment on architecture, performance, and scalability goals.
• Serve as a go-to technical expert for inference systems and distributed AI workloads.

Mentorship & Engineering Excellence

• Mentor senior and mid-level engineers; elevate design, coding, and operational standards across teams.
• Lead cross-team design reviews and ensure high-quality system thinking at scale.
• Raise the bar for engineering rigor, testing, and observability practices.

What We’re Looking For

• ~8–12+ years of experience building large-scale distributed systems or cloud platforms
• Proven track record of leading cross-team technical initiatives at scale
• Strong coding skills in Go, Python, or C++
• Deep expertise in Kubernetes at production scale, including orchestration, scheduling, and service design
• Strong understanding of networked systems, performance optimization, and system design
• Bachelor’s degree in Computer Science, Engineering, or a related technical field 

Inference & Performance Expertise

• Hands-on experience with inference systems, including:
• batching and micro-batching strategies
• caching and memory optimization
• mixed precision (BF16/FP8)
• streaming token delivery
• Demonstrated ability to improve tail latency (P95/P99) and system reliability through metrics-driven engineering

Nice to Have

• Contributions to inference frameworks such as vLLM, Triton, TensorRT-LLM, Ray Serve, TorchServe
• Experience with GPU systems and performance optimization (CUDA, NCCL, RDMA, NUMA, GPU interconnects)
• Experience leading multi-team or org-level initiatives
• Exposure to large-scale AI/ML infrastructure or hyperscale cloud environments

The base salary range for this role is 116,000 GBP to 155,000 GBP. The starting salary will be determined based on job-related knowledge, skills, experience, and market location. We strive for both market alignment and internal equity when determining compensation. In addition to base salary, our total rewards package includes a discretionary bonus, equity awards, and a comprehensive benefits program (all based on eligibility). 

What We Offer

The range we’ve posted represents the typical compensation range for this role. To determine actual compensation, we review the market rate for each candidate which can include a variety of factors. These include qualifications, experience, interview performance, and location.

In addition to a competitive salary, we offer a variety of benefits to support your needs. The benefits below reflect our US-based offerings for full-time employees; for roles in other locations, benefits vary and are shared during the hiring process. These include:

• Medical, dental, and vision insurance - 100% paid for by CoreWeave
• Company-paid Life Insurance 
• Voluntary supplemental life insurance 
• Short and long-term disability insurance 
• Flexible Spending Account
• Health Savings Account
• Tuition Reimbursement 
• Ability to Participate in Employee Stock Purchase Program (ESPP)
• Mental Wellness Benefits through Spring Health 
• Family-Forming support provided by Carrot
• Paid Parental Leave 
• Flexible, full-service childcare support with Kinside
• 401(k) with a generous employer match
• Flexible PTO
• Catered lunch each day in our office and data center locations
• A casual work environment
• A work culture focused on innovative disruption

California Applicants

California Consumer Privacy Act 

Equal Opportunity & Accommodations

CoreWeave is an equal opportunity employer, committed to fostering an inclusive and supportive workplace. All qualified applicants and candidates will receive consideration for employment without regard to race, color, religion, sex, disability, age, sexual orientation, gender identity, national origin, veteran status, or genetic information.

As part of this commitment and consistent with the Americans with Disabilities Act (ADA), CoreWeave will ensure that qualified applicants and candidates with disabilities are provided reasonable accommodations for the hiring process, unless such accommodation would cause an undue hardship. If reasonable accommodation is needed, please contact: careers@coreweave.com.

Export Control Compliance

This position requires access to export controlled information.  To conform to U.S. Government export regulations applicable to that information, applicant must either be (A) a U.S. person, defined as a (i) U.S. citizen or national, (ii) U.S. lawful permanent resident (green card holder), (iii) refugee under 8 U.S.C. § 1157, or (iv) asylee under 8 U.S.C. § 1158, (B) eligible to access the export controlled information without a required export authorization, or (C) eligible and reasonably likely to obtain the required export authorization from the applicable U.S. government agency.  CoreWeave may, for legitimate business reasons, decline to pursue any export licensing process.