Roles / Collective Communications Software Engineer

Collective Communications Software Engineer

AIS-015v1

AI/ML Silicon Software Stack & Co-DesignSoftwareIC4–IC7Individual contributor

Also known as: Scale-Out Software Engineer; NCCL/RCCL Engineer; Distributed Training Communications Engineer

See this role on the AI accelerator lifecycle or on the SoC lifecycle

Summary

Makes thousands of accelerators behave like one machine by building the collective communication library — all-reduce, all-gather, all-to-all, reduce-scatter — over NVLink, InfiniBand, RoCE and scale-up fabrics.

A day in the life

Work in progressNo day-in-the-life description has been written for this role yet. If you do this work, we would welcome yours. Suggest a source or a correction

Responsibilities

Project

Work done for a product or program.

Implement and tune collective algorithms including ring, tree, double-binary-tree and hierarchical all-reduce across topologies.
Program RDMA verbs, GPUDirect and DMA-BUF paths for zero-copy transfers between accelerator memory and the NIC.
Perform topology discovery and build routing plans that respect rail-optimized InfiniBand and NVLink switch fabrics.
Optimize message chunking, pipelining and protocol selection to hit near line-rate bus bandwidth at every message size.
Implement in-network reduction support such as SHARP offload and evaluate its benefit per workload.
Debug stalls, hangs and silent data corruption in multi-node runs using trace correlation across thousands of ranks.
Integrate the library with PyTorch process groups, DeepSpeed, FSDP, Megatron-LM and JAX collectives.
Collaborate with NIC, switch and SerDes teams on congestion control, adaptive routing and link-error handling.

Function

Work that keeps the discipline running: methods, tools, suppliers and people.

Build fault tolerance and elastic behaviour so a failed rank or link degrades rather than kills a training job.
Benchmark with nccl-tests and OSU micro-benchmarks and correlate against measured large-scale training step time.

Major tasks

Project

Produce and hand over the role's key deliverables
- Collective communications library releases with tuned algorithm selection tables
- Topology discovery and routing plan generator
- Framework process-group integrations for PyTorch and JAX
- Multi-node performance characterization and bus-bandwidth reports
- Fault tolerance and elastic-collective design specification
Develop the softwareGeneral
- Design to the hardware specification
- Write and review the code
- Test on models, emulation or silicon
Release and supportGeneral
- Package and document releases
- Fix customer-reported defects

Function

Maintain the platformGeneral
- Keep build, test and release tools working
- Track toolchain and operating system changes
Raise qualityGeneral
- Run code reviews
- Extend automated tests

Works with

Runtime engineers
framework integration engineers
cluster fleet engineers
NIC and switch silicon teams
hyperscaler platform engineers
HPC customers

Typical backgrounds

MS or PhD in Computer Science or Computer Engineering with HPC or networking focus; 5–15 years in distributed systems or supercomputing software.

Demand signal

0 open listings right now
0 Talent Wafer employer inquiries

Counts live listings tagged to this role, and employer interest in candidates from Talent Wafer runs for this role (all runs to date). Hub activity only, not the wider market.

Comp tier (within semis)

4 / 5

Remote feasibility

Medium

Semis core vs. portable

Semis-primary, transfers

Geographic hotspots

Santa Clara, Seattle, Austin, Yokneam and Tel Aviv, Bangalore, Shanghai, Tokyo, Zurich

Certifications / standards training

—

Transfers to (industries)

Hyperscale cloud networking, supercomputing centres, high-frequency trading networks, storage fabric vendors, telecom core networks

Risk if vacant

Large training clusters scale poorly or hang unpredictably, wasting tens of millions of dollars of accelerator time and making the platform unusable for frontier-scale customers.

Collective Communications Software Engineer

Summary

A day in the life

Responsibilities

Project

Function

Major tasks

Project

Function

Works with

Register or sign in to read on