Projects

Sparse AllReduce

HPC
Parallel Processing
Distributed Machine Learning
MPI
NCCL
PyTorch

This project benchmarked multiple sparse allreduce algorithms using PyTorch, MPI, and NCCL.

Overview

This project was done for a course in parallel processing. I implemented three sparse allreduce algorithms, and tested their performance on the Easley cluster at CARC.

What I built

  • Implemented three sparse allreduce algorithm capable of running on CPUs with MPI and GPUs using NCCL.
  • Created python scripts and shell scripts to run a scaling study on the algorithms.

Technologies

Python, PyTorch, Distributed Computing, NCCL, MPI