Project Overview:

This project addresses the computational bottlenecks in deep learning inference by developing optimized CUDA kernels for sparse matrix operations. While GPUs excel at dense computations, many modern neural networks benefit from sparsity, making specialized sparse kernels critical for performance.

Key Contributions:

• Developed custom CUDA kernels for sparse matrix-vector multiplication with format-specific optimizations
• Implemented block-sparse operations tailored for transformer attention mechanisms
• Created a pruning-aware convolution operator that dynamically adapts to various sparsity patterns
• Designed memory-efficient sparse matrix storage formats optimized for GPU memory access patterns
• Integrated the optimized kernels with popular deep learning frameworks like PyTorch and TensorFlow

Technical Details:

The implementation leverages advanced CUDA features including shared memory optimization, warp-level primitives, and thread coarsening to maximize throughput. The kernels support various sparsity patterns (structured, unstructured, and block sparsity) with format-specific optimizations.

For transformer models, specialized attention mechanism kernels were developed that exploit the inherent sparsity in attention masks. The convolution operators use pruning awareness to dynamically skip computations for zero weights, significantly improving inference speed.

Performance Results:

• 3.8x speedup for sparse matrix-vector multiplication compared to cuSPARSE
• 2.5x faster inference for transformer models with 80% sparsity
• 65% reduction in memory footprint for large language models
• Linear scaling across multiple GPU configurations

These optimizations enable significant speedups for inference workloads while maintaining model accuracy, making deployment of large-scale models more cost-effective.