Topic

#llm-inference

2 articles found

Taming Graphics Cards: A C++ Backend for Efficient GPU Processing

A comprehensive guide to optimizing LLM inference by eliminating padding overhead with hardware-aware sequence packing. The post I Built a C++ Backend So My GPU Would Stop Eating Air appeared first on Towards Data Science.

#ai#gpu-optimization#c-plus-plus#machine-learning#llm-inference

· 3 days ago· Dev.to

AMD GPU outperforms Mac on Llama 8B, falters with Phi-3.

I wrote a post yesterday about why GPUs barely help small text embeddings at batch=1. Different workload, same machines. This time I ran a local LLM inference benchmark across the same three boxes. The result complicated my hardware mental model in a way I think is worth sharing. Three machines. A M

#amd-gpu#llm-inference#cloud-computing#machine-learning