Topic
6 articles found
I’m trying to think more clearly about latency when using streaming LLM responses, and I’m curious how others here measure it. For normal API calls, latency is fairly straightforward: request starts, response completes, measure total time. With streaming LLM responses, I’m finding that one number is
Significant contributors to this article include Sneha Aradhey, Software Engineer, Google Kubernetes Engine, and Michael MacDonald, Sr Software Engineer, Google Cloud Managed Lustre. Enterprise production environments are shifting to distributed, multi-node architectures to serve long-context window
As LLMs continue to evolve, I’ve found myself changing the way I write prompts. I’m curious have you changed your prompting style over the past year? What techniques or habits have made the biggest difference for you? 1 post - 1 participant Read full topic
Has anyone able to run the code from serving-llms-efficiently-with-vllm-–-part-ii in kaggle or colab. I am unable to complete the steps mentioned in here 1 post - 1 participant Read full topic
Recently, I came across an essay titled "On the Death of Scaling" by Sara Hooker (Co-founder of Adaption Labs). In this essay, Sara explains the shortcomings of the simple path followed by frontier labs to lead the market. She discusses where the notion of "scaling is death" comes from and what to c
The assertion that large language models (LLMs) are "terrible at video games" warrants a nuanced technical examination. While LLMs demonstrate remarkable capabilities in text generation, translation, and code comprehension, their performance in interactive, real-time, and often visually complex envi
