Fast groq-hosted LLMs vs browser jank

Fast groq-hosted LLMs vs browser jank

Groq is now serving LLMs such as Llama 3 so quickly that JavaScript which attempts to render Markdown strings on every new token can cause performance issues in browsers.

Taras Glek's solution was to move the rendering to a requestAnimationFrame() callback, effectively buffering the rendering to the fastest rate the browser can support.

Via lobste.rs

browser browsers callback every groq javascript llama llama 3 llms markdown performance render rendering solution strings token