You can swap experts in and out of VRAM, it just increases inference time substa... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		popinman322 8 months ago \| parent \| context \| favorite \| on: The Llama 4 herd You can swap experts in and out of VRAM, it just increases inference time substantially. Depending on the routing function you can figure out all the active experts ahead of the forward pass for a single token and pipeline the expert loading.

boroboro4 8 months ago [–]

Chosen expert (on each layer) depends on the input of previous layer. Not sure how you can preload the experts before forward pass.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact