research
Edge LLMs and diffusion
Running llama.cpp language models and diffusion models on Raspberry Pi and NVIDIA Jetson.
Idea
How far down can modern generative models go? These experiments run quantized LLMs with llama.cpp and diffusion models directly on small devices, with no cloud.
Approach
- Quantized GGUF models with llama.cpp on Raspberry Pi and Jetson.
- Diffusion models on Jetson with GPU acceleration.
- Measure tokens per second, memory, and power to find the practical limits of each device.
Stack
llama.cpp, NVIDIA Jetson, Raspberry Pi, TensorRT.