Getting Started with Local AI: Squeezing a 27B Model Down to 13.5GB—Do the Memory Math First

Niels Rogge shared his Getting started with Local AI slides at a hackathon in Qatar and posted the Google Slides link. The material covers what local AI is, an overview of models, deployment details, and local agents. The images and slides are below.

The slides start by laying out the real costs of local AI: data is not sent to third parties, it can run offline, you are not locked into a single provider, you do not have to worry about downtime, and there are no quotas or usage limits. Niels Rogge also offers a judgment: most tasks do not require frontier intelligence. That point is crucial for local deployment, because many scenarios need stability, controllability, and low cost—not the top spot on a leaderboard.

For real-world deployment, start with quantization. The slides give a straightforward definition: quantization shrinks a model to a smaller size while bringing performance close to the original model. Precision is measured in bits per parameter: float16 is 16 bits, int8 is 8 bits, and int4 is 4 bits. Take a 27B model as an example: float16 requires 27B × 2 bytes = 54GB, int8 requires 27GB, and int4 requires about 13.5GB. Once the numbers are on the table, the barrier to local AI shifts from whether you have a GPU to whether you have enough RAM and VRAM. But fitting in int4 does not mean the experience is the same: quantization causes precision loss, and engineering still has to handle details such as GGUF, prefill, and decode. The slides list these separately, showing that getting started does not end with downloading a model.

In the model architecture section, the slides discuss dense and MoE. In the replies, someone asked whether dense models are dinosaurs now. That question is worth keeping in mind: MoE is attractive on inference cost, while dense models still have a place in simple deployments and small-scale scenarios. The material does not offer a conclusion, so do not rush to pick a side.

On the tooling entry point, the slides mention using @Llama_app. The Google Slides link is here: https://docs.google.com/presentation/d/1zr10eYFVxJhN8RBAHh_eeHVnZSXz9UvwT3No1N4x4ug/edit?usp=sharing

If you only want to remember one sentence: the selling point of local AI is often described as being free, but what is actually more valuable is control. Control has a price, and that price is usually written into the memory bill and quantization loss.