Micro language models for distributed agents: efficiency, privacy, and sustainability

In the era of autonomous AI agents, relying solely on large language models (LLMs) can become inefficient. LLMs require costly infrastructure, consume significant amounts of energy, and are not well suited for edge devices. In addition, sending sensitive data to the cloud introduces latency, bandwidth costs, and privacy risks. This talk introduces small language models (SLMs): models with millions—rather than billions—of parameters, designed for specialized tasks. These models consume less energy, provide faster inference, and can run directly on edge or embedded devices. Thanks to techniques such as quantization, pruning, and efficient fine-tuning, SLMs make it possible to move intelligence from the cloud to the edge, reducing latency while protecting local data. I will present use cases where SLMs are integrated into distributed architectures, from collaborative robots to IoT devices and federated systems designed for privacy preservation. I will also compare costs and performance with LLM-based approaches, illustrate optimization techniques, and introduce a method for migrating agents from general-purpose LLMs to specialized SLMs, following insights from recent NVIDIA studies. The goal of this talk is to demonstrate that micro language models are not only powerful enough for most agentic AI functions, but also represent a more sustainable and cost-effective alternative to general-purpose models, enabling a more democratic and widely accessible AI ecosystem.