What local model runtimes does this skill support?

This skill supports Foundry Local, LM Studio, Ollama, and ONNX Runtime (including Phi-3 vision for multimodal inference). It provides sample code for each of these local serving options.

When should I use these local model samples instead of cloud APIs?

Use these samples when you need offline use, have data privacy requirements, want to control costs, or need edge deployment. They let Semantic Kernel talk to models running locally rather than calling hosted cloud APIs.

What types of outputs can these samples generate?

The samples can generate chat completions, text completions, text embeddings, and multimodal completions that incorporate image input (via the Phi-3 vision sample) — all processed locally.

Do these samples require a local model to already be running?

Yes, each sample assumes the corresponding local runtime or model is already installed and running before you use it.

Skill

Integrate Local LLMs with Semantic Kernel

Name: Semantic Kernel - Local Models
Availability: OnlineOnly
Author: Semantic Kernel

7 Semantic Kernel samples for running local LLMs via Foundry Local, LM Studio, Ollama, and ONNX (including Phi-3 vision).

Get skill

Works with ollamaonnx

Semantic Kernel

Own this? Claim it

Spark score

out of 100

Updated today

Version dotnet-1.78.0

Add to Favorites

Why it matters

Leverage the power of local Large Language Models (LLMs) like Ollama and ONNX directly within your applications using the Semantic Kernel SDK. This asset enables you to build sophisticated AI features without relying on cloud-based APIs.

Outcomes

What it gets done

Connect to and utilize local LLM inference engines (Ollama, ONNX).

Implement chat completion and text embedding functionalities locally.

Develop AI-powered applications with on-device LLM capabilities.

Integrate vision models for local image understanding.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/sk-concept-localmodels | bash

Overview

Semantic Kernel - Local Models

Seven Semantic Kernel Python samples for local LLM inference via Foundry Local, LM Studio, Ollama, and ONNX Runtime, including Phi-3 vision. Use for offline, privacy-sensitive, or cost-controlled deployments where a locally hosted model replaces a cloud API.

What it does

This is a set of 7 Semantic Kernel Python concept samples for running local LLM inference instead of calling a cloud API: a Foundry Local chatbot (foundry_local_chatbot.py), LM Studio chat completion and text embedding (lm_studio_chat_completion.py, lm_studio_text_embedding.py), Ollama chat completion (ollama_chat_completion.py), and three ONNX runtime samples covering chat completion, Phi-3 vision multimodal completion, and text completion (onnx_chat_completion.py, onnx_phi3_vision_completion.py, onnx_text_completion.py).

When to use - and when NOT to

Use these when you need Semantic Kernel to talk to a model running locally rather than a hosted API - for offline use, data privacy requirements, cost control, or edge deployment. Pick based on your local runtime: LM Studio or Ollama for their respective serving layers, ONNX for direct runtime inference (including the Phi-3 vision sample if you need local multimodal/image understanding), or Foundry Local for that specific local serving option.

Not the right samples if you're calling a cloud-hosted model API - see Semantic Kernel's cloud-provider chat completion samples for that instead. Each sample assumes the corresponding local runtime/model is already installed and running.

Inputs and outputs

Inputs: chat messages or text/image prompts, sent to a locally running model server or ONNX runtime session instead of a remote API.

Outputs: chat completions, text completions, text embeddings, or (for the Phi-3 vision sample) multimodal completions incorporating image input - all generated locally.

Integrations

Microsoft Semantic Kernel (Python), Foundry Local, LM Studio, Ollama, ONNX Runtime (including Phi-3 vision).

Who it's for

Developers who need Semantic Kernel agents to run against locally hosted models for privacy, cost, or offline requirements, and need a sample matching their specific local serving stack.

Source README

Running local LLMs with Semantic Kernel: Ollama, ONNX, and other local inference

Examples (7 files):

foundry_local_chatbot.py
lm_studio_chat_completion.py
lm_studio_text_embedding.py
ollama_chat_completion.py
onnx_chat_completion.py
onnx_phi3_vision_completion.py
onnx_text_completion.py

FAQ

Common questions

Discussion