Documentation – Multimodal

Net: Vision presets

Thu, 23 Apr 2026 00:00:00 +0000

The SDK ships four built-in vision presets. Each preset configures both the base language model and its multimodal projector (mmproj) — the two files load together on first Create.

Available presets

Preset	Model	Base context	Quantization	mmproj file
`Qwen25VL3BPreset`	Qwen 2.5 VL 3B Instruct	128 000	UD-IQ2_XXS	`mmproj-F16.gguf`
`Qwen3VL2BPreset`	Qwen 3 VL 2B Instruct	262 144	Q4_K_M	`mmproj-Qwen3VL-2B-Instruct-Q8_0.gguf`
`Gemma3VisionPreset`	Gemma 3 Vision (Latex fine-tune)	8 096	Q4_K_M	`Gemma-3-Vision-Latex.mmproj-f16.gguf`
`Ministral3VisionPreset`	Ministral 3 8B Instruct (Mistral AI, 2512 release)	262 144	Q4_K_M	`Ministral-3-8B-Instruct-2512-BF16-mmproj.gguf`

See Supported presets for the Hugging Face source repositories.

Picker

Need	Try
Smallest footprint, long context	`Qwen3VL2BPreset` (2B parameters, 262K context)
General-purpose vision Q&A	`Qwen25VL3BPreset` (3B, 128K)
Text-heavy images (documents, LaTeX)	`Gemma3VisionPreset`
Strongest reasoning on complex images	`Ministral3VisionPreset` (8B, 262K)

All four produce reasonable image descriptions and simple spatial reasoning. For OCR-style tasks on dense text, lean toward Gemma 3 Vision or Ministral 3 — the larger projectors handle small text better.

Memory

Vision presets load two files: the base model and the projector. Add the projector memory footprint on top of the base model — typically 200 MB to 2 GB depending on precision.

Preset	Base (VRAM/RAM)	Projector	Total
`Qwen25VL3BPreset` (UD-IQ2_XXS)	~2 GB	~0.8 GB (F16)	~3 GB
`Qwen3VL2BPreset`	~2 GB	~0.5 GB (Q8_0)	~2.5 GB
`Gemma3VisionPreset`	~3 GB	~1 GB (F16)	~4 GB
`Ministral3VisionPreset`	~6 GB	~2 GB (BF16)	~8 GB

Add KV cache on top (scales with ContextParameters.ContextSize). For long contexts, reduce TypeV to Q8_0 to claw back memory.

Using a vision preset

Same pattern as any other preset. Pass images via the media parameter of SendMessageAsync or SendMessageToSessionAsync:

using Aspose.LLM;
using Aspose.LLM.Abstractions.Parameters.Presets;

var preset = new Qwen3VL2BPreset();
using var api = AsposeLLMApi.Create(preset);

byte[] imageBytes = File.ReadAllBytes("document.png");

string reply = await api.SendMessageAsync(
    "Transcribe the text in this image verbatim.",
    media: new[] { imageBytes });

Console.WriteLine(reply);

See Attaching images for format and size rules.

Customizing a vision preset

The same override patterns as text presets apply. See Presets for the three approaches:

Override before Create — tweak fields on the preset instance.
Subclass — inherit from a built-in vision preset and set defaults in the constructor.
From scratch — extend PresetCoreBase and populate both BaseModelSourceParameters and MmprojSourceParameters.

Additional vision-only knobs live on MtmdContextParameters — control projector GPU offload, threading, and verbosity.

What’s next

Attaching images — how to pass image bytes.
Chat templates — how the SDK selects the right template per model.
Multimodal context parameters — vision-side tuning knobs.
Model source parameters — configure the mmproj download source.

Net: Attaching images

Thu, 23 Apr 2026 00:00:00 +0000

The media parameter of SendMessageAsync and SendMessageToSessionAsync accepts image byte arrays. The engine detects each image’s format from its magic bytes, validates size, and passes it through the vision projector alongside the text prompt.

Quick example

byte[] imageBytes = File.ReadAllBytes("cat.jpg");

string reply = await api.SendMessageAsync(
    "Describe the animal in this image.",
    media: new[] { imageBytes });

The parameter is IEnumerable<byte[]>?. Pass null or omit it entirely for a text-only message.

Supported formats

Format	Magic bytes (hex)
JPEG	`FF D8`
PNG	`89 50 4E 47`
BMP	`42 4D`
GIF	`47 49 46`
WebP	`52 49 46 46 .. .. .. .. 57 45 42 50`

Formats outside this list are rejected. If your source is TIFF, HEIC, SVG, or another format, convert to one of the supported formats before passing to the API.

50 MB per-attachment limit

Each image must be 50 MB or smaller. Larger images throw:

System.InvalidOperationException: Image size exceeds maximum allowed (50MB).

In practice, downsize images to the projector’s native resolution (typically 336 or 448 pixels on the short side) before attaching — large images are resized internally and eat wall time.

Multiple images per message

Pass several byte arrays in one call:

byte[] img1 = File.ReadAllBytes("before.png");
byte[] img2 = File.ReadAllBytes("after.png");

string reply = await api.SendMessageAsync(
    "Compare these two screenshots. What changed?",
    media: new[] { img1, img2 });

The engine processes images in the order they appear in the array. The chat template places them at marker positions in the prompt.

Not every vision model handles arbitrary numbers of images equally well — most are tuned for one or two. Refer to the model’s own documentation for recommended limits.

`MediaAttachment` class

When SendMessageAsync receives byte[], it wraps each array in a MediaAttachment internally. The class is also public if you need to handle attachments explicitly — for example, when adding them to ChatMessage instances manually.

namespace Aspose.LLM.Abstractions.Models;

public class MediaAttachment
{
    public byte[] Data { get; set; }
    public MediaFormat Format { get; set; }
    public string MimeType { get; set; }
    public int Width { get; set; }
    public int Height { get; set; }
    public string? Description { get; set; }
    public string? FileName { get; set; }

    public MediaAttachment();
    public MediaAttachment(byte[] data, MediaFormat format);

    public static MediaAttachment FromBytes(
        byte[] data,
        string? fileName = null,
        string? description = null);

    public void Validate();
}

`FromBytes` — auto-detecting format

The idiomatic constructor:

using Aspose.LLM.Abstractions.Models;

var attachment = MediaAttachment.FromBytes(
    File.ReadAllBytes("scan.png"),
    fileName: "scan.png",
    description: "Invoice scanned at 300 DPI");

The factory:

Reads the first 12 bytes.
Matches against the magic-byte table above.
Sets Format, MimeType, FileName, and Description accordingly.
Returns the attachment.

If the magic bytes do not match, Format is MediaFormat.Unknown and Validate throws when called.

`Validate`

Enforces three checks:

Check	Failure
Data is non-null and non-empty	`InvalidOperationException: Image data cannot be null or empty.`
Data size ≤ 50 MB	`InvalidOperationException: Image size exceeds maximum allowed (50MB).`
Format != Unknown	`InvalidOperationException: Unknown or unsupported image format.`

The SDK calls Validate automatically when you send a message. You can call it earlier to fail-fast in application code.

`MediaFormat` enum

public enum MediaFormat
{
    Unknown = 0,
    JPEG,
    PNG,
    BMP,
    GIF,
    WebP,
}

`ChatMessage` with media

For scenarios that construct the chat history manually (for example, seeding ChatParameters.History with a multimodal turn), build ChatMessage objects directly:

using Aspose.LLM.Abstractions.Models;

var attachment = MediaAttachment.FromBytes(File.ReadAllBytes("label.png"));
var msg = ChatMessage.CreateUserMessage("Read this label.", attachment);

// Attach to preset history:
preset.ChatParameters.History = new List<ChatMessage> { msg };

ChatMessage.HasMedia and ChatMessage.TotalMediaSize are convenience properties for inspection.

Common errors

Symptom	Cause	Fix
`Unknown or unsupported image format`	File is TIFF, HEIC, SVG, or corrupt.	Convert to JPEG/PNG/BMP/GIF/WebP.
`Image size exceeds maximum allowed (50MB)`	Very high-resolution image.	Downscale or recompress before attaching.
Garbled reply, no image detail	Chat template mismatch or wrong preset.	Verify you are using a vision preset; see Chat templates.
Reply ignores the image	Prompt is too generic.	Reference “the image”, “this diagram”, etc. explicitly.

What’s next

Vision presets — pick the right preset for your images.
Chat templates — how the engine inserts image markers.
Debugging vision — diagnose misalignments and garbled output.

Net: Chat templates

Thu, 23 Apr 2026 00:00:00 +0000

Every vision model family has its own prompt format — a specific way to mark where images are inserted and how text turns are wrapped. Aspose.LLM for .NET ships eight templates and selects the right one automatically from the model’s GGUF metadata at load time.

You do not configure templates yourself in the current release. The selection is automatic and internal. This page exists so you can identify which templates are supported and recognize the model families behind them.

Supported templates

Template	Model family	Typical models
LLaVA	LLaVA-style vision fine-tunes	LLaVA-1.5, LLaVA-1.6, various community LLaVAs
Qwen2VL	Qwen 2 / 2.5 VL	`Qwen2-VL-`, `Qwen2.5-VL-`
Qwen3VL	Qwen 3 VL	`Qwen3-VL-*`
Pixtral	Mistral Pixtral	`Pixtral-12B` and derivatives
InternVL	InternVL series	`InternVL2-`, `InternVL3-`
Gemma3	Gemma 3 Vision	`gemma-3-*-vision` variants
Llama4	Llama 4 Vision	`Llama-4-*-Vision`
MiniCPMV	MiniCPM-V	`MiniCPM-V-*`

How selection works

When the engine loads a vision model, it:

Reads the model’s metadata (architecture name, base model name, and vision-specific keys) from the GGUF file.
Matches those values against the template dispatch table in Aspose.LLM.Interop.Multimodal.VisualModelChatTemplates.
Picks the template whose marker tokens and turn format match the detected model family.

If no template matches, the engine falls back to the default text template. Vision turns then emit a generic marker that the model may or may not recognize — output quality degrades.

Built-in preset → template mapping

Each built-in vision preset is already tested against its matching template:

Preset	Template auto-selected
`Qwen25VL3BPreset`	Qwen2VL
`Qwen3VL2BPreset`	Qwen3VL
`Gemma3VisionPreset`	Gemma3
`Ministral3VisionPreset`	Pixtral (Mistral vision models share the Pixtral template)

If you extend one of these presets or use a custom GGUF from the same family, template detection still works.

Custom vision models

If you build a custom preset pointing at a non-Aspose GGUF from one of the eight supported families, template auto-selection should work out of the box — the engine relies on metadata keys that most upstream GGUF conversions preserve.

If selection fails — the response is garbled or includes literal marker tokens like <image> in the output — the model’s metadata is missing the expected keys. Options:

Pick a different GGUF export of the same model with richer metadata.
File a support request via the Aspose Support Forum with the model’s Hugging Face URL so the team can add detection for that specific export.

The current SDK version does not expose a public override for forcing a specific template. A future release may add that surface.

Checking which template was picked

Enable debug logging and look for the [MM] tags:

preset.EngineParameters.EnableDebugLogging = true;

Lines like [MM] selected template: Qwen3VL appear shortly after model load. See Debugging vision for the full log taxonomy.

What’s next

Vision presets — built-in presets with their matching templates.
Debugging vision — inspect the selected template in logs.
Attaching images — the sending side.

Net: Debugging vision

Thu, 23 Apr 2026 00:00:00 +0000

Vision flows involve several extra stages compared to plain text: image preprocessing, projector evaluation, marker tokenization, KV alignment, and generation. When something goes wrong, the failure is rarely a clean exception — instead you see garbled, repetitive, or off-topic output. This page shows how to diagnose those failures.

Turn on debug logging

using Microsoft.Extensions.Logging;

using var loggerFactory = LoggerFactory.Create(builder =>
    builder.AddConsole().SetMinimumLevel(LogLevel.Debug));
var logger = loggerFactory.CreateLogger<Program>();

var preset = new Qwen3VL2BPreset();
preset.EngineParameters.EnableDebugLogging = true;
preset.MtmdContextParameters.Verbosity = 3; // mtmd debug level
preset.MtmdContextParameters.PrintTimings = true;

using var api = AsposeLLMApi.Create(preset, logger);

With these settings, the native layers emit tagged lines during every chat operation.

Log tag taxonomy

Tags appear at the start of each log line. Grep by tag to isolate one concern at a time.

Tag	Origin	What it shows
`[MM]`	Multimodal layer	Projector load, template selection, chunk counts, alignment.
`[CTX]`	Context manager	Context state, batch dispatch, KV reservations.
`[KV]`	KV cache	Evictions, defragmentation, cleanup strategy application.
`DBG tok:`	Tokenizer	Raw tokens emitted for each prompt and generation step.
`DBG mtmd-tokenize`	`mtmd_tokenize`	Per-chunk tokenization details when images are inserted.

Typical line shapes:

[MM] selected template: Qwen3VL
[MM] projector loaded: mmproj-Qwen3VL-2B-Instruct-Q8_0.gguf (524 MB)
[MM] tokenize: text chunks = 3, image chunks = 1
[CTX] batch dispatched: tokens=1536, seq_id=0
[KV] cleanup: strategy=RemoveOldestMessages, freed=4096 tokens

The `parse_mm_logs.zsh` helper

The Aspose.LLM SDK repository includes a helper at parse_mm_logs.zsh that filters a raw log file into digestible sections. It groups output by concern:

Pairing — base model + projector match.
Projector load — file path and size.
Template choice — which of the eight templates was selected.
Marker tokenization — how the image marker tokens were emitted in the prompt.
Chunks — text/image chunk counts and their positions.
Alignment — whether the image chunks line up with the prompt markers.
Eval — projector evaluation timings.
Context adoption — which KV positions hold image embeddings.
Conditioning — tokens produced in the first decode step.
KV state — reservation and eviction.
Token stream — every generated token.
Final answer — the trimmed response text.

Usage:

./parse_mm_logs.zsh < raw-run.log > sectioned-run.txt

The sectioned output makes misalignments obvious — the “Alignment” section is usually where the problem sits when output is garbled.

Common vision failures

Garbled output or literal marker tokens in reply

Symptom: the reply contains <image>, <img>, or other bracketed tokens verbatim. Or the model describes something unrelated to the image.

Root cause: chat template mismatch. The engine picked the wrong template (or the fallback) because the GGUF’s metadata does not expose the model family expected by the dispatch table.

Fix:

Verify you are using a supported preset family (Qwen VL, Gemma 3 Vision, Ministral, LLaVA, Pixtral, InternVL, Llama 4, MiniCPMV). See Chat templates.
If using a custom GGUF, try a different export from Hugging Face with richer metadata.
Inspect [MM] selected template: ... in logs — if it says “fallback”, detection failed.

Repeated first token or truncated reply

Symptom: the response loops on the first few tokens or cuts off mid-sentence after one word.

Root cause: MaxTokens too low for a reasoning model, or KV cache cleanup too aggressive.

Fix:

Raise ChatParameters.MaxTokens to 1024 or 2048.
Change CacheCleanupStrategy to KeepSystemPromptAndFirstUserMessage if the model loses the image reference during generation.

Slow first response

Symptom: 30+ seconds before the first token on a well-specced GPU.

Root cause: projector loaded on CPU while base model is on GPU, or the image is very large.

Fix:

Set MtmdContextParameters.UseGpu = true.
Downscale large images before attaching.
Enable MtmdContextParameters.PrintTimings to see per-stage time.

Image format error

Symptom: InvalidOperationException: Unknown or unsupported image format.

Fix: convert to JPEG, PNG, BMP, GIF, or WebP. See Attaching images.

Image too large

Symptom: InvalidOperationException: Image size exceeds maximum allowed (50MB).

Fix: downscale or recompress. The projector processes images at 336-448 pixels regardless — oversized source is wasted work.

Collecting a bug report

When escalating to support:

Run with EnableDebugLogging = true, MtmdContextParameters.Verbosity = 3, MtmdContextParameters.PrintTimings = true.
Capture the full log file.
Pipe through parse_mm_logs.zsh for a sectioned version.
Include:
- Preset class name.
- Model and mmproj Hugging Face paths (or custom GGUF details).
- A sample image that reproduces the issue (if legally shareable).
- The exact prompt that triggered the failure.
- The sectioned log.

Submit via the Aspose Support Forum or a paid support ticket.

What’s next

Chat templates — the eight supported template families.
Multimodal context parameters — GPU offload and verbosity for the projector.
Attaching images — correct image input.

Documentation – Multimodal

Net: Vision presets

Available presets

Picker

Memory

Using a vision preset

Customizing a vision preset

What’s next

Net: Attaching images

Quick example

Supported formats

50 MB per-attachment limit

Multiple images per message

MediaAttachment class

FromBytes — auto-detecting format

Validate

MediaFormat enum

ChatMessage with media

Common errors

What’s next

Net: Chat templates

Supported templates

How selection works

Built-in preset → template mapping

Custom vision models

Checking which template was picked

What’s next

Net: Debugging vision

Turn on debug logging

Log tag taxonomy

The parse_mm_logs.zsh helper

Common vision failures

Garbled output or literal marker tokens in reply

Repeated first token or truncated reply

Slow first response

Image format error

Image too large

Collecting a bug report

What’s next

`MediaAttachment` class

`FromBytes` — auto-detecting format

`Validate`

`MediaFormat` enum

`ChatMessage` with media

The `parse_mm_logs.zsh` helper