D-FINE-seg

Real-Time Object Detection, Instance and Semantic Segmentation

Quick Start • Usage • Export • Inference • Benchmarks • Video Tutorial • Colab

D-FINE-seg is a framework for real-time object detection, instance segmentation, and semantic segmentation - one codebase, one config flag (task: detect | segment | sem_seg), five model sizes (N -> X).

• End-to-end workflow - dataset prep -> training (DDP, EMA, AMP, mosaic) -> export (ONNX, TensorRT, OpenVINO, CoreML, LiteRT) -> benchmarked multi-backend inference
• Accuracy - on Cityscapes, beats YOLO26 and RF-DETR on detection & instance-seg F1 and leads mIoU on semantic segmentation, at real-time latency with 2-3x fewer params; also higher F1 than YOLO26 on TACO and VisDrone (TensorRT FP16, end-to-end protocol)
• Paper - D-FINE-seg: Object Detection and Instance Segmentation Framework with Multi-Backend Deployment
• Not a fork: the detection core follows the D-FINE paper; segmentation heads, training, export and inference are implemented from scratch.

One frame, three tasks, one config flag:

_{Full tables below}

Highlights

• Instance segmentation head (task: segment) - lightweight mask head on top of D-FINE's HybridEncoder PAN outputs: stride 8/16/32 features fused to 1/4 resolution, then a dot-product between per-query mask embeddings (3-layer MLP) and the shared mask features yields per-instance masks
• Semantic segmentation head (task: sem_seg) - reuses the pretrained instance-seg mask fuser on full-frame features, followed by a small conv neck and 1x1 classifier: no queries, no NMS
• Mask-aware training - box-cropped BCE + Dice mask losses (instance seg) and CE + multi-class soft Dice with ignore_index (semantic seg), mask supervision inside contrastive denoising, and Dice + sigmoid-focal mask costs in the Hungarian matcher - all train-time only, zero inference cost
• COCO-pretrained weights for detection and instance segmentation, auto-downloaded on first use - fine-tuning starts from a trained mask decoder, not from scratch
• Multi-channel inputs - train on RGB + thermal / depth / NIR stacks (4-channel .npy), not just RGB
• Modern training stack - Muon optimizer, DDP, EMA, mosaic + affine augs, OneCycle, early stopping, WandB
• Beyond the model - ByteTrack tracking, SAM3 auto-labeling, Gradio demo, INT8 quantization (OpenVINO / CoreML / LiteRT)

Quick Start

Installation

pip install dfine-seg           # inference + training
pip install 'dfine-seg[all]'    # + every export backend, SAM3, Gradio demo

COCO-pretrained weights (detection and instance segmentation) auto-download from Hugging Face on first use - no manual download needed.

Two-line predict:

from dfine_seg import load_model, read_image, Visualizer

model = load_model("s")                              # COCO detection, weights auto-downloaded
model = load_model("s", task="segment")              # COCO instance segmentation
model = load_model("output/models/exp/model.pt")     # your checkpoint - size/task/classes/input size auto-detected, supports: .pt | .engine | .onnx | .xml

img = read_image("path/to/image.jpg")
out = model(img)[0]
print(out["boxes"], out["scores"], [model.names[int(i)] for i in out["labels"]])

drawn = Visualizer(model)(img, out)                  # annotated BGR copy - boxes, masks or a sem_seg overlay

load_model returns the very same wrapper you would construct by hand (dfine_seg/infer/) - it resolves the weights and picks the backend, then gets out of the way. Extra keyword arguments pass straight through (load_model("s", conf_thresh=0.3)), output tensors stay on the device the model ran on, and those wrapper files remain self-contained enough to copy into your own app. Visualizer reads the class count and names off the model it is given, then draws whatever that model returned - boxes, instance masks or a dense label map - so one call covers every task (BGR uint8 in, BGR uint8 out).

To train from a pip install, materialize a config and go:

dfine init          # writes ./config.yaml - edit train.root and train.label_to_name
dfine split
dfine train         # Hydra overrides work: dfine train model_name=m train.epochs=100

From source (contributors)

git clone https://github.com/ArgoHA/D-FINE-seg.git
cd D-FINE-seg
uv sync

This creates a .venv/ with the package installed editable and every extra present, pinned by uv.lock. Activate it with source .venv/bin/activate, or run anything via uv run ... (the Makefile already does this).

Pretrained weights are auto-downloaded from Hugging Face on first use, so no manual setup is needed - into pretrained/ for the config-driven commands, and into the shared Hugging Face cache for load_model("s") when there is no pretrained/ copy to reuse. To download manually instead, grab dfine_<size>_<dataset>.pt (size ∈ {n, s, m, l, x}, dataset ∈ {coco, obj2coco}) and place it in pretrained/. Segmentation weights are also available in the Hugging Face model card.

Prepare Your Data

Two annotation formats are supported: YOLO (default) and COCO JSON. Semantic segmentation uses PNG masks instead (see below).

YOLO format (default)

data/dataset/
├── images/    # all images: .jpg, .png, etc. (.npy for multi-channel - see below)
└── labels/    # all labels: one .txt per image (same filename stem)

Detection labels: class_id xc yc w h (normalized)

Segmentation labels: class_id x1 y1 x2 y2 ... xN yN (normalized polygon coordinates)

Input types & channel order: 3-channel .jpg/.png (BGR, read via cv2.imread), 3-channel .npy (RGB, read via np.load), or 4-channel .npy (RGB+extras, e.g. RGB+thermal).

Semantic segmentation masks (task: sem_seg)

data/dataset/
├── images/    # same as YOLO layout
└── labels/    # one single-channel uint8 .png per image (same stem), pixel value = class id

Every pixel gets a class from label_to_name (background included). Pixels with value train.sem_seg.ignore_index (default 255) are excluded from loss and metrics and during inference 255 is the "background" or "ignored" class. dfine split works unchanged; keep_ratio: True is supported (letterbox pad is filled with ignore_index, so pad pixels don't supervise); coco_dataset: True is not supported for this task.

Multi-channel inputs (RGB + thermal / depth / NIR / …)

Set train.in_channels: 4 (3 or 4 supported) to train on RGB + one extra modality
(thermal / depth / NIR). Stacks are .npy uint8 HWC arrays in images/ (RGB in planes
0-2, extras after) with YOLO labels as usual; a mismatched channel count is skipped with
a warning. See dfine_seg/etl/m3fd_to_yolo.py for a ready-made RGB+thermal converter.

COCO JSON format

Place standard COCO JSON annotation files alongside your images folder. Splits are detected automatically by filename:

data/dataset/
├── images/       # all images
├── train.json    # COCO-format annotations for train split
├── val.json      # COCO-format annotations for val split
└── test.json     # (optional) COCO-format annotations for test split

Enable COCO mode by setting coco_dataset: True in your config (see below). No CSV split generation step is needed - the splits are read directly from the JSON files.

If you only have a single coco.json, run dfine split to produce train.json / val.json (and test.json when the ratios leave room) from it. It splits by image using the same split: ratios, seed and ignore_negatives as the YOLO path, keeps each image's annotations with it, and copies categories / info / licenses into every output.

Configure

Edit config.yaml - key settings:

task: detect  # detect | segment | sem_seg
exp_name: my_exp  # experiment name (used in output paths)
model_name: s  # n / s / m / l / x

train:
  root: /path/to/project  # project root, will be used for outputs
  data_path: /path/to/dataset  # folder with images/ and labels/ (YOLO) or *.json files (COCO)
  coco_dataset: False  # set True to use COCO JSON annotations (train.json / val.json / test.json)
  label_to_name:
    0: class_a
    1: class_b
  epochs: 75
  batch_size: 8
  img_size: [640, 640]  # (h, w)

Usage

Notes:

• Most commands read config.yaml and work off your trained run; exceptions - init (writes the config), predict, hw_bench, demo, version (no config needed). Every command's flags, defaults and examples: dfine <command> -h.
• YOLO format: dfine train requires train.csv and val.csv in train.data_path (generated by dfine split).
• COCO format: set coco_dataset: True - train.json and val.json are loaded directly; dfine split is only needed if you have a single coco.json to split.
• dfine infer runs Torch inference on train.path_to_test_data and writes to train.infer_path. infer / export / bench auto-pick the latest <exp_name>_<date> run under train.path_to_save.

Every dfine command also has a make alias (make train == dfine train), and any config key can be overridden inline:

dfine train exp_name=my_exp model_name=m train.epochs=100

Enable DDP (multi-GPU) by setting train.ddp.enabled: True and train.ddp.n_gpus: N in config. Then just run dfine train - it auto-launches with torchrun.

Training Features

Export

Tip: FP16 is the best latency/accuracy trade-off for GPU (TensorRT) and CPU (OpenVINO). For Apple Silicon (CoreML), FP32 is faster.

Warning: run TensorRT engines at batch 1. On TRT 10.13.3.9 a batched engine does not compute batch elements independently - feeding four identical images through a single execute_async_v3 call returns four different results (score spread up to 0.20, and different labels), so a detection's score depends on what it happened to be batched with. Batch 1 is exact, and reproduces in raw TensorRT for FP16 and FP32 and for every optimization-profile shape, while torch and ONNX Runtime stay identical across slots (NVIDIA/TensorRT#4813).

After export, a parity self-check (export.parity, on by default) runs each backend on a shared input and writes one cosine per backend - over the sorted top-K detection scores vs torch - to parity.csv next to the weights.

For task: sem_seg every backend gets the same fused-argmax graph: a single int32 sem_seg output [B, H, W] (label map at input resolution, no detection postprocessor), and parity compares per-pixel argmax agreement instead of score cosine.

Inference

Backends

Six inference backends in dfine_seg/infer/:

Output contract: detection / instance segmentation wrappers return labels, boxes, scores (+ masks [N, H, W] for segment); sem_seg wrappers return a single sem_seg [H, W] label map at original image resolution. For sem_seg, dfine infer writes palette overlays + GT-style grayscale PNG label maps (crops and tracking are box-based and skipped).

Also provided:

• Bytetrack - simple implementation of object tracker
• SAM3 - text-promptable zero-shot segmentation for auto-labeling (multi-class: repeat --prompt or pass "car, person")

Multi-Object Tracking

A simplified ByteTrack (Zhang et al., ECCV 2022) is included for persistent object tracking across video frames - uses constant-velocity motion prediction with EMA-smoothed velocity instead of a Kalman filter, blends IoU with centroid distance in the match cost, and does per-class matching by default.

Gradio Demo

dfine demo         # == make demo   (pip: pip install 'dfine-seg[demo]')

A web UI for running inference on uploaded images and videos (or a webcam snapshot). It starts on COCO detection s - no configuration, weights download on first use. The Model panel then swaps in any other model at runtime: a size preset, or a path/upload of your own .pt / .onnx / .engine / .xml, with a box for the class names. detect, segment and sem_seg checkpoints all render, and SAM3 is selectable as a second backend for text-promptable segmentation (prompts are comma- or newline-separated - each one becomes a class).

It serves on 0.0.0.0:7860 (LAN-reachable) and prints a warning on startup: the Model panel loads any path the browser sends, so anyone who can reach the port can load files off this machine. dfine demo --host 127.0.0.1 restores local-only.

Benchmarks

Metrics

Detection / instance segmentation - GT objects and predictions are matched one-to-one: a prediction is a TP if IoU > 0.5 (box for detect, mask for segment) and the class matches; only the highest-IoU prediction per GT counts, extra overlapping ones are FPs; a class mismatch is one FP + one FN.

• F1 / Precision / Recall - computed from those TP/FP/FN counts at train.conf_thresh.
• IoU (penalized) - mean IoU over all outcomes: TPs contribute their IoU, FPs and FNs contribute 0 (= sum of TP IoUs / (TPs + FPs + FNs)).
• mAP_50 / mAP_50_95 - COCO-style average precision (mask versions for segment).

Semantic segmentation - all metrics come from one pixel confusion matrix accumulated over the whole eval set at original image resolution (ignore_index pixels excluded). A pixel of class i predicted as j counts as an FN for i and an FP for j - each confused pixel penalizes both classes.

• mIoU (decision metric) - macro-averaged: per-class pixel IoU = TP / (TP + FP + FN), averaged over classes present in GT, so every class has equal weight regardless of pixel count.
• pixel_acc - micro: fraction of all valid pixels classified correctly, so it is dominated by large classes.

Cityscapes - vs YOLO26 and RF-DETR (fine-tuning)

This is the main dataset where numbers are being updated. Other benchmarks are older and are not updated with every latency/accuracy improvement in this repo.
500 Cityscapes val images at original 2048x1024, TensorRT 10.13 FP16, batch 1, RTX 5070 Ti. Every framework runs its own shipped inference code, scored by one validator against the same GT. Confidence thresholds were calculated for each framework separately to maximixe the F1. Two latency columns - e2e (end-to-end, including each framework's CPU preprocessing) and engine (pure TensorRT execute) - because they can disagree. Full protocol and every known asymmetry: cityscapes-benchmark.

Detection

Instance segmentation

Semantic segmentation

RF-DETR has no semantic segmentation task, so this one is D-FINE-seg vs YOLO26.

Other datasets

Outputs

Result examples

Training

Benchmarking

WandB dashboard

Inference

Citation

If you use D-FINE-seg in your research, please cite:

@article{saakyan2026dfineseg,
  title={D-FINE-seg: Object Detection and Instance Segmentation Framework with multi-backend deployment},
  author={Saakyan Argo and Solntsev Dmitry},
  eprint={2602.23043},
  journal={arXiv preprint arXiv:2602.23043},
  year={2026}
}

And the original D-FINE paper:

@misc{peng2024dfine,
      title={D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement},
      author={Yansong Peng and Hebei Li and Peixi Wu and Yueyi Zhang and Xiaoyan Sun and Feng Wu},
      year={2024},
      eprint={2410.13842},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

The detection core is based on the D-FINE paper and architecture. The mask head design follows the Mask DINO paradigm. Thank you to both teams for their excellent work.

Benchmarks in this project use the VisDrone and TACO datasets. We thank the authors for making these datasets publicly available.