Spark
Sign In
Skill

Train and fine-tune sentence-transformer embedding models

A router skill for training SentenceTransformer, CrossEncoder, or SparseEncoder models, pointing to production script templates and required references.

Get skill
Works with huggingfaceelasticsearchopensearchlucene
huggingface
Maintainer?
79
Spark score
out of 100
Updated 23 days ago
Version 1.0.0
Add to Favorites

Why it matters

Train or fine-tune sentence-transformer models for retrieval, similarity search, clustering, classification, and reranking tasks. The skill routes you to production-ready templates and reference documentation for bi-encoders (dense embeddings), cross-encoders (rerankers), and sparse encoders (SPLADE), ensuring you use the correct loss functions, evaluators, and training arguments for your specific use case.

Outcomes

What it gets done

01

Generate training scripts from production templates for SentenceTransformer, CrossEncoder, or SparseEncoder models

02

Configure loss functions and evaluators matched to your data shape and task type

03

Set up training arguments with proper precision, warmup, checkpointing, and Hub push settings

04

Validate training runs with baseline comparisons and verdict scoring (WIN/MARGINAL/REGRESSION)

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/ag-train-sentence-transformers | bash

Overview

Train a sentence-transformers Model

This skill routes sentence-transformers training to the correct model type (SentenceTransformer, CrossEncoder, SparseEncoder), required reference files, and production script template, enforcing non-negotiable contracts like baseline capture, a scrapeable VERDICT line, and type-specific gotchas. Use it when training or fine-tuning any sentence-transformers model - bi-encoder embeddings, cross-encoder reranking, or sparse SPLADE retrieval.

What it does

Acts as a router - not a manual - for training or fine-tuning sentence-transformers models across three types: SentenceTransformer (bi-encoder, maps input to a fixed-dim dense vector, for retrieval/similarity/clustering/classification/paraphrase mining/dedup/multimodal), CrossEncoder (reranker, scores query-passage pairs jointly for two-stage retrieval), and SparseEncoder (SPLADE, sparse vocabulary-space vectors for learned-sparse retrieval on inverted-index backends like Elasticsearch/OpenSearch/Lucene) - with tiebreaker phrases ("embedding"/"vector search" → SentenceTransformer, "rerank"/"two-stage" → CrossEncoder, "SPLADE"/"sparse" → SparseEncoder) and an explicit instruction to ask if still ambiguous. It explicitly forbids synthesizing a training script from the router file alone: the per-type production template (scripts/train_<type>_example.py) must be copied as the actual starting point, since it contains load-bearing scaffolding - an autocast helper, a model-card class, a logger-silencing list, force=True, seeding, TF32 config, version-compatible imports, and named-evaluator metric handling - that prior agent runs have repeatedly gotten wrong when rolling their own from a synthesized snippet. Required reading before writing any code is split into per-type files (losses mapped to data shape and incompatibilities like Cached* losses conflicting with gradient checkpointing; evaluators mapped to tasks and their metric_for_best_model key format; for SentenceTransformer specifically, model architecture and pooling rules) and cross-cutting files read regardless of type: training-argument knobs and precision rules (load fp32 and autocast bf16/fp16, never torch_dtype=bfloat16 directly), dataset column-matching and hard-negative-mining recipes, base-model selection including a flagged ModernBERT max_seq_length=8192 trap, and a symptom-indexed troubleshooting file to skim even on a healthy run since its "metrics don't improve" and "hub push fails" entries catch frequent bugs cheaply before they fire. Hardware, HF Jobs execution, and prompt-prefix (query:/passage: style) reference files load only when applicable. Defaults run locally unless hardware can't fit the job, execute a single run and only propose iteration if the result is weak or marginal, and push to the Hub at end-of-run wrapped in try/except (with additional in-trainer push enabled specifically on HF Jobs' ephemeral environment). The produced script must satisfy non-negotiable contracts: capture a baseline_eval score before trainer.train() starts; emit a single scrapeable end-of-run line in the exact format VERDICT: WIN|MARGINAL|REGRESSION | score=... | baseline=... | delta=...; silence noisy HTTP/HF-download loggers to WARNING; tee logs to a run-named file; wrap the final model.push_to_hub(...) in try/except; smoke-test with max_steps=1 on a tiny data slice before any long run; for CrossEncoder specifically, include EarlyStoppingCallback(patience>=3) since rerankers often peak mid-training and regress afterward; and for SparseEncoder specifically, log query/corpus active-dimension counts on the verdict line, since high nDCG with collapsed sparsity isn't actually a win. The seven-step workflow: identify the model type, load that type's required reading, copy the matching production template, replace the model/dataset/run names and loss/evaluator for the actual task (cross-checked against the loss and evaluator reference files), smoke-test, run, then log the experiment and propose iteration if the verdict is weak or marginal.

pip install "sentence-transformers[train]>=5.0"
pip install trackio
hf auth login

When to use - and when NOT to

Use it when training or fine-tuning any sentence-transformers model - a bi-encoder embedding model, a cross-encoder reranker, or a sparse SPLADE encoder.

Inputs and outputs

Input is a model type (or ambiguous request to be disambiguated), a base model, and a dataset. Output is a training script copied from the correct production template, customized with the right loss/evaluator/data shape, smoke-tested, run, and ending in a scrapeable VERDICT line plus a Hub-pushed model.

Integrations

Uses the sentence-transformers[train] library, Hugging Face Hub authentication for model push, optional experiment trackers (trackio, wandb, tensorboard, mlflow), and Hugging Face Jobs for hardware that exceeds local capacity.

Who it's for

ML engineers training or fine-tuning embedding, reranking, or sparse retrieval models who want the correct production-grade training script scaffolding (baseline capture, smoke tests, verdict reporting, type-specific gotchas like CrossEncoder early stopping) rather than a synthesized script missing load-bearing details prior runs have gotten wrong.

Source README

Train a sentence-transformers Model

When to Use

Use this skill when you need train or fine-tune sentence-transformers models across SentenceTransformer (bi-encoder; dense or static embedding model; for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), CrossEncoder (reranker; pair scoring for two-stage retrieval / pair...

This SKILL.md is a router, not a manual. It tells you which references and example scripts to load for your task. The actual content - recommended losses, evaluators, training-script structure, model selection, training-arg knobs, troubleshooting - lives in references/ and scripts/.

Do not synthesize a training script from this file alone. Open the per-type production template (scripts/train_<type>_example.py) and copy it as your starting point. The templates contain load-bearing scaffolding (autocast helper, model-card class, logger silencing list, force=True, seed, TF32, version-compatible imports, named-evaluator metric handling) that prior agent runs have repeatedly missed when rolling their own from a synthesized snippet.

1. Identify the model type

Tag Class What it does When to pick
[SentenceTransformer] SentenceTransformer (bi-encoder) Maps each input to a fixed-dim dense vector Retrieval, similarity, clustering, classification, paraphrase mining, dedup
[CrossEncoder] CrossEncoder (reranker) Scores (query, passage) pairs jointly Two-stage retrieval (rerank top-100 from bi-encoder), pair classification
[SparseEncoder] SparseEncoder (SPLADE) Sparse vectors over the vocabulary Learned-sparse retrieval, inverted-index backends (Elasticsearch / OpenSearch / Lucene)

Tiebreakers when the request is ambiguous: "embedding model" / "vector search" / "similarity" → [SentenceTransformer]. "rerank" / "ranker" / "two-stage" → [CrossEncoder]. "SPLADE" / "sparse" / "inverted index" → [SparseEncoder]. If still unclear, ask.

2. Required reading

Read these in full before writing any code. Do not triage by perceived relevance.

Per-type - always required

[SentenceTransformer]

  • references/losses_sentence_transformer.md - loss-to-data-shape mapping; BatchSamplers.NO_DUPLICATES requirement for MNRL-family; Cached* ↔ gradient_checkpointing incompatibility.
  • references/evaluators_sentence_transformer.md - evaluator-to-task mapping; metric_for_best_model key construction (named vs unnamed); per-evaluator primary_metric values.
  • references/model_architectures.md - encoder vs decoder vs static vs Router pipelines; pooling rules (mean / cls / lasttoken); auto-mean-pooling behavior for fresh-start MLM bases.
  • scripts/train_sentence_transformer_example.py - production template; copy this as your starting point.

[CrossEncoder]

  • references/losses_cross_encoder.md - pointwise / pairwise / listwise / distillation; pos_weight derivation; activation_fn=Identity() mandatory for non-BCE losses (silent eval-rank collapse otherwise).
  • references/evaluators_cross_encoder.md - CrossEncoderRerankingEvaluator recipe; named-evaluator key format eval_{name}_{primary_metric}.
  • scripts/train_cross_encoder_example.py - production template; copy this as your starting point.

[SparseEncoder]

  • references/losses_sparse_encoder.md - SpladeLoss wrapper requirement; FLOPS regularizer weights; smoke-test active-dim ramp behavior.
  • references/evaluators_sparse_encoder.md - SparseNanoBEIREvaluator (English-only) and the in-domain alternative; eval_{name}_{primary_metric} key format.
  • scripts/train_sparse_encoder_example.py - production template; copy this as your starting point.

Cross-cutting - always required (regardless of task)

  • references/training_args.md - TrainingArguments knobs, precision rules (load fp32 + autocast bf16/fp16; never torch_dtype=bfloat16), warmup_steps (float) vs deprecated warmup_ratio, save_steps must be a multiple of eval_steps for load_best_model_at_end, schedulers, HPO, tracker, resume, hub-push variants.
  • references/dataset_formats.md - column-matching rules (label name auto-detection; column-order-not-name); reshaping recipes; hard-negative mining options.
  • references/base_model_selection.md - discovery commands; per-type model namespaces; ModernBERT-family max_seq_length=8192 trap; datasets >= 4 script-loader rejection; non-English starting-point shortcuts.
  • references/troubleshooting.md - symptom-indexed failure recipes. Skim the section headings on every run, even a healthy one; the "Metrics don't improve" and "Hub push fails" entries cover bugs that bite frequently and are cheaper to recognize before they fire than to debug after.

Cross-cutting - load when applicable

  • references/hardware_guide.md - VRAM sizing, multi-GPU, FSDP / DeepSpeed, HF Jobs flavors. Required for >24GB models, multi-GPU, or HF Jobs runs.
  • references/hf_jobs_execution.md - required when running on HF Jobs.
  • references/prompts_and_instructions.md - required when using prompt-tuned bases (E5, BGE, GTE, Qwen3-Embedding, Instructor, Nomic, etc.) or adding query: / passage: style prefixes.

Variant scripts (open when the task matches)

  • [SentenceTransformer] scripts/train_sentence_transformer_<matryoshka|multi_dataset|with_lora|distillation|make_multilingual|static_embedding>_example.py.
  • [CrossEncoder] scripts/train_cross_encoder_<distillation|listwise>_example.py.
  • [SparseEncoder] scripts/train_sparse_encoder_distillation_example.py.
  • Hard-negative mining CLI - scripts/mine_hard_negatives.py.

3. Defaults

Override only if the user specifies otherwise:

  • Local execution. Pitch HF Jobs only if local hardware can't fit the job.
  • Single run. After it completes, propose experimentation if the user would benefit (weak/marginal verdict, "see how high you can push it" framing, etc.). Iteration rules in references/training_args.md (Experimentation section).
  • Public Hub push at end-of-run, wrapped in try-except. On HF Jobs (ephemeral env) ALSO enable in-trainer push (push_to_hub=True + hub_strategy="every_save"); details in references/hf_jobs_execution.md.

4. Constraints the produced script must satisfy

These are non-negotiable contracts. Implementation lives in the production templates and references - do not reinvent.

  • Capture the pre-training evaluator score as baseline_eval before trainer.train().
  • Emit a single end-of-run line: VERDICT: WIN|MARGINAL|REGRESSION | score=... | baseline=... | delta=.... A monitor scrapes for this.
  • Silence httpx, httpcore, huggingface_hub, urllib3, filelock, fsspec to WARNING (otherwise HF download URLs flood the agent's context).
  • Tee logs to logs/{RUN_NAME}.log.
  • End with model.push_to_hub(...) wrapped in try/except.
  • Smoke-test before any long run (max_steps=1 + tiny dataset slice). The production templates show one common pattern (SMOKE_TEST env var).
  • [CrossEncoder] Include EarlyStoppingCallback(patience>=3) - CE rerankers often peak mid-training and regress.
  • [SparseEncoder] Log query_active_dims / corpus_active_dims on the verdict line; high nDCG with collapsed sparsity is not a win. The keys come back name-prefixed (e.g. ..._query_active_dims); use suffix matching to pluck them - see the SPARSE production template for the exact pattern.

5. Workflow

  1. Identify the model type (§1). Ask if ambiguous.
  2. Load the §2 required-reading files for that type.
  3. Open scripts/train_<type>_example.py and copy it as your starting point.
  4. Replace MODEL_NAME, DATASET_NAME, RUN_NAME, the loss, and the evaluator with the user's task. Cross-check loss/data-shape match against references/losses_<type>.md; cross-check the metric_for_best_model key against references/evaluators_<type>.md (named evaluators format the key as eval_{name}_{primary_metric}).
  5. Smoke-test (max_steps=1).
  6. Run.
  7. After the run, append to logs/experiments.md and propose iteration if the verdict is weak/marginal.

Prerequisites

pip install "sentence-transformers[train]>=5.0"        # add [train,image] / [audio] / [video] for [SentenceTransformer] multimodal
pip install trackio                                    # optional tracker; or wandb / tensorboard / mlflow
hf auth login                                          # or set HF_TOKEN with write scope (for Hub push)

GPU strongly recommended. CPU works only for demos and [SentenceTransformer] StaticEmbedding.

Limitations

  • Use this skill only when the task clearly matches its upstream product or API scope.
  • Verify commands, API behavior, pricing, quotas, credentials, and deployment effects against current official documentation before making changes.
  • Do not treat generated examples as a substitute for environment-specific tests, security review, or user approval for destructive or costly actions.

FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.