backend

sentence_transformers.backend.export_optimized_onnx_model(model: SentenceTransformer | SparseEncoder | CrossEncoder | MultiVectorEncoder, optimization_config: OptimizationConfig | Literal['O1', 'O2', 'O3', 'O4'], model_name_or_path: str, push_to_hub: bool = False, create_pr: bool = False, file_suffix: str | None = None) → None[source]

Export an optimized ONNX model from a SentenceTransformer, SparseEncoder, CrossEncoder, or MultiVectorEncoder model.

The O1-O4 optimization levels are defined by Optimum and are documented here: https://huggingface.co/docs/optimum-onnx/main/en/onnxruntime/usage_guides/optimization

The optimization levels are:

  • O1: basic general optimizations.

  • O2: basic and extended general optimizations, transformers-specific fusions.

  • O3: same as O2 with GELU approximation.

  • O4: same as O3 with mixed precision (fp16, GPU-only)

See the following pages for more information & benchmarks:

Parameters:

  • model (SentenceTransformer | SparseEncoder | CrossEncoder | MultiVectorEncoder) – The SentenceTransformer, SparseEncoder, CrossEncoder, or MultiVectorEncoder model to be optimized. Must be loaded with backend=”onnx”.

  • optimization_config (OptimizationConfig | Literal["O1", "O2", "O3", "O4"]) – The optimization configuration or level.

  • model_name_or_path (str) – The path or Hugging Face Hub repository name where the optimized model will be saved.

  • push_to_hub (bool, optional) – Whether to push the optimized model to the Hugging Face Hub. Defaults to False.

  • create_pr (bool, optional) – Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False.

  • file_suffix (str | None, optional) – The suffix to add to the optimized model file name. Defaults to None.

Raises:

  • ImportError – If the required packages optimum and onnxruntime are not installed.

  • ValueError – If the provided model is not a valid SentenceTransformer, SparseEncoder, CrossEncoder, or MultiVectorEncoder model loaded with backend=”onnx”.

  • ValueError – If the provided optimization_config is not valid.

Returns:

None

sentence_transformers.backend.export_dynamic_quantized_onnx_model(model: SentenceTransformer | SparseEncoder | CrossEncoder | MultiVectorEncoder, quantization_config: QuantizationConfig | Literal['arm64', 'avx2', 'avx512', 'avx512_vnni'], model_name_or_path: str, push_to_hub: bool = False, create_pr: bool = False, file_suffix: str | None = None) → None[source]

Export a quantized ONNX model from a SentenceTransformer, SparseEncoder, CrossEncoder, or MultiVectorEncoder model.

This function applies dynamic quantization, i.e. without a calibration dataset. Each of the default quantization configurations quantize the model to int8, allowing for faster inference on CPUs, but are likely slower on GPUs.

See the following pages for more information & benchmarks:

Parameters:

  • model (SentenceTransformer | SparseEncoder | CrossEncoder | MultiVectorEncoder) – The SentenceTransformer, SparseEncoder, CrossEncoder, or MultiVectorEncoder model to be quantized. Must be loaded with backend=”onnx”.

  • quantization_config (QuantizationConfig) – The quantization configuration.

  • model_name_or_path (str) – The path or Hugging Face Hub repository name where the quantized model will be saved.

  • push_to_hub (bool, optional) – Whether to push the quantized model to the Hugging Face Hub. Defaults to False.

  • create_pr (bool, optional) – Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False.

  • file_suffix (str | None, optional) – The suffix to add to the quantized model file name. Defaults to None.

Raises:

  • ImportError – If the required packages optimum and onnxruntime are not installed.

  • ValueError – If the provided model is not a valid SentenceTransformer, SparseEncoder, CrossEncoder, or MultiVectorEncoder model loaded with backend=”onnx”.

  • ValueError – If the provided quantization_config is not valid.

Returns:

None

sentence_transformers.backend.export_static_quantized_openvino_model(model: SentenceTransformer | SparseEncoder | CrossEncoder | MultiVectorEncoder, quantization_config: OVQuantizationConfig | dict | None, model_name_or_path: str, dataset_name: str | None = None, dataset_config_name: str | None = None, dataset_split: str | None = None, column_name: str | None = None, push_to_hub: bool = False, create_pr: bool = False, file_suffix: str = 'qint8_quantized') → None[source]

Export a quantized OpenVINO model from a SentenceTransformer, SparseEncoder, CrossEncoder, or MultiVectorEncoder model.

This function applies Post-Training Static Quantization (PTQ) using a calibration dataset, which calibrates quantization constants without requiring model retraining. Each default quantization configuration converts the model to int8 precision, enabling faster inference while maintaining accuracy.

See the following pages for more information & benchmarks:

Parameters:

  • model (SentenceTransformer | SparseEncoder | CrossEncoder | MultiVectorEncoder) – The SentenceTransformer, SparseEncoder, CrossEncoder, or MultiVectorEncoder model to be quantized. Must be loaded with backend=”openvino”.

  • quantization_config (OVQuantizationConfig | dict | None) – The quantization configuration. If None, default values are used.

  • model_name_or_path (str) – The path or Hugging Face Hub repository name where the quantized model will be saved.

  • dataset_name (str, optional) – The name of the dataset to load for calibration. If not specified, the sst2 subset of the glue dataset will be used by default.

  • dataset_config_name (str, optional) – The specific configuration of the dataset to load.

  • dataset_split (str, optional) – The split of the dataset to load (e.g., ‘train’, ‘test’). Defaults to None.

  • column_name (str, optional) – The column name in the dataset to use for calibration. Defaults to None.

  • push_to_hub (bool, optional) – Whether to push the quantized model to the Hugging Face Hub. Defaults to False.

  • create_pr (bool, optional) – Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False.

  • file_suffix (str, optional) – The suffix to add to the quantized model file name. Defaults to qint8_quantized.

Raises:

  • ImportError – If the required packages optimum and openvino are not installed.

  • ValueError – If the provided model is not a valid SentenceTransformer, SparseEncoder, CrossEncoder, or MultiVectorEncoder model loaded with backend=”openvino”.

  • ValueError – If the provided quantization_config is not valid.

Returns:

None