o S"gF@sddlmZddlZddlZddlZddlmZddlmZm Z m Z ddl Z ddl m Z mZeeZeraddlmZzddlmZWn eyLYnwz ddlmZmZWn ey`Ynw d,d-ddZ d,d.ddZ     d/d0d#d$Z  %d1d2d*d+ZdS)3) annotationsN)Path) TYPE_CHECKINGCallableLiteral)disable_datasets_cachingis_datasets_availableSentenceTransformer)OVQuantizationConfig)OptimizationConfigQuantizationConfigFmodelr optimization_config4OptimizationConfig | Literal['O1', 'O2', 'O3', 'O4']model_name_or_pathstr push_to_hubbool create_pr file_suffix str | NonereturnNonec sddlm}ddlm}zddlm}m} ddlm} Wn t y't dwt ||r@t |r@t |d|r@t |dj |sDt d|dj } | | t trf| jvr\t dp_t| d urld tfd d d |||ddd S)a Export an optimized ONNX model from a SentenceTransformer model. The O1-O4 optimization levels are defined by Optimum and are documented here: https://huggingface.co/docs/optimum/main/en/onnxruntime/usage_guides/optimization The optimization levels are: - O1: basic general optimizations. - O2: basic and extended general optimizations, transformers-specific fusions. - O3: same as O2 with GELU approximation. - O4: same as O3 with mixed precision (fp16, GPU-only) See https://sbert.net/docs/sentence_transformer/usage/efficiency.html for more information & benchmarks. Args: model (SentenceTransformer): The SentenceTransformer model to be optimized. Must be loaded with `backend="onnx"`. optimization_config (OptimizationConfig | Literal["O1", "O2", "O3", "O4"]): The optimization configuration or level. model_name_or_path (str): The path or Hugging Face Hub repository name where the optimized model will be saved. push_to_hub (bool, optional): Whether to push the optimized model to the Hugging Face Hub. Defaults to False. create_pr (bool, optional): Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False. file_suffix (str | None, optional): The suffix to add to the optimized model file name. Defaults to None. Raises: ImportError: If the required packages `optimum` and `onnxruntime` are not installed. ValueError: If the provided model is not a valid SentenceTransformer model loaded with `backend="onnx"`. ValueError: If the provided optimization_config is not valid. Returns: None rr  Transformer)ORTModelForFeatureExtraction ORTOptimizer)AutoOptimizationConfigPlease install Optimum and ONNX Runtime to use this function. You can install them with pip: `pip install optimum[onnxruntime]` or `pip install optimum[onnxruntime-gpu]`]The model must be a Transformer-based SentenceTransformer model loaded with `backend="onnx"`.z\optimization_config must be an OptimizationConfig instance or one of 'O1', 'O2', 'O3', 'O4'.N optimizedcj|dSN)r)optimizesave_dirrr optimizer_/mnt/skqttb/ctump_chatbot/chatbot/lib/python3.10/site-packages/sentence_transformers/backend.pyjz-export_optimized_onnx_model..export_optimized_onnx_modelonnxexport_functionexport_function_nameconfigrrrrbackend)sentence_transformersr (sentence_transformers.models.Transformerroptimum.onnxruntimerr!optimum.onnxruntime.configurationr ImportError isinstancelen auto_model ValueErrorfrom_pretrainedr_LEVELSgetattrsave_or_push_to_hub_model) rrrrrrr rrrr ort_modelr)r'r*r-sT '         r-quantization_configFQuantizationConfig | Literal['arm64', 'avx2', 'avx512', 'avx512_vnni']c s.ddlm}ddlm}zddlm}m} ddlm} Wn t y't dwt ||r@t |r@t |d|r@t |dj |sDt d|dj } | | t trvdvr[t d d d } t| d d pujjd | d urjjdtfddd|||ddd S)a Export a quantized ONNX model from a SentenceTransformer model. This function applies dynamic quantization, i.e. without a calibration dataset. Each of the default quantization configurations quantize the model to int8, allowing for faster inference on CPUs, but are likely slower on GPUs. See https://sbert.net/docs/sentence_transformer/usage/efficiency.html for more information & benchmarks. Args: model (SentenceTransformer): The SentenceTransformer model to be quantized. Must be loaded with `backend="onnx"`. quantization_config (QuantizationConfig): The quantization configuration. model_name_or_path (str): The path or Hugging Face Hub repository name where the quantized model will be saved. push_to_hub (bool, optional): Whether to push the quantized model to the Hugging Face Hub. Defaults to False. create_pr (bool, optional): Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False. file_suffix (str | None, optional): The suffix to add to the quantized model file name. Defaults to None. Raises: ImportError: If the required packages `optimum` and `onnxruntime` are not installed. ValueError: If the provided model is not a valid SentenceTransformer model loaded with `backend="onnx"`. ValueError: If the provided quantization_config is not valid. Returns: None rr r)r ORTQuantizer)AutoQuantizationConfigrr )arm64avx2avx512 avx512_vnnizqquantization_config must be an QuantizationConfig instance or one of 'arm64', 'avx2', 'avx512', or 'avx512_vnni'.NF) is_static_ _quantizedcr"r#quantizer%rrB quantizerr)r*r+r,z5export_dynamic_quantized_onnx_model..#export_dynamic_quantized_onnx_modelr.r/)r4r r5rr6rrDr7rEr8r9r:r;r<r=rr? weights_dtypenamelowerr@) rrBrrrrr rrrDrErAquantization_config_namer)rOr*rQusV !        rQqint8_quantized"OVQuantizationConfig | dict | None dataset_namedataset_config_name dataset_split column_namec  sddlm} ddlm} zddlm} m} m}m}Wn t y%t dwt s-t dt | rEt rEt d| rEt dj | sItd|durP|}dj }| |d ||td d |||fDr}td d |||fDs}td fd d|dur|nd}|dur|nd}|dur|nd}durndtj||fdd|dur|jnd|dWdn1swYtfddd||||| dddS)a Export a quantized OpenVINO model from a SentenceTransformer model. This function applies Post-Training Static Quantization (PTQ) using a calibration dataset, which calibrates quantization constants without requiring model retraining. Each default quantization configuration converts the model to int8 precision, enabling faster inference while maintaining accuracy. See https://sbert.net/docs/sentence_transformer/usage/efficiency.html for more information & benchmarks. Args: model (SentenceTransformer): The SentenceTransformer model to be quantized. Must be loaded with `backend="openvino"`. quantization_config (OVQuantizationConfig | dict | None): The quantization configuration. If None, default values are used. model_name_or_path (str): The path or Hugging Face Hub repository name where the quantized model will be saved. dataset_name(str, optional): The name of the dataset to load for calibration. If not specified, the `sst2` subset of the `glue` dataset will be used by default. dataset_config_name (str, optional): The specific configuration of the dataset to load. dataset_split (str, optional): The split of the dataset to load (e.g., 'train', 'test'). Defaults to None. column_name (str, optional): The column name in the dataset to use for calibration. Defaults to None. push_to_hub (bool, optional): Whether to push the quantized model to the Hugging Face Hub. Defaults to False. create_pr (bool, optional): Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False. file_suffix (str, optional): The suffix to add to the quantized model file name. Defaults to `qint8_quantized`. Raises: ImportError: If the required packages `optimum` and `openvino` are not installed. ValueError: If the provided model is not a valid SentenceTransformer model loaded with `backend="openvino"`. ValueError: If the provided quantization_config is not valid. Returns: None rr r)OVConfigOVModelForFeatureExtractionr OVQuantizerzPlease install datasets, optimum-intel and openvino to use this function. You can install them with pip: `pip install datasets optimum[openvino]`zaPlease install datasets to use this function. You can install it with pip: `pip install datasets`zaThe model must be a Transformer-based SentenceTransformer model loaded with `backend="openvino"`.N)rBcss|]}|duVqdSNr)).0paramr)r)r* sz9export_static_quantized_openvino_model..zEither specify all of `dataset_name`, `dataset_config_name`, `dataset_split`, and `column_name`, or leave them all unspecified.csj|ddddS)N max_lengthiT)paddingrc truncation) tokenizerexamples)rr)r*preprocess_functionszCexport_static_quantized_openvino_model..preprocess_functiongluesst2trainsentencecs |Sr_r)rg)r[rir)r*r+%s z8export_static_quantized_openvino_model..i,)rXrYri num_samplesrZcsj|dS)N)save_directory ov_configrMr%)calibration_datasetrprPr)r*r++s&export_static_quantized_openvino_modelopenvinor/)r4r r5r optimum.intelr\r]r r^r8rr9r:r;r<r=anyallrget_calibration_datasetrnr@)rrBrrXrYrZr[rrrr rr\r]r r^ov_modelr))rqr[rrprirPr*rrsv *      "     rrr.r0rr1r3cCs|dkr d|d}n |dkrd|d}t} || |dkrCt| |} t| d| |t| d| |d | } |dkrht| |} | jd d d t| |} | |} t| | | } |rd } |rt| d d dd dd}d|d|d|d|d|d|d } t j | ||dd|d|| |dnIt||} | jd d d t| |} | |} t | | |dkrt| |d }t| |d }t ||WddSWddSWddS1swYdS)Nr.model_z.onnxrsopenvino_model_z.xmlzopenvino_model.xmlzopenvino_model.binz.binT)parentsexist_ok(z( z, z, )z )zGHello! *This pull request has been automatically generated from the [`zT`](https://sbert.net/docs/package_reference/util.html#sentence_transformers.backend.zI) function from the Sentence Transformers library.* ## Config ```python a ``` ## Tip: Consider testing this pull request before merging by loading the model from this PR with the `revision` argument: ```python from sentence_transformers import SentenceTransformer # TODO: Fill in the PR number pr_number = 2 model = SentenceTransformer( "z5", revision=f"refs/pr/{pr_number}", backend="z#", model_kwargs={"file_name": "a "}, ) # Verify that everything works as expected embeddings = model.encode(["The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium."]) print(embeddings.shape) similarities = model.similarity(embeddings, embeddings) print(similarities) ``` rz Add exported z model ) folder_path path_in_reporepo_id repo_typecommit_messagecommit_descriptionr) tempfileTemporaryDirectoryrshutilmove with_suffixas_posixmkdirreprreplacehuggingface_hub upload_foldercopy)r0r1r2rrrrr3 file_namer&dst_dirsource destinationropt_config_string bin_sourcebin_destinationr)r)r*r@8st        !   N:"r@)FFN)rr rrrrrrrrrrrr)rr rBrCrrrrrrrrrr)NNNNFFrV)rr rBrWrrrXrrYrrZrr[rrrrrrrrr)FFNr.)r0rr1rrrrrrrrrr3r) __future__rloggingrrpathlibrtypingrrrrsentence_transformers.utilrr getLogger__name__logger)sentence_transformers.SentenceTransformerr rtr r8r7r r r-rQrrr@r)r)r)r*sT       ] X t