o S"go&@sddlmZddlZddlZddlmZddlZddlZddl m Z ddl m Z ddlmZddlmZddlmZdd lmZGd d d ejZdS) ) annotationsN)Path) load_file) save_file) Tokenizer)nn)PreTrainedTokenizerFast)get_device_namecseZdZ  d4d5fd d Zd6ddZd7ddZd8ddZed9ddZd9ddZ d:d;d!d"Z dd/d0Z e d?d2d3ZZS)@StaticEmbeddingN tokenizer#Tokenizer | PreTrainedTokenizerFastembedding_weights np.ndarray | torch.Tensor | None embedding_dim int | NonereturnNonec stt|tr|j}n t|tstd|dur0t|tjr&t |}t j j |dd|_n|dur>t |||_ntd|jj|_|jj|_||_|j|dd|_dS)a Initializes the StaticEmbedding model given a tokenizer. The model is a simple embedding bag model that takes the mean of trained per-token embeddings to compute text embeddings. Args: tokenizer (Tokenizer | PreTrainedTokenizerFast): The tokenizer to be used. Must be a fast tokenizer from ``transformers`` or ``tokenizers``. embedding_weights (np.ndarray | torch.Tensor | None, optional): Pre-trained embedding weights. Defaults to None. embedding_dim (int | None, optional): Dimension of the embeddings. Required if embedding_weights is not provided. Defaults to None. Example:: from sentence_transformers import SentenceTransformer from sentence_transformers.models import StaticEmbedding from tokenizers import Tokenizer # Pre-distilled embeddings: static_embedding = StaticEmbedding.from_model2vec("minishlab/potion-base-8M") # or distill your own embeddings: static_embedding = StaticEmbedding.from_distillation("BAAI/bge-base-en-v1.5", device="cuda") # or start with randomized embeddings: tokenizer = Tokenizer.from_pretrained("FacebookAI/xlm-roberta-base") static_embedding = StaticEmbedding(tokenizer, embedding_dim=512) model = SentenceTransformer(modules=[static_embedding]) embeddings = model.encode(["What are Pandas?", "The giant panda, also known as the panda bear or simply the panda, is a bear native to south central China."]) similarity = model.similarity(embeddings[0], embeddings[1]) # tensor([[0.8093]]) (If you use potion-base-8M) # tensor([[0.6234]]) (If you use the distillation method) # tensor([[-0.0693]]) (For example, if you use randomized embeddings) Raises: ValueError: If the tokenizer is not a fast tokenizer. ValueError: If neither `embedding_weights` nor `embedding_dim` is provided. zThe tokenizer must be fast (i.e. Rust-backed) to use this class. Use Tokenizer.from_pretrained() from `tokenizers` to load a fast tokenizer.NF)freezez?Either `embedding_weights` or `embedding_dim` must be provided. base_model)super__init__ isinstancer _tokenizerr ValueErrornpndarraytorch from_numpyr EmbeddingBagfrom_pretrained embeddingget_vocab_sizenum_embeddingsrr no_paddinggetr)selfr r rkwargs __class__n/mnt/skqttb/ctump_chatbot/chatbot/lib/python3.10/site-packages/sentence_transformers/models/StaticEmbedding.pyrs& -       zStaticEmbedding.__init__texts list[str]dict[str, torch.Tensor]c Ksj|jj|dd}dd|D}ttdgdd|ddD}tjdd|Dtjd }||d S) NF)add_special_tokenscSsg|]}|jqSr))ids).0encodingr)r)r* _sz,StaticEmbedding.tokenize..rcSsg|]}t|qSr))len)r0 token_idsr)r)r*r2ascSsg|] }|D]}|qqSr)r))r0r4token_idr)r)r*r2bs)dtype) input_idsoffsets)r encode_batchrrrcumsumtensorlong)r%r+r& encodings encodings_idsr9r8r)r)r*tokenize]s ( zStaticEmbedding.tokenizefeaturescKs||d|d|d<|S)Nr8r9sentence_embedding)r )r%rAr&r)r)r*forwardeszStaticEmbedding.forwarddict[str, float]cCsiSNr)r%r)r)r*get_config_dictiszStaticEmbedding.get_config_dictintcCstjSrE)mathinfrFr)r)r*max_seq_lengthlszStaticEmbedding.max_seq_lengthcCs|jSrE)rrFr)r)r* get_sentence_embedding_dimensionpsz0StaticEmbedding.get_sentence_embedding_dimensionTsave_dirstrsafe_serializationboolcKsT|rt|tj|dn t|tj|d|jtt |ddS)Nmodel.safetensorspytorch_model.bintokenizer.json) save_safetensors_file state_dictospathjoinrsaver rNr)r%rMrOr&r)r)r*rYsszStaticEmbedding.saveload_dircKsttt|d}tjtj|dr ttj|d}nt j tj|dt ddd}z|d}Wn t yC|d}Ynwt ||d S) NrSrQrRcpuT) map_location weights_onlyzembedding.weight embeddings)r )r from_filerNrrVrWexistsrXload_safetensors_filerloaddeviceKeyErrorr )rZr&r weightsr)r)r*rbzs    zStaticEmbedding.load model_name vocabularylist[str] | Nonerc str | Nonepca_dims apply_zipf use_subwordc Cszzddlm}Wn tytdwt}|||||||d}t|jtjr/t |j} n|jj } |j } || | |dS)a Creates a StaticEmbedding instance from a distillation process using the `model2vec` package. Args: model_name (str): The name of the model to distill. vocabulary (list[str] | None, optional): A list of vocabulary words to use. Defaults to None. device (str): The device to run the distillation on (e.g., 'cpu', 'cuda'). If not specified, the strongest device is automatically detected. Defaults to None. pca_dims (int | None, optional): The number of dimensions for PCA reduction. Defaults to 256. apply_zipf (bool): Whether to apply Zipf's law during distillation. Defaults to True. use_subword (bool): Whether to use subword tokenization. Defaults to True. Returns: StaticEmbedding: An instance of StaticEmbedding initialized with the distilled model's tokenizer and embedding weights. Raises: ImportError: If the `model2vec` package is not installed. r)distillz\To use this method, please install the `model2vec` package: `pip install model2vec[distill]`)rhrcrkrlrmr r) model2vec.distillrn ImportErrorr rr rrrrweightr ) clsrgrhrcrkrlrmrn static_modelr r r)r)r*from_distillations* z!StaticEmbedding.from_distillationmodel_id_or_pathcCsjzddlm}Wn tytdw||}t|jtjr't |j}n|jj }|j }||||dS)aH Create a StaticEmbedding instance from a model2vec model. This method loads a pre-trained model2vec model and extracts the embedding weights and tokenizer to create a StaticEmbedding instance. Args: model_id_or_path (str): The identifier or path to the pre-trained model2vec model. Returns: StaticEmbedding: An instance of StaticEmbedding initialized with the tokenizer and embedding weights the model2vec model. Raises: ImportError: If the `model2vec` package is not installed. r) StaticModelzSTo use this method, please install the `model2vec` package: `pip install model2vec`ro) model2vecrwrqrrr rrrrrrr )rsrvrwrtr r r)r)r*from_model2vecs  zStaticEmbedding.from_model2vec)NN)r r r rrrrr)r+r,rr-)rAr-rr-)rrD)rrH)T)rMrNrOrPrr)rZrNrr )NNrfTT)rgrNrhrircrjrkrrlrPrmrPrr )rvrNrr )__name__ __module__ __qualname__rr@rCrGpropertyrKrLrYrb classmethodrury __classcell__r)r)r'r*r s* J      5r ) __future__rrIrVpathlibrnumpyrrsafetensors.torchrrarrT tokenizersrr transformersrsentence_transformers.utilr Moduler r)r)r)r*s