o ?"go @sdddlZddlmZddlmZddlmZmZee Z er$ddl Z   d.deededd ee d e d effd d Z   d.deededd ee d e d effddZ   d.deededd ee d e d effddZ d/deddd ee d e d effddZ d/deddd ee d e d effddZ d/deddd ee d e d effddZeeeeeedZ  d0dedededeedeef ddZd/dedeefd d!Zd/dedeefd"d#Zd/dedeefd$d%Zd/dedeefd&d'Zd/dedeefd(d)Zd/dedeefd*d+ZeeeeeedZd/dedeefd,d-Z dS)1N)Optional)PretrainedConfig)is_torch_availableloggingconfigdevicez torch.deviceseq_lenreturnz torch.Tensorc Ks|durt|dkrtd|d|t|dkr#|d}|d}n!|durD|j}t|dr2|jnd}t|d |j|j}t||}d}d|t j d|d t j d  ||} | |fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrzUnexpected arguments: `**rope_kwargs` and `config` are mutually exclusive in `_compute_default_rope_parameters`, got `rope_kwargs`= and `config`=basedimpartial_rotary_factor?head_dimdtype)len ValueError rope_thetahasattrrgetattr hidden_sizenum_attention_headsinttorcharangeint64floatto) rrr rope_kwargsr r rrattention_factorinv_freqr$b/mnt/skqttb/ctump_chatbot/chatbot/lib/python3.10/site-packages/transformers/modeling_rope_utils.py _compute_default_rope_parameterss&   *r&cKsx|durt|dkrtd|d|t|dkr|d}n |dur(|jd}t|||fi|\}}||}||fS)a Computes the inverse frequencies with linear scaling. Credits to the Reddit user /u/kaiokendev Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrzUnexpected arguments: `**rope_kwargs` and `config` are mutually exclusive in `_compute_linear_scaling_rope_parameters`, got `rope_kwargs`=r factor)rr rope_scalingr&)rrr r!r'r#r"r$r$r%'_compute_linear_scaling_rope_parametersGs   r)c Ks|durt|dkrtd|d|t|dkr+|d}|d}|d}|d}n)|durT|j}t|d r:|jnd }t|d |j|j} t| |}|j }|j d}d } |dur`||kr`|n|}|||||d ||d }d |t j d|d t j d||} | | fS)a4 Computes the inverse frequencies with NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length, used to update the dynamic RoPE at inference time. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrzUnexpected arguments: `**rope_kwargs` and `config` are mutually exclusive in `_compute_dynamic_ntk_parameters`, got `rope_kwargs`=r r r max_position_embeddingsr'rrrrrr)rrrrrrrrrr*r(rrrrr ) rrr r!r r r*r'rrr"r#r$r$r%_compute_dynamic_ntk_parametersps2    $*r+c sHt|dkr td||j}t|dr|jnd}t|d|j|j}t||}|j }|j d} |j d} | durEd t | d} |j d pLd } |j d pTd } ddfdd} dd}|td|d||}d|}d| |}| | | |||\}}d ||||d|}|d |||}|| fS)a Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://arxiv.org/abs/2309.00071) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rzYUnexpected arguments: `**rope_kwargs` should be unset in `_compute_yarn_parameters`, got rrrr'r"Ng? beta_fast beta_slowrcSs*|t||dtjdt|S)zPInverse dimension formula to find the dimension based on the number of rotationsr)mathlogpi) num_rotationsr r r*r$r$r%find_correction_dims*z5_compute_yarn_parameters..find_correction_dimcs@t||||}t||||}t|dt||dfS)z.Find dimension range bounds based on rotationsrr)r/floorceilmaxmin)low_rothigh_rotr r r*lowhighr3r$r%find_correction_rangesz7_compute_yarn_parameters..find_correction_rangecSs>||kr|d7}tj|tjd|||}t|dd}|S)NgMbP?rrr)rrfloat32clamp)r7r6r linear_func ramp_funcr$r$r%linear_ramp_factors z4_compute_yarn_parameters..linear_ramp_factorr)rrrrrrrrrr*r(getr/r0rrrr )rrr r!r rrr r*r'r"r,r.r=rB pos_freqsinv_freq_extrapolationinv_freq_interpolationr:r;inv_freq_extrapolation_factorr#r$r<r%_compute_yarn_parameterss8        rHcKs@t|dkr td||j}t|dr|jnd}t|d|j|j}t||}|j d}|j d} |j d} |j d } t|d rO|j } |j |j } n|j } | d url| dkr]d} nt d t | t | } |r||| kr|tj|tj|d } n tj| tj|d } tjd|dtj|d |}d| ||}|| fS)a  Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rz]Unexpected arguments: `**rope_kwargs` should be unset in `_compute_longrope_parameters`, got rrr long_factor short_factorr'r" original_max_position_embeddingsNr)rrr)rrrrrrrrrr(rCrKr*r/sqrtr0rtensorr>rrr)rrr r!r rrr rIrJr'r"rK ext_factorsinv_freq_shaper#r$r$r%_compute_longrope_parameterss8        rPcKst|||fi|\}}|jd}|jd}|jd}|jd} | |} | |} dtj|} t| | k|||} | | |||}d|| ||| }| | k| | k}t||| } | |fS)a Computes the inverse frequencies for llama 3.1. Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r'low_freq_factorhigh_freq_factorrKrr)r&r(r/r1rwhere)rrr r!r#r"r'rQrRold_context_lenlow_freq_wavelenhigh_freq_wavelenwaveleninv_freq_llama smooth_factorsmoothed_inv_freqis_medium_freqr$r$r%_compute_llama3_parameters1s    r\)defaultlineardynamicyarnlongropellama3 rope_type received_keys required_keys optional_keys ignore_keyscCsd|vr|dh8}|d|dur||8}||}|r&td|d||dur1|||}n||}|rDtd|d|dSdS)zYCompare the received keys in `config.rope_scaling` against the expected and optional keystypercNz9Missing required keys in `rope_scaling` for 'rope_type'='z': z5Unrecognized keys in `rope_scaling` for 'rope_type'=')addKeyErrorloggerwarning)rcrdrerfrg missing_keys unused_keysr$r$r%_check_received_keysis  rocCs@|j}|d|dd}dh}t|}t||||ddS)Nrcrhrg)r(rCsetkeysro)rrgr(rcrerdr$r$r%!_validate_default_rope_parameterss  rscCsx|j}|d|dd}ddh}t|}t||||d|d}|dus0t|tr0|dkr:td|dSdS)Nrcrhr'rpr8`rope_scaling`'s factor field must be a float >= 1, got r(rCrqrrro isinstancerrkrl)rrgr(rcrerdr'r$r$r%(_validate_linear_scaling_rope_parameterss rwcCs|j}|d|dd}ddh}dh}t|}t|||||d|d}|dus4t|tr4|dkr>td|dSdS)Nrcrhr'rKrprrtru)rrgr(rcrerfrdr'r$r$r%)_validate_dynamic_scaling_rope_parameterss rxc Cs6|j}|d|dd}ddh}hd}t|}t|||||d|d}|dus5t|tr5|dkr=td||d}|durWt|trO|d krWtd ||d } | durmt| tsmtd | |d } | durt| tstd| | pd| pdkrtd| d| ddSdS)Nrcrhr'>r,r.r"rprrtr"rL`rope_scaling`'s attention_factor field must be a float greater than 0, got r,z6`rope_scaling`'s beta_fast field must be a float, got r.z6`rope_scaling`'s beta_slow field must be a float, got r-rzO`rope_scaling`'s beta_fast field must be greater than beta_slow, got beta_fast=z( (defaults to 32 if None) and beta_slow=z (defaults to 1 if None)ru) rrgr(rcrerfrdr'r"r,r.r$r$r%_validate_yarn_parameterss6    rzcCs|j}|d|dd}hd}hd}t|}t|||||dt|dr,|jnd}t|d|j|j }t ||} |d } t | t sYt d d | DrYtd | t| | d ksptd| d dt| |d} t | t st dd | Drtd| t| | d kstd| d dt| t|drtddS|d} | durtdnt | tr| dkrtd| |d} | durt | tr| dkrtd| dSdSdS)Nrcrh>rcrIrJ>r'r"rKrprrrrJcs|] }t|ttfVqdSNrvrr.0xr$r$r% z0_validate_longrope_parameters..zC`rope_scaling`'s short_factor field must be a list of numbers, got rz5`rope_scaling`'s short_factor field must have length z, got rIcsr{r|r}r~r$r$r%rrzB`rope_scaling`'s long_factor field must be a list of numbers, got z4`rope_scaling`'s long_factor field must have length rKaYThis model has set a `original_max_position_embeddings` field, to be used together with `max_position_embeddings` to determine a scaling factor. Please set the `factor` field of `rope_scaling`with this ratio instead -- we recommend the use of this field over `original_max_position_embeddings`, as it is compatible with most model architectures.r'z1Missing required keys in `rope_scaling`: 'factor'rtr"gry)r(rCrqrrrorrrrrrrvlistallrkrlr warning_oncer)rrgr(rcrerfrdrrr rJrIr'r"r$r$r%_validate_longrope_parameterssH        rc Cs6|j}|d|dd}hd}t|}t||||d|d}|dus0t|tr0|dkr8td||d}|d }|dusIt|tsQtd ||dusZt|tsbtd |||krqtd |d ||d} | dus~t| t std| | |j krtd| d|j dSdS)Nrcrh>r'rcrQrRrKrpr'rrtrQrRz<`rope_scaling`'s low_freq_factor field must be a float, got z=`rope_scaling`'s high_freq_factor field must be a float, got zc`rope_scaling`'s high_freq_factor field must be greater than low_freq_factor, got high_freq_factor=z and low_freq_factor=rKzP`rope_scaling`'s original_max_position_embeddings field must be an integer, got zg`rope_scaling`'s original_max_position_embeddings field must be less than max_position_embeddings, got z and max_position_embeddings=) r(rCrqrrrorvrrkrlrr*) rrgr(rcrerdr'rQrRrKr$r$r%_validate_llama3_parameterssL  rcCsdt|dd}|dur dS|d|dd}t|}|dur'|||ddStd|ddS) zO Validate the RoPE config arguments, given a `PretrainedConfig` object r(Nrcrhr]rpzTMissing validation function mapping in `ROPE_VALIDATION_FUNCTIONS` for 'rope_type'='')rrCROPE_VALIDATION_FUNCTIONSrkrl)rrgr(rc validation_fnr$r$r%rope_config_validation(s   r)NNNr|)NN)!r/typingrconfiguration_utilsrutilsrr get_logger__name__rkrrtuplerr&r)r+rHrPr\ROPE_INIT_FUNCTIONSstrrqrorsrwrxrzrrrrr$r$r$r%s     +  *  4  P  @  /  2&