o ?"g<@sdZddlmZmZmZddlZddlmZer'ddlm Z m Z ddlm Z Gdd d Z d ejd d fd dZejjdddejdejdejd ejfddZ   ddejjdejdejdejdeejd fdeedeedeejd eejejffddZdS)a7 Partially inspired by torchtune's flex attention implementation Citation: @software{torchtune, title = {torchtune: PyTorch's finetuning library}, author = {torchtune maintainers and contributors}, url = {https//github.com/pytorch/torchtune}, license = {BSD-3-Clause}, month = apr, year = {2024} } )OptionalTupleUnionN)is_torch_flex_attn_available) BlockMaskflex_attention)create_block_maskcsJeZdZdZdZdZdZfddZej j ddddZ d d Z Z S) WrappedFlexAttentionzh We are doing a singleton class so that flex attention is compiled once when it's first called. NFcs|jdur t||_|jSN) _instancesuper__new__)clsargskwargs __class__j/mnt/skqttb/ctump_chatbot/chatbot/lib/python3.10/site-packages/transformers/integrations/flex_attention.pyr7s zWrappedFlexAttention.__new__ recursivecCs(|jdurtjtdd|_d|_dSdS)z> Initialize or update the singleton instance. F)dynamicTN)_is_flex_compiledtorchcompiler_compiled_flex_attentionselfrrr__init__=s  zWrappedFlexAttention.__init__cCs|jSr )rrrrr__call__FszWrappedFlexAttention.__call__)__name__ __module__ __qualname____doc__r rrrrcompilerdisablerr __classcell__rrrrr .s   r attention_mask_2dreturnrcs4|j}|j\}}fdd}t||d|||dS)a Create a block causal document mask for a batch of sequences, both packed and unpacked. Create Block causal logic and passing it into :func:`torch.nn.attention.flex_attention.create_block_mask`. The resultant BlockMask is a compressed representation of the full block causal mask. BlockMask is essential for performant computation of flex attention. See: https://pytorch.org/blog/flexattention/ Args: attention_mask_2d (torch.Tensor): Attention mask for packed and padded sequences of shape (batch_size, total_seq_len). e.g. For unpacked sequence: [[1, 1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 1, 0, 0]] For packed sequence: [[1, 1, 1, 2, 2, 2, 0], [1, 1, 2, 2, 2, 3, 3]] Returns: BlockMask cs<||k}||f||fk}||fdk}||@|@S)z Defines the logic of a block causal mask by combining both a standard causal mask and a block diagonal document mask. See :func:`~torchtune.modules.attention_utils.create_block_causal_mask` for an illustration. rr) batch_idxhead_idxq_idxkv_idx causal_mask document_mask padding_mask document_idsrrcausal_mask_modks z4make_flex_block_causal_mask..causal_mask_modN)mask_modBHQ_LENKV_LENdevice)r9shapecreate_block_causal_mask_flex)r(r9 batch_size total_seq_lenr3rr1rmake_flex_block_causal_maskJs   r>FrquerykeyvaluecKst}||||fi|Sr )r )r?r@rArflex_attention_compiledrrrcompile_friendly_flex_attentionsrCmoduleattention_maskscalingsoftcap head_maskc sd} dt|tr |} n|dur%ddddddd|jdffdd} t|||| | d|dd\} } | |j} | dd} | | fS)Ncs^dur t|}dur||d||}dur-|||dd}|S)Nr)rtanh)scorer*r+r,r-r.rHrGrr score_modsz)flex_attention_forward..score_modT)rM block_mask enable_gqascale return_lser) isinstancerr:rCtodtype transpose contiguous) rDr?r@rArErFrGrHrrNrM attn_outputattention_weightsrrLrflex_attention_forwards*  &  rZ)NNN)r$typingrrrrutilsr!torch.nn.attention.flex_attentionrrr r;r Tensorr>r%r&rCnnModulefloatrZrrrrsR   8