o iî€ã@súddlZddlmZddlmZddlmZddlmZm Z m Z ddl Z ddl Z ddl mZddlmZmZddlmZdd lmZd d lmZmZmZd d lmZd d lmZmZgd¢ZeGdd„dƒƒZ dee!de!fdd„Z"de j#de!de!de$e j#e!ffdd„Z%de j#de!de!de!de j#f dd„Z&e j' (d¡e j' (d¡Gdd„dej)ƒZ*de j#d e!de j#fd!d"„Z+d#e j#d$e j#d%e$e!e!e!fd&e$e!e!e!fd'e j#d(e j#d)e j#de j#fd*d+„Z,de j#d,e j#d-e-fd.d/„Z.e j' (d+¡e j' (d/¡Gd0d1„d1ej)ƒZ/Gd2d3„d3ej)ƒZ0Gd4d5„d5ej)ƒZ1Gd6d7„d7ej)ƒZ2d8e3e d9e4d:e ed;e-d„Z5Gd?d@„d@eƒZ6GdAdB„dBeƒZ7eƒedCe6j8fdDddEdFœd:e e6d;e-dr?r@Úlayers©Ú __class__r'r(rBBs    z Pool.__init__r0Úthwc Csöt|ddƒ\}}tj|ddd\}}| dd¡}|jdd…\}}}| |||f|¡ ¡}|jr<|jdur<| |¡}|  |¡}|jdd…\}} } | |||d¡ dd¡}tj ||fdd}|jsm|jdurm| |¡}t |dd|ƒ}||| | ffS) Nér,©r,r )Úindicesr3réÿÿÿÿ©r3) r9ÚtorchÚ tensor_splitÚ transposer6ÚreshapeÚ contiguousr@rFr=Úcatr;) rGr0rKr8Ú class_tokenÚBÚNÚCÚTÚHÚWr'r'r(ÚforwardSs    z Pool.forward)NF)r!r"r#rDÚModulerÚboolrBrQÚTensorÚtupler$r^Ú __classcell__r'r'rIr(r<Asûþýüûú>r<Ú embeddingÚdcCs@|jd|kr |Stjj| dd¡ d¡|dd d¡ dd¡S)Nrr,Úlinear)ÚsizeÚmode)r6rDÚ functionalÚ interpolateÚpermuter4r:)rdrer'r'r(Ú _interpolatemsýùrlÚattnÚqÚq_thwÚk_thwÚ rel_pos_hÚ rel_pos_wÚ rel_pos_tc% Cs6|\}}} |\} } } tdt|| ƒdƒ} tdt| | ƒdƒ}tdt|| ƒdƒ}t| |dƒ}t|| dƒ}t |¡dd…df|t | ¡ddd…fd| |}t| | dƒ}t| | dƒ}t | ¡dd…df|t | ¡ddd…fd| |}t| |dƒ}t|| dƒ}t |¡dd…df|t | ¡ddd…fd| |}t|| ƒ}t||ƒ}t||ƒ}|| ¡}|| ¡}|| ¡}|j\}}}}|dd…dd…dd…f ||||| |¡} t d| |¡}!t d| |¡}"|   dddddd ¡ ||||| |¡} t  | |  dd¡¡  dd¡}#|#  |||| || ¡  dddddd ¡}#|!dd…dd…dd…dd…dd…ddd…df|"dd…dd…dd…dd…dd…dddd…f|#dd…dd…dd…dd…dd…dd…ddf ||||| | | | ¡}$|dd…dd…dd…dd…f|$7<|S) Nr r,çð?zbythwc,hkc->bythwkzbythwc,wkc->bythwkrrrLé) r$ÚmaxrQÚarangerlÚlongr6rTÚeinsumrkÚmatmulrSÚview)%rmrnrorprqrrrsÚq_tÚq_hÚq_wÚk_tÚk_hÚk_wÚdhÚdwÚdtÚ q_h_ratioÚ k_h_ratioÚdist_hÚ q_w_ratioÚ k_w_ratioÚdist_wÚ q_t_ratioÚ k_t_ratioÚdist_tÚRhÚRwÚRtrXÚn_headÚ_r3Úr_qÚrel_h_qÚrel_w_qÚrel_q_tÚrel_posr'r'r(Ú _add_rel_pos|sH <<<      **$..ÿ.þü(r˜ÚshortcutÚresidual_with_cls_embedc CsZ|r | |¡|S|dd…dd…dd…dd…f|dd…dd…dd…dd…f7<|Sr+)Úadd_)r0r™ršr'r'r(Ú _add_shortcut¸s  Dÿrœcs²eZdZdejfdeededededeedeedeed eed ed ed ed ede dej fddf‡fdd„ Z de j deeeefdee j eeeefffdd„Z‡ZS)ÚMultiscaleAttentionçÚ input_sizeÚ embed_dimÚ output_dimrrrrr Ú residual_poolršÚ rel_pos_embedÚdropoutÚ norm_layer.r*Nc sptƒ ¡||_||_||_|||_dt |j¡|_| |_ | |_ t   |d|¡|_ t   ||¡g}| dkr@| t j| dd¡t j|Ž|_d|_t|ƒdksUt|ƒdkrrdd„|Dƒ}tt j|j|j||||jd d | |jƒƒ|_d|_d|_t|ƒdks„t|ƒdkr·d d„|Dƒ}tt j|j|j||||jd d | |jƒƒ|_tt j|j|j||||jd d | |jƒƒ|_d|_d|_d|_| r6t|dd…ƒ}t|ƒd kr×||dn|}t|ƒd krå||dn|}d t||ƒd}d |d d}t  t ||j¡¡|_t  t ||j¡¡|_t  t ||j¡¡|_t j j!|jddt j j!|jddt j j!|jdddSdS)NrtrržT©Úinplacer,cSóg|]}t|dƒ‘qS©r ©r$)Ú.0rnr'r'r(Ú æóz0MultiscaleAttention.__init__..F)ÚstrideÚpaddingÚgroupsÚbiascSr¨r©rª)r«Úkvr'r'r(r¬÷r­rr ç{®Gáz”?©Ústd)"rArBr r¡rÚhead_dimÚmathÚsqrtÚscalerr¢ršrDÚLinearÚqkvrCÚDropoutrEÚprojectÚpool_qr/r<ÚConv3dÚpool_kÚpool_vrqrrrsrvÚlenÚ ParameterrQÚzerosÚinitÚ trunc_normal_)rGrŸr r¡rrrrr r¢ršr£r¤r¥rHÚ padding_qÚ padding_kvrgÚq_sizeÚkv_sizeÚ spatial_dimÚ temporal_dimrIr'r(rBÅs   ù ö ù ö ù ö õzMultiscaleAttention.__init__r0rKc Cs:|j\}}}| |¡ ||d|j|j¡ dd¡jdd\}}}|jdur-| ||¡\}} n|} |jdur<| ||¡d}|j durI|  ||¡\}}t   |j || dd¡¡} |j durr|jdurr|jdurrt| ||| |j |j|jƒ} | jdd} t   | |¡}|jrˆt|||jƒ| dd¡ |d|j¡}| |¡}||fS)Nrr,r rPrrO)r6r»rTrr¶rSÚunbindrÀrÁr¾rQrzr¹rqrrrsr˜Úsoftmaxr¢rœršr¡r½) rGr0rKrXrYrZrnÚkr.rprmr'r'r(r^!s6 2   ù  zMultiscaleAttention.forward)r!r"r#rDÚ LayerNormr&r$r`Úfloatrr_rBrQrarbr^rcr'r'rIr(rÄsBòþýüûúùø ÷ ö õ ô ó òñ>\rcs”eZdZddejfdeedededededede d e d e d ej fd d f‡fdd„ Z de jdeeeefd ee jeeeefffdd„Z‡ZS)ÚMultiscaleBlockržrŸÚcnfr¢ršr£Úproj_after_attnr¤Ústochastic_depth_probr¥.r*Nc stƒ ¡||_d|_t|jƒdkr.dd„|jDƒ} dd„| Dƒ} ttj| |j| ddƒ|_|r3|j n|j } | |j ƒ|_ | | ƒ|_ t |j tjƒ|_t||j | |j|j|j|j|j||||| d |_t| d| |j gtj|dd|_t|d ƒ|_d|_|j |j kr‹t |j |j ¡|_dSdS) Nr,cSs g|] }|dkr |dn|‘qSrMr')r«r)r'r'r(r¬Vs z,MultiscaleBlock.__init__..cSr¨r©rª)r«rÏr'r'r(r¬Wr­)r®r¯) rrrr r£r¢ršr¤r¥rL)Úactivation_layerr¤r§Úrow)rArBrÔÚ pool_skipr/rr<rDÚ MaxPool3drrÚnorm1Únorm2Ú isinstanceÚ BatchNorm1dÚneeds_transposalrrrrr rmr ÚGELUÚmlpr Ústochastic_depthr½rº) rGrŸrÓr¢ršr£rÔr¤rÕr¥Ú kernel_skipÚ padding_skipÚattn_dimrIr'r(rBEsP ÿ  ó û  ÿzMultiscaleBlock.__init__r0rKc Csä|jr| | dd¡¡ dd¡n| |¡}| ||¡\}}|jdus%|js'|n| |¡}|jdur3|n| ||¡d}|| |¡}|jrR| | dd¡¡ dd¡n| |¡}|jdus_|jra|n| |¡}|| |  |¡¡|fS)Nr,r r) rÞrÚrSrmr½rÔrØrárÛrà) rGr0rKÚx_norm1Úx_attnÚthw_newÚx_skipÚx_norm2Úx_projr'r'r(r^s**zMultiscaleBlock.forward)r!r"r#rDrÐr&r$rr`rÑrr_rBrQrarbr^rcr'r'rIr(rÒDs4 öþýüûúùø ÷ ö õ>:rÒc sPeZdZdedeeefdededdf ‡fdd„ Zd ejdejfd d „Z ‡Z S) ÚPositionalEncodingÚ embed_sizeÚ spatial_sizeÚ temporal_sizer£r*Ncs’tƒ ¡||_||_t t |¡¡|_d|_ d|_ d|_ |sGt t |jd|jd|¡¡|_ t t |j|¡¡|_ t t |¡¡|_ dSdS)Nrr,) rArBrírîrDrÃrQrÄrWÚ spatial_posÚ temporal_posÚ class_pos)rGrìrírîr£rIr'r(rBs $ýzPositionalEncoding.__init__r0cCs¼|j | d¡d¡ d¡}tj||fdd}|jdur\|jdur\|jdur\|jj \}}tj |j|dd}|  |j d¡ |j dd¡  d|¡¡tj|j d¡|fdd d¡}|  |¡|S)NrrOr,rP)rWÚexpandrgr4rQrVrïrðrñr6Úrepeat_interleaver›rîrT)rGr0rWÚhw_sizerìÚ pos_embeddingr'r'r(r^›s &  zPositionalEncoding.forward) r!r"r#r$rbr`rBrQrar^rcr'r'rIr(rëŒs*rëc$sÒeZdZ         ddeeefd ed eed ed ed edededededede e de j fde e de j fdeeeefdeeeefdeeeefddf"‡fdd„ Z dejdejfdd„Z‡ZS) rçà?ržéN©rérù©r rLrL©r,rrrírîÚ block_settingr¢ršr£rÔr¤Úattention_dropoutrÕÚ num_classesÚblock.r¥Úpatch_embed_kernelÚpatch_embed_strideÚpatch_embed_paddingr*cstƒ ¡t|ƒt|ƒ}|dkrtdƒ‚| durt} | dur&ttjdd} tj d|dj |||d|_ dd „t |f||j j ƒDƒ}t|dj |d |d f|d|d |_t ¡|_t|ƒD]/\}}| ||d }|j | ||||||| || d ¡t|jƒdkrdd „t ||jƒDƒ}q`| |djƒ|_t tj|ddt |dj| ¡¡|_| ¡D][}t|tjƒrÔtjj|jddt|tjƒrÓ|j durÓtj !|j d¡q¯t|tjƒrõ|jdurçtj !|jd ¡|j durôtj !|j d¡q¯t|tƒr | "¡D] }tjj|ddqÿq¯dS)aÄ MViT main class. Args: spatial_size (tuple of ints): The spacial size of the input as ``(H, W)``. temporal_size (int): The temporal size ``T`` of the input. block_setting (sequence of MSBlockConfig): The Network structure. residual_pool (bool): If True, use MViTv2 pooling residual connection. residual_with_cls_embed (bool): If True, the addition on the residual connection will include the class embedding. rel_pos_embed (bool): If True, use MViTv2's relative positional embeddings. proj_after_attn (bool): If True, apply the projection after the attention. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob: (float): Stochastic depth rate. Default: 0.0. num_classes (int): The number of classes. block (callable, optional): Module specifying the layer which consists of the attention and mlp. norm_layer (callable, optional): Module specifying the normalization layer to use. patch_embed_kernel (tuple of ints): The kernel of the convolution that patchifies the input. patch_embed_stride (tuple of ints): The stride of the convolution that patchifies the input. patch_embed_padding (tuple of ints): The padding of the convolution that patchifies the input. rz+The configuration parameter can't be empty.Ngíµ ÷Æ°>)Úepsr)Ú in_channelsÚ out_channelsÚ kernel_sizer®r¯cSóg|]\}}||‘qSr'r'©r«rgr®r'r'r(r¬ìr­z!MViT.__init__..r,r )rìrírîr£rt) rŸrÓr¢ršr£rÔr¤rÕr¥cSrr'r'rr'r'r(r¬ r­rOTr¦r³r´rž)#rArBr rÂr5rÒrrDrÐr¿rÚ conv_projÚzipr®rëÚ pos_encodingÚ ModuleListÚblocksÚ enumeraterCrrr>rEr¼rºÚheadÚmodulesrÜrÅrÆÚweightr±Ú constant_Ú parameters)rGrírîrür¢ršr£rÔr¤rýrÕrþrÿr¥rrrÚtotal_stage_blocksrŸÚstage_block_idrÓÚsd_probÚmÚweightsrIr'r(rBªs€ )ûü ÷ÿ€ þ  €   €  €ôz MViT.__init__r0cCsŠt|ddƒd}| |¡}| d¡ dd¡}| |¡}|jjf|jj}|jD] }|||ƒ\}}q'| |¡}|dd…df}|  |¡}|S)Nrur rr,) r9r ÚflattenrSr rîrír r>r)rGr0rKrÿr'r'r(r^"s     z MViT.forward) röržržr÷NNrørúrû)r!r"r#rbr$rrr`rÑrrrDr_rBrQrar^rcr'r'rIr(r©s\ ï þýüûúùø ÷ ö õ ô óò ñ ð ïîxrrürÕrÚprogressÚkwargsc KsÚ|dur1t|dt|jdƒƒ|jdd|jddksJ‚t|d|jdƒt|d|jdƒ| dd ¡}| dd ¡}td|||| d d ¡| d d¡| dd ¡| dd ¡|dœ|¤Ž}|durk| |j|dd¡|S)NrþÚ categoriesÚmin_sizerr,rírîÚmin_temporal_size©éàr ér¢FršTr£rÔ)rírîrür¢ršr£rÔrÕ)rÚ check_hashr')rrÂÚmetaÚpoprÚload_state_dictÚget_state_dict)rürÕrrrrírîÚmodelr'r'r(Ú_mvit9s,       ø ÷ r(c @óJeZdZedeedddddddedd d d d d dœidddœ dZeZdS)rz:https://download.pytorch.org/models/mvit_v1_b-dbeb1030.pthr©é©çÍÌÌÌÌÌÜ?r-r-©çÍÌÌÌÌÌÌ?r/r/©Ú crop_sizeÚ resize_sizeÚmeanrµr!zShttps://github.com/facebookresearch/pytorchvideo/blob/main/docs/source/model_zoo.mdúœThe weights were ported from the paper. The accuracies are estimated on video-level with parameters `frame_rate=7.5`, `clips_per_video=5`, and `clip_len=16`ip¢.ú Kinetics-400gJ +‡žS@gh‘í|?eW@©zacc@1zacc@5gu“V¦Q@gœÄ °rxa@© rrrÚrecipeÚ_docsÚ num_paramsÚ_metricsÚ_opsÚ _file_size©ÚurlÚ transformsr#N© r!r"r#rrr rÚKINETICS400_V1ÚDEFAULTr'r'r'r(rZó4ûþÿï÷rc @r))rz:https://download.pytorch.org/models/mvit_v2_s-ae3be167.pthrr*r,r.r0r!zChttps://github.com/facebookresearch/SlowFast/blob/main/MODEL_ZOO.mdr4ir5gœÄ °r0T@gÃõ(\ªW@r6gu“VP@g?5^ºI|`@r7r>NrAr'r'r'r(r{rDrÚ pretrained)rT)rrcKsØt |¡}gd¢gd¢gd¢ggd¢ggd¢gggggggggggd¢gggd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gggd¢ggd¢gggggggggggd¢gggd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd œ}g}tt|d ƒƒD],}| t|d ||d ||d ||d ||d||d||d|d ¡q«tddd|dd| dd¡||dœ|¤ŽS)a¿ Constructs a base MViTV1 architecture from `Multiscale Vision Transformers `__. .. betastatus:: video module Args: weights (:class:`~torchvision.models.video.MViT_V1_B_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.video.MViT_V1_B_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.video.MViT`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.video.MViT_V1_B_Weights :members: ©r,r r rLrLrLrLrLrLrLrLrLrLrLérG©é`éÀrJé€rKrKrKrKrKrKrKrKrKrKérL)rJrJrKrKrKrKrKrKrKrKrKrKrKrLrLrL©rrr©r,r r ©r,rGrG©r,rLrL©r,r,r,©rrrrrrr rrrrrrr rr!FrÕçš™™™™™É?)rírîrür¢ršrÕrrNr')rÚverifyÚrangerÂrCrr(r$©rrrÚconfigrüÚir'r'r(rœs„ .ð.ðè,       ùÿ  ø ÷rcKsDt |¡}gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢ggd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢ggd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢ggd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd¢gd œ}g}tt|d ƒƒD],}| t|d ||d ||d ||d ||d||d||d|d ¡qßtddd|dddd| dd¡||dœ |¤ŽS)aCConstructs a small MViTV2 architecture from `Multiscale Vision Transformers `__ and `MViTv2: Improved Multiscale Vision Transformers for Classification and Detection `__. .. betastatus:: video module Args: weights (:class:`~torchvision.models.video.MViT_V2_S_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.video.MViT_V2_S_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.video.MViT`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.video.MViT_V2_S_Weights :members: rF)rIrIrJrJrKrKrKrKrKrKrKrKrKrKrKrLrHrMrQrNrOrPrRrrrrrrr rr!TFrÕrS) rírîrür¢ršr£rÔrÕrrNr')rrTrUrÂrCrr(r$rVr'r'r(rþsÈ ððððÆN       ùÿ  ö õr);r·Úcollections.abcrÚ dataclassesrÚ functoolsrÚtypingrrrrQÚtorch.fxÚtorch.nnrDÚopsr r Útransforms._presetsr Úutilsr Ú_apirrrÚ_metarÚ_utilsrrÚ__all__rr$r/rarbr9r;ÚfxÚwrapr_r<rlr˜r`rœrrÒrërr&rÑr(rrrBrrr'r'r'r(ÚsŽ         &"  ,ÿþ ý üûúù ø<  Hÿþýüû ú!!!*`.