U Ukhx@sddlZddlZddlmZmZddlmZddlmZddlmZddl m Z ddl Z ddl Z ddlZ ddlZ ddlmZddl mZmZmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5ddl6m7Z7ddl8m9Z9m:Z:ddlmZm;Z;mdS)N) defaultdict namedtuple)copy)List) OrderedDict) timedelta)IntegrityError) ForeignKeyfuncinspect)jsong)Config)command)UNIQUE_VIOLATION)errors)close_all_sessions) CmfRelBaseCmfType)fields)models) CmfOrmErrorCmfOrmUniqueErrorCmfOrmIntegrityError) BaseModelMetaDEFAULT_DATASOURCE BaseModel CmfGM2MModel)BaseDataDriver BaseMapperMapperinc_select_count)imutable_deep_copy)cache_obj_lock_release_allemit_delayed_events)rDictAnyc seZdZdZiZeZejj j eedZ e Z dZddZfddZedd Zed d Zed d ZedddZdddZddZddZddZddZddZeddZddZd d!Zd"d#Zd$d%Z d&d'Z!e"d(d)Z#e"d*d+Z$e%d,d-Z&dd/d0Z'e"de(e&d2d3d4Z)dd5d6Z*e"d7d8Z+ej,dd1d1fd9d:Z-e"d;d<Z.e"d=d>Z/d?d@Z0e"ddAdBZ1de&dCdDdEZ2d1dd1dFdGdHZ3dddd1dd1d.ddd1dI dJdKZ4dddLdMdNZ5ddd1dd1d1dOdPdQZ6ddRdSZ7dTdUZ8dVdWZ9d1dXdYdZZ:d[d\Z;dd]d^ZdcdddedfZ?ddgdhdiZ@ddgdjdkZAfdldmZBdndoZCdpdqZDdrdsZEdtduZFddvdwZGefdxdyZHdzd{ZId|d}ZJd~dZKZLS)SQLAlchemyDataDriveru dataproviders['default'] = { 'type': 'sqlalchemy', 'sqlachemy.url': 'postgresql://user:password@host/db', 'sqlachemy.echo': True, # Достаточно для отладки запросов в develop mode } # Для отладки в продакшене логгеры: sqlalchemy.engine - controls SQL echoing. set to logging.INFO for SQL query output, logging. DEBUG for query + result set output. These settings are equivalent to echo=True and echo="debug" on create_engine.echo, respectively. sqlalchemy.pool - controls connection pool logging. set to logging.INFO to log connection invalidation and recycle events; set to logging.DEBUG to additionally log all pool checkins and checkouts. These settings are equivalent to pool_echo=True and pool_echo="debug" on create_engine.echo_pool, respectively. sqlalchemy.dialects - controls custom logging for SQL dialects, to the extend that logging is used within specific dialects, which is generally minimal. sqlalchemy.orm - controls logging of various ORM functions to the extent that logging is used within the ORM, which is generally minimal. Set to logging.INFO to log some top-level information on mapper configurations. Example: logging.basicConfig() logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO) )Zclass_registryZmetadataNcCstjj|jdS)N)Zbind) sqlalchemyormZ sessionmakerengineselfr/"./cmf/data_providers/sqlalchemy.py_session_factoryFsz%SQLAlchemyDataDriver._session_factorycsLtj||tj|jfdd|D|_tj| |_ | dS)NcSs*i|]"\}}|dr|ddd|qS)z sqlalchemy.r) startswithreplace.0kvr/r/r0 js z1SQLAlchemyDataDriver.__init__..) super__init__r*Zengine_from_configconfigitemsr,r+Zscoped_sessionr1Session init_metar.argskwargs __class__r/r0r;eszSQLAlchemyDataDriver.__init__c Cstg}|jD] \}}t|tr$q|js2|js2qd|g}g}|jrNt|j}|jr|r|jrd|krtd|d|j d|D]}|dkr| t j |d|jiq|dkr| t j d|j d|d||d |d id q| t j d|j d|d||||jd qq| t j |d|jiqt|d rp|jrp| t j d|j df|jdddi|S)Ndefaultu В поле .uU указана уникальность, но не указан индекс defaultuniqueZgin_trgmZix__ZginZ gin_trgm_ops)postgresql_usingZpostgresql_ops)rIrGorderno_partition_byZ_orderno_partition_by_clusterordernoF)rK)rr= issubclassrindexrG index_usinglistr class_nameappendr*ZIndex tablenamehasattrrJ)cls cmf_modelZindexescmf_field_name cmf_fieldrArNZidx_typer/r/r0Z __get_indexesssV      z"SQLAlchemyDataDriver.__get_indexescCs |j|}|r|j||SN)cached_queriesget move_to_end)rTkeyretr/r/r0 get_querys  zSQLAlchemyDataDriver.get_querycCs||j|<|jsZtdtddd}|d}|d}|dkrHd}|dkrTd}||_t|j|jkr|jjdd\}}td |dS) N SC_PAGE_SIZE SC_PHYS_PAGESiiF)lastzDEV: set_query evicted key )rYcached_queries_limitossysconflenpopitemprint)rTqueryr\Zsys_memsize_mbZ chunk_countZ queries_limitZ evicted_keyrHr/r/r0 set_querys zSQLAlchemyDataDriver.set_querycCs4|j}|jrdSt|tjr(t}nt|tjr>tj}nt|tj r^t |j |j }nt|tj r|t|jg}|jr|t|jtj||j|jdSt|tjrt}n\t|tjrt|j}n@t|tjrt|j}n$t|tjrdSt|tjrdSt|tjr,dSt|tjr>dSt|tjrPdSt|tj rltj!dd}nt|tj"rt#}nt|tj$rt}n~t|tj%rdSt|tj&rdSt|tj'rtj(}nDt|tj)rtj*}n.t|tj+r ddl,m-}|}nt.d|tj|||j/|j|jdS) u0Создадим SA поле для моделиN) primary_keynullableT)timezoner)TSVECTORu"Не найден тип поля )rErmrl)0rPvirtualrLrZ CmfBigIntr*ZBIGINTCmfIntIntegerZ CmfNumericZNumericZ precisionZscaleCmfTUUIDStringZ max_lengthZ foreign_keyrQr ZColumnrlrmZCmfTextZTEXTZCmfStrZCmfBytesZ LargeBinary CmfRelationCmfGenericRelation CmfGenericM2M CmfDateRangeCmfM2M CmfDateTime TIMESTAMPZCmfDateZDATEZCmfJson CmfBackrefCmfGenericBackrefCmfBoolZBooleanZCmfTimeZTimeZ CmfTsVectorZsqlalchemy.dialects.postgresqlro ExceptionrE)rTZ _cmf_modelrWZ_db_field_name column_nameZsa_typerAror/r/r0_make_sa_columnsv             z$SQLAlchemyDataDriver._make_sa_columncCst|jj||d}|S)N)schema)r r,Z get_columns)r. table_namercolumnsr/r/r0inspect_table_columnssz*SQLAlchemyDataDriver.inspect_table_columnscCs||}|jj|j|jSrX)dp_modelr,Zdialect has_table __table__)r.rUsa_modelr/r/r0rs zSQLAlchemyDataDriver.has_tablecCs|d|j}t|tjr$|d}td|}d|d|d|}|jsv|jdkrdd||d|jd}td | |td | td dS) NZTZzadd_custom_column open sessionz alter table z add column IF NOT EXISTS  uHЗначение по умолчанию не может быть None: z default z NOT NULLzadd_custom_column run executezadd_custom_column run commitzadd_custom_column done) rtyperLrrzrir>rmrEexecutecommit)r.rrZcmf_field_typeZsa_column_typessqlr/r/r0add_custom_columns     z&SQLAlchemyDataDriver.add_custom_columnc Cstdddd}tjtjdddd|d<tjtjdddd|d <tjtjd dd |d <tjtjd dd |d<t|tf|}ddl}t| dd| |rt d|ddSddl}t| dd| |rt d|ddSt j |jj}|}z.t d||t d|t dWnBtk rn}z"|jt d|d|W5d}~XYnXdS)u5 Создаем М2М таблицу Tcustom.r2abstract __module__ __qualname__uID Объекта)captionrmrMleft_iduID Элементаright_idИмя объекта)rrmZleft_name_cacheuИмя элементаZright_name_cacherN皙?g?u$add_custom_m2m_model Таблица uT уже создана в БД. Вероятно, на другом инстансеz!add_custom_m2m_model create tablezadd_custom_m2m_model run commitzadd_custom_m2m_model doneuBadd_custom_m2m_model Ошбика создания таблицы : )rFieldrs CmfStr256rrrandomtimesleepZuniformrrir*r CreateTablerrr>rrr transactionrollback)r. model_namerBrUrsmtprer/r/r0add_custom_m2m_model.s6      z)SQLAlchemyDataDriver.add_custom_m2m_modelc Csdddd}tjtjddddddd|d <tjtjd dd |d <tjtjd ddddd|d<tjtjdddddd|d<tjtjddddd|d<tjtjdddddd|d<t|tf|}|S)NTrr2ru)Идентификатор объектаu3Автоматически генерируетсяF)rcommentrmrlreadonlyvisibleidr)rrMnameuСортировкаr)rrMrrmrErKuКодu3Код в реальном мире из жизни)rrGrrrmcodeu:Код родителя в каскадном выборе)rrrmrMZchoice_parent_idu СкрытьZ cmf_hidden) rrrsrrqZCmfStr64r~rr)r.rrBrUr/r/r0gen_custom_choice_modelRs`       z,SQLAlchemyDataDriver.gen_custom_choice_modelc Cs~||}tj|jj}|}z|||Wn@t k rx}z"|j t d|d|W5d}~XYnXdS)ur Создаем пользовательский справочник выбора таблицу u-Ошбика создания таблицы rN) rr*rrrrr>rrrrrri)r.rrUrrrr/r/r0add_custom_choice_models    z,SQLAlchemyDataDriver.add_custom_choice_modelc Cs|j}|j|}|s|j}g}g}ddi}|jD].\}} |} ||| | } | dkrZq6|| q6|| ||||t |d} | dd|Dt ||j f| }||j|<|S)u] Вернём SA модель для CMF модели. TODO: db_name Zextend_existingTN)Z __tablename____table_args__cSsi|] }|j|qSr/rr6cr/r/r0r9sz5SQLAlchemyDataDriver.dp_model_cls..)__name__models_registryrZrRrr=rrQextend"_SQLAlchemyDataDriver__get_indexestupleupdater sa_base_model) rTrUrrrRZ sa_columnsrZ__table_kwargs__rVrWZ sa_field_nameZ sa_columnZsa_model_kwargsr/r/r0 dp_model_clss0   z!SQLAlchemyDataDriver.dp_model_clscCs ||SrX)r)r.rUr/r/r0rszSQLAlchemyDataDriver.dp_modelcCs tt|jS)uG Вернём CMF модель для SA модели. )getattrrr)r.rr/r/r0rUszSQLAlchemyDataDriver.cmf_modelcCs t|tjSrX) isinstancer)r)r.Z dp_instancer/r/r0 is_instancesz SQLAlchemyDataDriver.is_instancecCs0i}|D]}|jj||jjj<qtj|d|S)N table_type)rrfullnamer*r+polymorphic_union)r.Z cmf_modelsaliasZtablesmodelr/r/r0 _union_modelssz"SQLAlchemyDataDriver._union_modelsc sBdfdd } i} |D]} | jj| | jjj<q| | d|S)Np_unionTc stj}ii|D]t}||ttjjs0||<i}jD]6}r^|jkr^qJ| |j|||j<|j |j<qJ|<qfddg}| D]\} |dk r$| tjj fdd|Dtjtjj| |ggdtjdjjkq| tjj fdd|Dgdtjdjjkqtjj||S) NTcspz||WStk rjrDtjtj||YStjtj||YSYnXdSrX)KeyErrorr*rcastZnulllabelZ type_coerce)rtable) cast_nulls colnamemapstypesr/r0cols "zPSQLAlchemyDataDriver._union_models2..hack_polymorphic_union..colcsg|]}|qSr/r/r6rrrr/r0 szWSQLAlchemyDataDriver._union_models2..hack_polymorphic_union..Zfrom_obj_idcsg|]}|qSr/r/rrr/r0rs)r*utilZ OrderedSetrrZSelectrrr\addrr=rQselectZliteral_columnZ_quote_ddl_exprrwhererrZcorrelate_exceptZ union_all) Z table_mapZ typecolnameZ aliasnamerZcolnamesr\mrresultZtype_left_field_name left_tablesub_ffl)rrrrrr0hack_polymorphic_unions`          zCSQLAlchemyDataDriver._union_models2..hack_polymorphic_unionr)rT)rrrZlateral) r.r right_tablerr right_modelmodels_is_unionrrZ join_subqrr/rr0_union_models2sMz#SQLAlchemyDataDriver._union_models2cCsXd}|d|kr&|d|d|dfS|d|krH|d|d|dfStd||dS)N)><==>=<==IN!=><<>NOT INLIKENOT LIKEILIKE NOT ILIKEz SIMILAR TOzNOT SIMILAR TOEXISTS NOT EXISTS MEMBER_OF@@HISTORY NOT HISTORYrrrzInvalid filter operation)r)paramsZop_listr/r/r0 _parse_params's   z"SQLAlchemyDataDriver._parse_paramscCsddlm}m}t|tjr:|jdkr2||jd|j}nXt|t rt }|D]>}t|tjr|jdkrt||jd| |jqN| |qN|}|dkrd}|dkr||kS|dkr||kS|dkr||kS|d kr||kS|d kr||kS|d kr||kS|d kr| |S|d kr(| |S|dkr<| |S|dkrR| |S|dkrf| |S|dkr|| |S|dkr| |S|dkr| |S|||SdS)Nr print_debugr<.u<collect_filter_exp: Значение не загруженоZNULLrrrr)rr)rrrrrrrrrrr) cmf.includerr<rrr_valueDEBUGvaluerOrQin_ZlikeZilikeop)operfieldvalrr<new_valr8r/r/r0 _expression5sZ                         z SQLAlchemyDataDriver._expressionJoinDatazJleft_table, right_table, alias, left_field_name, right_model, right_modelsTc Cs0|r|j}|j}nT|j|}|s.|||t|tjtjfrLt|tjrft d|jd|jd|jd}|r|d|n|}| } | s|rt d|d|jd|jddS| d } t | d krt j j| jj|d } n2|dk r |j|d||| | d |d } n || |} ||| ||| | } | S)NuXНедопустимый тип поля для вложенной фильтрации rFup. Вложенная фильтрация возможно только для CmfRelation и CmfGenericRelationZ_sub__ujНе возможно построить запрос для вложенной фильтрации по u, т.к. у поля u# не указаны related_modelsrrrT)rr)rPinstance_classrrZ_raise_invalid_fieldrLrurvCmfSubclassedGenericRelationrrelated_modelsrgr*r+aliasedrrrrr) r.rr field_name from_aliasis_models_required field_clsrZ alias_partrrr join_datar/r/r0_calc_join_datais:       z$SQLAlchemyDataDriver._calc_join_dataF)joinscCsb|r tndd|jD}|D]:}|j|kr2q"||jt|jj|j d|jjj k}q"|S)uP TODO cmf_deleted Все данные для необходимых присоединений в joins, добавим недостающие присоединения в запрос. froms могут быть от филдов, при этом без джойна, чтобы принудительно создать join нужен force = True. Или более продвинутый способ определения наличия нужного джойна. Сейчас при построении основного запроса, джойны принудительно добавляются. А при расчёте фильтров, когда нет алиасов от полей, джойн делается только если нет нужного алиаса. cSs"h|]}t|tjjjr|jqSr/)rr*r selectableAliasr)r6rr/r/r0 sz3SQLAlchemyDataDriver._make_joins..r) set statementlocate_all_fromsr outerjoinrrrrrr)rjrforceZavailable_joinsrr/r/r0 _make_joinss  z SQLAlchemyDataDriver._make_joinscCsdd}g}|dkrd}|jj}n|j}|dD]0}|||||}|||j}|j}|j}q.|S)NrF) rrrsplitrrQrrr)r. join_pathrrrrrrr/r/r0_calc_joins_by_paths  z(SQLAlchemyDataDriver._calc_joins_by_pathcCs0t|jdd}t|d|jd|dS)N,z, u+ не существует у модели u, но есть: )strrr4rrP)rrZfields_r/r/r0r sz)SQLAlchemyDataDriver._raise_invalid_fieldc 2 Cs |s |dfSt|dtrdg} |D]4} |j| ||||||d\}} | dkrLq"| | q"||| fS|ddkr|j|dd||||||dS|ddkr|j|dd|||tj|||dS|ddkr t|dks|dd krtd |j||d |d }|dfS| |\} } }t t |t r0|j }t t |trH|jj }t|ttfrg}|D]r}t t |t rt t |j tr||j jj n ||j n*t t |tr||jj n ||q`|}| d }|j|d}d}t|dkrTt |tjtjtjfrT|d} d |dd| |g}d} |dkr|jj}nrt|dkr| d \}}} |j|||d}|||}|d}|j}|j| }|j}d}n|dkr|jj}| dkr| dkr|jd}|std| d|j|dkr| | |t |tjr| dkrVt|tsVt|tsVtd| ||| d krx|dk rxtd| ||| dkrddl!m"}|j#|d||d}|dkrtd|d|j$r|t%j&}| d kr|j'(|)|j*|k|j+|jk,}||fS|j-sF|j'(|)|j*|k|j+|jk|.d |j/|,}ntj01tj2j3|j*g|gd!4|j*5|.d |j/|5|j+|jk}|6|t|j7d|k}|| d krt|j7ddknt|j7ddkfSn&|8}||}|j9r|j:}|j;}n"|j<r|j;}|j:}n td"||| d krT|j'(|)||k,}||fS|j=s|j'(|)||k|.d ||,}n~tj01tj2j3|g|gd!4|5|.d ||}|6|t|j7d|k}|| d krt|j7ddknt|j7ddkfS|| d kr|n|fS| d#kr|j#|d||d}|dkrPtd|d$|j>|ddd|d%} | j}!| j}"|j'(|"j7j}#|j$r|t%j&}$|$j}%|#|%|$j/|"j7jk}#|#)|$j*|k|$j+|jk}#nh|j<rd&}&d'}'nd'}&d&}'|8}||}$|$j}%t?|$|&}(t?|$|'})|#|%|)|"j7jk}#|#)|(|k}#t@|"j7d(rT|sTd(d)dg}*ng}*t@|"j7d*rz|sz|*d*d)dgg}*|r|*|g}*|j|*|!|!j|#|"||d\}#}*|#)|*}#|#,}#| Ad+r|#}#||#fStd,| ||n$t |tjtjf r&| dkr | Ad+rd-nd} dd |g}| d#kr8td.| |||j#|d||d}|dkrdtd|d/|j>|ddd|d%}+|+j},|+j}-|jBd0}.t?|-j7|.}/|j'(|/}#|.d)|g}*t@|-j7d(r|*d(d)dgg}*|r|*|g}*|j|*|,|,j|#|-||d\}#}*|#)|*}#|#,}#| Ad+ r|#}#||#fSt |tjC r| d0} | d1k r|j>|ddd|d%} | j}!| j}"|j'(|"j7j}#|t%j&}$|$j}%|#|%|$j/|"j7jk}#|#)t|.d |$j*||.d |$jD||$j+d2k}#|j#|| ||d}|t|.d |||.d ||#fS| d3k rt |tj r(tEd4t%jF}!|!jj}"|j#|jd||d}|dk rdtd|"d$|j'(|"j7j}#d5}'||!}0t?|0|'})|#)|)|k}#|d6d)|jgg}*|j|*|!|!j|#|"d7\}#}*|#)|*}#|#,}#| Ad+ r|#}#||#fS| dk rFt%jGjH|j| |ddd8gd9}1|1 s"d:g}1|j#|d||d}||.d ||1fS|j#|| ||d}|dk rxtd|d;| d<||.| ||fSdS)=Nr) field_tableinclude_deleted include_dummyZANDrOR)cmdr#r$r%order_by)rrutОперация order_by в фильтре должна быть в формате: ["order_by", "=", [поля..]]rrrFFr)rTANYrru Поле u не найдено у )rruПравое значение для IN, NOT IN для m2m должно быть list, т.е. в квадратных скобках, например "field", "IN", "[obj.id]"uНедопустимая операция над m2m полем (сравнение = только на None), допустимые IN, NOT IN, = None)rrrrr< with_aliasuВ таблице u3 нет поля id для фильтрации m2mrruDНадо указать для поля либо left либо right)rruG нет поля id для фильтрации в подзапросе)rrr cmf_deletedris_dummyzNOT uhНедопустимая операция над m2m полем, допустимые IN, NOT IN, = NonerulНедопустимая операция над backref полем, допустимые EXISTS, NOT EXISTSu7 нет поля id для фильтрации backrefrrZ rg_members)rruAHistory для м2м полей не поддерживаетсяobj_idr)r#')rrZ search_queryZonly_idssliceZNOTFOUNDu нет поля u для фильтрации)IrrOcollect_filter_exprQr*or_rgr order_queryrrLrrrrrrrrrZ CmfM2MBaser|r}joinrr rpartitionr rrrrPr rr< _query_columnZ nested_fieldsrZ RelationCachesessionrjfilterZ parent_idZ parent_fieldexistsZORM_RELCACHE_EXISTS_VIA_JOINrZchild_idr+r rrZdistinctrrr m2m_model_clsrightrrleftZORM_M2M_EXISTS_VIA_JOINrrrSr3backrefCmfRelationBaseZ parent_coderZCmfOrmColumnHistoryZ CmfFullSearchZfulltext_search)2r.Z in_filterrrrjr'r#r$r%Z filter_resr8ZexprrAr@Z_rightiZ left_partsrr/rrHrrr<Z field_column m2m_modelZ m2m_subqueryZ dp_m2m_modelZdp_m2m_field_selfZdp_m2m_field_theirZ self_columnZsub_dataZ sub_modelZ sub_tableZ sub_queryZ m2m_sa_modelZ m2m_tablemy_id_field_namesub_id_field_name my_sa_field sub_sa_fieldZsub_filter_expZ backref_dataZ backref_modelZ backref_tableZbackref_column_nameZbackref_columnZ sub_sa_modelZ found_idsr/r/r0r5sf                "             6        2                                    z'SQLAlchemyDataDriver.collect_filter_expcCs |dS)NZ _m2m_directr/rr/r/r0_m2m_depth_alias_name sz*SQLAlchemyDataDriver._m2m_depth_alias_namecCs |dS)NZ_m2m_idr/rJr/r/r0_m2m_alias_namesz$SQLAlchemyDataDriver._m2m_alias_namecCsh|dg}t|tr t|}|r8t|dts8|g}|D]"\}}||jkr@||d|gq@|S)u Преобразуем обычные kwargs в продвинутый фильтр для однообразной логики фильтрования :param model: :param kwargs: :return: r=rr) rZrr"r loadsrOr=rrQ)r.rrB smart_filterr7r8r/r/r0 _get_filters    z SQLAlchemyDataDriver._get_filtercCs|r4|jD]$}t|tjjjr|j|jkr|}qd|kr|d\}}|dkrvt |j |d}t t || |}qt |j |d}nt |j |d}|dk r|r| t |ddS|SdS)uДля полей "основной" таблицы не нужен префикс таблицы, для этого нужно указать with_alias=Falser)sumZavgcountminmaxNrF)rrrr*rrrrrrrr rr"r4)rrrjr/Zfrom_Z func_nameZcolumn_name_shortcolumnr/r/r0r;)sz"SQLAlchemyDataDriver._query_column parent_joinc s|r |jn|jj}|D]\}|j|t|d}|dk rJ||q|r|r\|jp`gn|g} fdd| D} | s|q| dj} |r|j nd} |j | || d|d} | sq|| |j |||| j | dqdS) u2Соберём поля и таблицы по fflr.Ncs>g|]6}|jrttjtjfrttjsqSr/)rrZrLrurvr )r6Z related_modelrrr/r0rHs  z?SQLAlchemyDataDriver._select_joined_columns..rF)rrrU) rrrr=r;boolrQZ right_modelsr rr_select_joined_columnsr)r.ZfflrrrrVrrrTr Z fields_clsZ left_modelrrr/rWr0rY=s6   z+SQLAlchemyDataDriver._select_joined_columns)r$group_byr%cOs||}|} ddlm} | jd|j|d} t|| } | rZ| } | | } nht } g}g}| ||||| j |} |j | |dd} t | dkr|  d}t||| t|dkr|j}tt|D]}|||||<q|||}|r,|j|||| ||d\} }| |} |rg}|D]T}||jjkr`||jj|n,|d |jjkr:||jj|d q:| j|} | S) Nr CMF_CACHEzquery: full_fields_loadT)rg{Gz?r$r%r)rr>rr\hashrrr^Z_cloneZ with_sessionrrYrjrrkrgrlrangerOr5r=rrrQrZ)r.rr^r$rZr%rArBrrr\r\rjZstart_calc_timerrZclone_qpkeyrDrN filter_exprr/r/r0 _create_query[sH         z"SQLAlchemyDataDriver._create_query) r(r^r4 for_updatemapperr$load_m2mrZaggregate_selectr%c Ost|t }t|}|| }|| } |j|f||| | | d| }|j|||| |d}|rn|j|}| r| }td}|j |||d}|S|r| }| }|j |||d}|r|r|j ||||| d|r| |SdS)Грубо селектим всю таблицу, фильтры: - в kwargs: поле -> значение )r^r$r%rZrh)r4rhrZsimpler^rr_N)rr! get_mappermap_args map_kwargsrd with_labelsr7r4all list_to_cmfwith_for_update _load_m2mpost_mapping_hook)r.rr(r^r4rerfr$rgrZrhr%rArB mapper_is_my_argsqZsql_resZres_objsr/r/r0rOsJ       zSQLAlchemyDataDriver.list)r^rZcs ||}||}|j|fd|i|}|r||fdd|D}|jj|jj|t fj | d} t |j| } g} | D]<} i} t|D]\}}| || |<q| d| d<| | q| S|jj|jjt g d} |j| }|S)rir^csg|]}jj|qSr/)rr)r6rrr/r0rsz.SQLAlchemyDataDriver.count..Nr+rQ)rlrmrdrnrr select_fromfromsZwith_only_columnsr rQrZr(rOr<r enumeraterQZscalar)r.rr^rZrArBrurvrZcount_qres_listresrowZrow_resrDrrQr/rwr0rQs>       "zSQLAlchemyDataDriver.count)r(r^rerfr$r%cOst|t } t|}||}|| } |j|f||||d| } | } |j| |ddg|d} |rr| } | } | r|j | ||d} |j | g||||d| r| | SdS)u Выбрать один объект по фильтру: - в args - первичный ключ(опционально) - в kwargs - другие поля: поле -> значение)r^r$r%rr)r4rrj)rfr$r%N) rr!rkrlrmrdrnr7rqfirst object_to_cmfrrrs)r.rr(r^rerfr$r%rArBrtrv sa_instanceZres_objr/r/r0rZs(    zSQLAlchemyDataDriver.getc= s,|r|s dSt|tfdddd}t|}tt}tt} tt} |D]&} || | | | j| qN| D]t} tt| } |D]`}||sqt| |d}|rt|t j rt|t j r||qt|t j t j fr| || qqz|D] \}}g}|D]8} tt| j} | j |}|r t|t j r rt| |}|jdk rt|jdk r ||jq d|_|jdkrd|_|}t| |}|j}|dkrtd|| n`t| |d}|dk r|r ||q t| |d|}t| |d}|dkrtd|| |r |dd }|||| q |D]\} }tt| d} | slqNtd d t|g}|j| ||||d }|D]H}|}||D]0} rt| |}||_||_n t| ||qqqN|r|j||||||d q| D]\}}|D]} | j} | | }t| |}t|t j r|jrt}|jd} | d fdd|Dg}!tt||f| ii}"|D]r}#|!}$d|#j kr|s|!dddgg}$d|#j kr|s|$dddgg}$|j|#|"|$||d}%|j|%|#d||#j<q|sHd dlm }&m!}'|&|'j"d| jd|jdqnVt#|dkrrtt$t%|&'}(n,t|(t)j*+dd|Dd'}(tt})|(D]}*t|*| }+|)|+|*qg},|D]} |)| }-rNt| |}|-sg|_|jdkrg|_q|,|-|||_|jdkr@t-|j|_|,.|jn<|-sdt| |gq|,|-||}t| |||,.|q|,r |j|,|||||d qt|t j r|j/rd}.d }/nd }.d}/|0}0|1|0}1t|1|.}2t|1|/}3|(|2|32|23fd!d|D}4tt}5tt}6t|4D]@\}7}8|8r>|7sVq>|6|8|7|5|8dd |8q>|5D](\} }9tt| } td d |9g}|st4| drdddg|g}|st4| drdddg|g}|j| ||||dd"},|,r"|j|,|||||d |,D]} |6| D]t}7||7D]d}:rt|:|};|;jdkrfg|;_|;j| n0t|:|d}|dkrg}t|:|||| qBq6q&q|D]f} r| |}<|.obj_idcSsdd}t|S)NcSsttSrX)rrOr/r/r/r0id_to_list_factoryszWSQLAlchemyDataDriver._load_m2m..model_to_id_factory..id_to_list_factoryr)rr/r/r0model_to_id_factorysz;SQLAlchemyDataDriver._load_m2m..model_to_id_factory.u#Не загружено id поле:rrr)r^rfr=r_rcsg|] }|qSr/r/r6or2r/r0rsz2SQLAlchemyDataDriver._load_m2m..r0rFr1)r^r=r$r%r*rzORM: skip load field rFz die no related modelsrcSsi|]\}}||jqSr/)r)r6r7rr/r/r0r9sz2SQLAlchemyDataDriver._load_m2m..rrrcsg|] }|qSr/r/rrr/r0rs)r^r=rfrg)6rr"rrOrQrPrrrLrrrCr8ZCmfBackrefBaser=rZrZ_oldZ id_fieldnamersetattr partitionr#rrrBdictr$r rdr7rRrrr<rrgnextitervaluesrorjr*r+rrprrrAr?rr=rrSZ apply_changes)=r.Zobjsr^rfr$r%rZto_load_mappingZ my_objs_by_idZmy_objs_by_model_nameZmodels_by_rel_fieldsobjrrrrZ model_to_idZalready_loaded_obj_listrZ id_field_nameZid_fieldZid_field_valueZ field_valueZrel_model_nameZid_to_obj_listZ_filterZ load_objsZload_objZ loaded_idZ models_listZobj_listZm2mqueryZbackref_field_nameZfilter_qZ field_fflZ rel_modelZmodel_query_filterrvrr<r{Zres_dictrZm2m_idZsub_objsZ ref_obj_listrFrGrErrHrIZ m2m_queryZ id_by_modelsZmy_id_by_sub_idZmy_idZsub_idZ sub_id_listZmy_objZmy_fieldZ obj_fieldr/)rr2r0rrs                                                        zSQLAlchemyDataDriver._load_m2mcCs6g}|D](}t|tjr&||jq||q|SrX)rrrrQr)r.rAZnew_argsargr/r/r0rls   zSQLAlchemyDataDriver.map_argscCs:i}|D](\}}t|tjr,|j||<q |||<q |SrX)r=rrrr)r.rBZ new_kwargsr7r8r/r/r0rms    zSQLAlchemyDataDriver.map_kwargs) no_reloadc Os||}||}|}||j}|} t} | || z|| |j | gdWnt k r} z4|j t | jttrt| | t| | W5d} ~ XYn0tk r} z|j | W5d} ~ XYnXd|_|rdS| j| |d} | d| _| S)um no_reload - не загружать объект из базы и не возвращать ZobjectsNT) cmf_entity)rlrmr>rrDr!rk cmf_to_objectrflushrrrrorigrlookuprrrris_newrrs) r.instancerrArBru_kwargsrrrrfrrr/r/r0create"s2        zSQLAlchemyDataDriver.createc s||}||}|}|j}tj}|tfdd|D} || | } z| | Wn0t k r} z|j | W5d} ~ XYnXS)u*TODO логическое удалениеc3s|]}t|VqdSrXrr6frr/r0 Nsz.SQLAlchemyDataDriver.delete..N)rlrmr>rrDrrlrrjrZdeleterrr) r.rrArBrurrrrbget_argsrrr/rr0rFs     zSQLAlchemyDataDriver.deletec Ks`||}|}||}|||}|rP|j||||||d\}} || }|jdd} | S)Nr_Zfetch)Zsynchronize_session)rr>rjrOr5r=r) r.rr$r%rBrrrjrNrcZ delete_cntr/r/r0 bulk_deleteWs       z SQLAlchemyDataDriver.bulk_deleter_)rcOsddlm}||}|} | |} |||} | r\|j| ||| ||d\} } | | } | j} | j }|| | |j }|D]}| |}q|| }| |}dd|DS)Nrrr_cSsg|] }|dqS)rr/)r6r}r/r/r0rysz4SQLAlchemyDataDriver.bulk_update..)r*rrr>rjrOr5r=Z _criterionZ_join_entitiesr returningrrxrr)r.rrr$r%rArBrrrrjrNrcZfilter_criteriaZ join_clausesstmtZ join_clauserr/r/r0 bulk_updatecs$         z SQLAlchemyDataDriver.bulk_updater3) chunk_sizec sddlm}ddlm}m}|s$gS||}|j}dd|jDdd|jD} |t} |D]0} fdd| D} | t |  | q`d d |D} | }z||}| D]\}}| t|}|r|D]4} |D]*}|d kr|| |<qtd |d qqtdt||D]&}||||}|||qq| WStk r~}z|j|W5d}~XYnXdS)Nrr)insertr cSsh|] }|jqSr/rrr/r/r0rsz3SQLAlchemyDataDriver.bulk_insert..cSs6h|].}|js|jdkr|jdkr|jdkr|jqS)rN)rmrZserver_defaultrErr/r/r0rs   csi|]\}}|kr||qSr/r/r5Z table_colsr/r0r9sz4SQLAlchemyDataDriver.bulk_insert..cSsg|] }|dqS)rr/)r6rr/r/r0rsz4SQLAlchemyDataDriver.bulk_insert..)Zcmf_created_atZcmf_modified_atubulk_insert: колонка 'u>' NOT NULL без default; передайте значение) collectionsrr*rr rrrrOr= frozensetkeysrQr>rZnow ValueErrorrargrrrrr)r.rrrrZ sa_insertr rZtblZrequired_namesZbucketsr}rZidsrZ base_stmtZkeysetZrowsZneed_addrrDZbatchrr/rr0 bulk_insert{sB    z SQLAlchemyDataDriver.bulk_insertr]c s>ddlm}|}|t}tj}|tfdd|D} i} tj D]\} } t | tj rpqZt | tj r~qZt | tj rqZt | tjrqZt | tjrqZt | tjrqZt | tjrqZt| } t| tjr| jdksZ| jrqZ| js| jsqZ| } | dkr(tdtd| d| | | | <| jrZd| _qZ| s`td d S||| }|t||d| dk|j }z$t!t"|#|d kst$d Wnt%k r}z2|j&'t|j(t)*t+rt,|t-|W5d}~XYn2tk r8}z|j&'|W5d}~XYnXS) Nrrc3s|]}t|VqdSrXrrrr/r0rsz1SQLAlchemyDataDriver.update_v2...u%Попытка update field=Ellipsis rFzDEV: warning! UPDATE u* не содержит изменений.ruAДоп.защита, что только 1 объект в update).r*rr>rrrlrlrrr=rLrurvryrwr|rxr}rrrrrpZ is_changedZ orm_dirtyZ db_formatrrrirrrrrgrOrAssertionErrorrrrrrrrrr)r.rr^rArBrrrrbrrr7r8Zvvrrr/rr0 update_v2sb              $$  zSQLAlchemyDataDriver.update_v2c s\ddlm}|jr,|jf|ddi|S||}||}|}|t}tj }|t fdd|D} | | | } t } | | z|j| gdWntk r} z0|jt| jttrt| t| W5d} ~ XYn2tk r2} z|j| W5d} ~ XYnXj} | j| |d}| | |_|S)Nrr-r^c3s|]}t|VqdSrXrrrr/r0rsz.SQLAlchemyDataDriver.update..r)rr^)rr<ZORM_RAW_UPDATE_V2rrlrmr>rrrlrrjrZr!rkrrrrrrrrrrrrrrrrs)r.rr^rArBr<rrrbrrrfrrrr/rr0rs6       zSQLAlchemyDataDriver.updatecstdSrX)r:before_requestr-rCr/r0rsz#SQLAlchemyDataDriver.before_requestcCsddlm}||t}|t|dkrjddlm}m}||j dt|t}|j t|dkrddlm}m}||j dt|t dd t dS) Nrr[g333333?rzPROF Session.commit() got rzPROF Session.remove() got TZnear_commit_or_rollback) rr\ cache_unlockZcache_transaction_startrr>rrr<rremover%r&)r.r\Zprof_strr<r/r/r0rs    zSQLAlchemyDataDriver.commitcCs>ddlm}tdd|||jtdS)Nrr[Tr)rr\r%rr>rrr&)r.r\r/r/r0r(s     zSQLAlchemyDataDriver.rollbackcOs|j||SrXr>rjr@r/r/r0rj0szSQLAlchemyDataDriver.querycOs|j||SrXrr@r/r/r0query_deprecated3sz%SQLAlchemyDataDriver.query_deprecatedcsddlm}|r |ddkr |Sg}|s:jp0g}|}|rB|}|rX|rX|sX|dg7}tdfdd }|D]&} d} | d r| d d } d } || } |jjdj | } | d kr|jjdj | } | d kr|j D]} | d | kr| d} qq| d krt d| dddl m }|jr|t| jtjjr@| d} n.calc_dp_column_nameF-rNTrexpruНе найдено поле uK или тип поля не поддерживает сортировку.r-r2)Zminutes)Zcmf.data_providers.baserZorderingrr"r3rryrrZZcolumn_descriptionsrrr<ZORM_ORDERBY_IDX_SKIPrrr*rrrrtZDateTimerZdescr()r.rjr(r4rhrrZ order_fieldsrr\Z order_descZdp_column_nameZ dp_columnZcolumn_descriptionr<r/r*r0r76sP                z SQLAlchemyDataDriver.order_querycst|dSrX)r: make_models)rTrrCr/r0rsz SQLAlchemyDataDriver.make_modelscCs^ttD]P}tt|}t|tr|jr(q|jr<|j|jkrr Z connectionZget_table_namesFileExistsErrorrrdb_metaZ create_allr,repathr9getcwdrZset_main_optionr<rZstamp)r.rZ inspectorZalembic_config_fileZalembic_configr/r/r0init_dbs   zSQLAlchemyDataDriver.init_dbcCs|jjdSrX)r,ZpoolZdisposer-r/r/r0rsz'SQLAlchemyDataDriver.close_all_sessions)N)N)NTNN)F)N)NT)N)FF)FF)NNNN)Mrrr__doc__rr*ZMetaDatarextZ declarativeZdeclarative_baserrrYrdr1r; classmethodrr^rkrrrrrrrrrrUrrr staticmethodrrrrrrrr r and_r5rKrLrOr;rYrdrOrQrZrrrlrmrrrrrrrrrrrrjrr7rr?rr __classcell__r/r/rCr0r)!s  5   K $3 & S  1  #   U   3 "$ $  $ 1=(  Yr)c@seZdZddZdS)SAModelAccessorcCs|dkr dSttt|S)N)Z_pytestfixturefunctionZ__test__)r)rrr)r.itemr/r/r0 __getattr__szSAModelAccessor.__getattr__N)rrrrr/r/r/r0rsr)?rerrrrrtypingrrZdatetimerZsqlalchemy.typesr*Zsqlalchemy.ext.declarativeZsqlalchemy.ormZsqlalchemy.sql.selectableZsqlalchemy.excrr r r Zflaskr r Zalembic.configrZalembicrZpsycopg2.errorcodesrZpsycopg2rrZcmf.fields.base_fieldsrrZcmfrrrZ base_errorrrrZmodels.base_modelrrrrbaser r!r"r#Zutil.immutablesr$Z util.cmfutilr%r&r'r(r)rr/r/r/r0sX             *