TfB<dZddlZddlmcmZddlZddlZddl m Z ej dGdde ZGdde Zd Zd d ZdS) z5 Created on Sat May 18 19:24:45 2024 @author: atdou N)pyplotc*eZdZdZdZdZdZdZdS)create_regress_dataz Description ----------- Creates an example dataset of arbitrary dimensions, roughly following a regression function, f, and also populated with outliers c6tj|_dS)aw Parameters ---------- dim: int the dimension of the X data Attributes ---------- dim: int data: pandas Datafarme this is the dataframe we're trying to create. It will ultimately be of the form: ["x_0", "x_1", ..., "x_n", "y", "f", "Residuals", "Outlying?"] Npandas DataFramedataselfs bC:\Users\atdou\OneDrive\Desktop\Files\Coding\Python\Programs\XtraMLTools\test_auxiliary_classes.py__init__zcreate_regress_data.__init__$&& c||_||_||_||_||_t j|j|z |jf|j|jz z|jz|_t j||jf|j|jz z|jz|_tj t j |j|jfdt|jD|_ dS)b Description ----------- generating N random points along x axis, n_out of which are (possible) outliers. Parameters ---------- x_min: float x_max: float N: int number of total points, including outliers n_out: int the number of outliers dim: int the dimension of X/number of columns of X Attributes ---------- N: int n_out: int dim: int x_min: float x_max: float X_norm: numpy array of X coordinates of going to be ostensibly normal y points X_out: numpy array of X coordinates of going to be ostensibly outlier y points X: pandas DataFrame X_norm and X_out concatenated and turned into a dataframe c2g|]}dt|zSx_str.0ns r z-create_regress_data.set_X..Ks",R,R,RQT#a&&[,R,R,RrcolumnsN)Nn_outdimx_minx_maxnumpyrandomX_normX_outr r concatenaterangeXr r"r#rr r!s rset_Xzcreate_regress_data.set_X$s>   l))46%<*ABBDJtzDYZ[_[ee \((%)9::DJtz.ms#@@@qdiill@@@rc:g|]}|Sr/r0r2s rrz-create_regress_data.set_y..ns#>>>aTYYq\\>>>r)sizeN)r1r$arrayr&r'r%normalrr uniformr Seriesr(fy) r r1devmin_ymax_yf_normf_outy_normy_outs ` rset_yzcreate_regress_data.set_yMs> @@@@DK@@@AA >>>>4:>>>??3u|22tz8I2JJJJ ,,UE ,KKKu0&%AABBu0&%AABBrctj|jdg}tj|jdg}tj|j||gd|_|jd|jdz |jd<||_||_tj |jdgd}|d|dz }|jd }|jd }|jd kr'|d|j|zz |d|j|zz} } n%|jd kr||j|zz ||j|zz} } g} tt|jD]]} |jj| df| ks|jj| df| kr| d H| d ^| |jd <dS)a@ Description ----------- Now we concatenate the X, and y, and f points. And we classify points as outlying or not. Parameters ---------- metric: string this is "IQR", or "std" factor: float this is the parameter that multiplies either IQR = Q[3] - Q[1], or std, when classifying outlies, i.e., (Q[1] - factor*IQR, Q[3] + factor*IQR), or (mean - factor*std, mean + factor*std) Attributes ---------- metric: string factor: float r:rr;)axis Residuals)rg?g?g?rEIQRstdTF Outlying?N)r r r:r;concatr*r metricfactorr$quantilemeanrJr)lenlocappend) r rMrNr:r;QrIrPrJ Res_low_boundRes_high_boundrow_outlier_setis rclassify_outliersz%create_regress_data.classify_outliersts&  TVu 5 5 5  TVu 5 5 5M461a.q999 !%3$)C.!@ +  N49[12F2F2F G Gd1Q4iy%**,,i $((** ;%  ,-aD4;s?,BAaD4;WZ?DZ>MM [E ! !,0S,@$t{SVBV>Ms49~~&& . .A a m,}<<$)-PQR]P]B^aoBoBo&&t,,,,&&u----!0 +rN)__name__ __module__ __qualname____doc__rr,rCrYr/rrrr sb ''' 'T'T'TR%C%C%CN'1'1'1'1'1rrc$eZdZdZdZdZdZdS)create_classy_dataz Description ----------- Creates an example dataset of arbitrary dimensions, roughly following a classification function, f, and also populated with outliers c6tj|_dS)a\ Parameters ---------- dim: int the dimension of the X data Attributes ---------- dim: int data: pandas Datafarme this is the dataframe we're trying to create. It will ultimately be of the form: ["x_0", "x_1", ..., "x_n", "Class"] Nrr s rrzcreate_classy_data.__init__rrc*||_||_||_||_||_t j|j|z |jf|j|jz z|jz|_t j||jf|j|jz z|jz|_t j |j|jf|_ tj |j dt|jD|_dS)rc2g|]}dt|zSrrrs rrz,create_classy_data.set_X..s":`:`:`14A;:`:`:`rrN)rr r!r"r#r$r%r&r'r(X_arrayr r r)r*r+s rr,zcreate_classy_data.set_Xs>   l))46%<*ABBDJtzDYZ[_[ee \((%)9::DJtz 0, etc. Can obviously be more complicated than this. f_values: list of the possible values f can output. These would be the numbers associated with the different classes p_norm: float probability the normal points assume value f says they should. p_out: float probability the outlier points assume value f says they should. Attributes ---------- func: function this is f f_values: list this is the list of possible values f can output. This should be integers starting from 0. f: pandas DataFrame this is f evaluated at all X points y: pandas DataFrame y-coordinates of normal and outlier points cg|]}dS)Fr/rjs rrz,create_classy_data.set_y..s*U*U*UQ5*U*U*Urcg|]}dS)Tr/rfs rrz,create_classy_data.set_y..s)J)J)J1$)J)J)Jrc:g|]}|Sr/r0r2s rrz,create_classy_data.set_y..s#BBBtyy||BBBrc|dkrnfdtD}||<tjtjd|}|S)NFc&g|] }dz dz z S)rEr/)rrg num_valuesps rrzDcreate_classy_data.set_y..class_assigner.. s&EEEaacJqL)EEErrE)r)r$argmaxr% multinomial)sr:probscrmrlp_normp_outs @rclass_assignerz0create_classy_data.set_y..class_assignershU((AEEEEE53D3DEEEEE!H U\55a??@@AHrrEintrKr:r;N)r1f_valuesrQr$r6r)rr r(rc frompyfuncr r9astyper;r*copyr ) r r1rxrsrtoutlier_status_normoutlier_status_outoutlier_status_arrayf_arrayrunum_class_assignerrls ` `` @rrCzcreate_classy_data.set_ys_D   '' #k*U*U%@S:T:T*U*U*UVV"[)J)Jdj8I8I)J)J)JKK$02EGY1Z[[+BBBBT\BBBCC        #-naCC112FPPQQXXY^__FKKMM !5 +  # #rN)rZr[r\r]rr,rCr/rrr_r_sO ''' 'b'b'bR4 4 4 4 4 rr_ci}|D]}||}|ddk|ddkz}|ddk|ddkz}|ddk|ddkz}|ddk|ddkz}tj||g||gg||<|S)a Description ----------- outputs a dictionary of classification matrices comparing ROR outlier predictions on data_exp vs their actual status. Should create a data object and then fit or transform a ROR on it. Then feed the data object into data_exp argument above, and then feed in the dictionary of dataframes desired from ROR. Parameters ---------- data_exp: pandas dataframe this is a dataframe object as created above in that data class, i.e., data.data data_dict: dictionary of pandas dataframes should fit and or transform ROR to data.data. Then can create a dictionary of fit or transform df's with columns ["y_pred", "Residuals", "Outlying_Predictions"]. These are found in ROR.train_data, ROR.train_data_ave, and ROR.test_data, ROR.test_data_ave. So can test any of these guys. Returns ------- dictionary: of classification matrices rKTOutlying_PredictionF)sumr$r6) data_exp data_dictCM_dictnamer TPFPFNTNs rCM_ROR_predictionsrs,G77 $d*t4I/JD/PQ V V X X $e+5J0KT0QR W W Y Y $d*t4I/JE/QR W W Y Y $e+5J0KU0RS X X Z Z bWbW$566 Nrx_0c |d}d}||||z }|d|dz }||d|zz }||d|zz}|dd|zz } |dd|zz} |d||d|d<tjtj} ||} | | || d d | |||d|dd | ttj ||d| ttj | | d| ||| d| | | | d| dtj|dkrgd} d}|D]}||}| |d z}| j}| ||||d|d||| |||d|d|d|ddd| |dz }dSdS)a Description ----------- in data_exp.data, it plots user specified col_X vs. y, coloring the normal points grey, and the outlier points red. and it does the same for each dataframe in the data_dict. Further, it circles the outlier predictions of the models'' data in the data_dict, so can compare actual outliers (red) with predicted (green circles). x_min/max, y_min/max set the edges of the graph. Parameters ---------- data_exp : pandas dataframe, specifically of format create_data.data this is the experimental data that has the true outliers classed. data_dict : dictionary of pandas dataframes the dataframes could be from a ROR object, in ROR.train_data[m][n], and ROR.test_data[1], and ROR.train_data_ave, and ROR.test_data_ave, etc. col_X : string the column in X that we want to plot vs. y. As X can have multiple columns. greyredr;g?rK)TFColor)byr:black)color)rrpr3z!Feature Space and Actual OutliersN)darkblue orangered limegreenpurplery_predrv) linewidthrlabelrnoneod) edgecolors facecolorsmarkerrprE)maxminmaprfigureaxes sort_valuesplotscatter set_xtickslistr$linspace set_yticksset_xlim set_xlabelset_ylim set_ylabel set_titlegridindexlegend)rrcol_X color_norm color_outx_rangey_ranger"r#y_miny_maxaxdata_exp_sortedcolorsrgrr hueindicess rplot_ROR_predictionsr6s&JIuo!!##huo&9&9&;&;;Gsm!!HSM$5$5$7$77G UO   ! !CK /E UO   ! !CK /E SM    #g+ -E SM    #g+ -E -11:2V2VWWHW MOOO B**e*44OGGOE "OC$8'GJJJJJx x7HAJNNNMM$u~eUA6677888MM$u~eUA6677888KKuMM#KKuMM#LL4555 KMMMDAAA   DT?D1+C%+G GGHUOG,d8nW.EQR\_imG n n n JJxt,A'BCXc]SWXmSnEo#&Vcs  T T T IIKKK qDAA  r)r)r]builtins @py_builtins_pytest.assertion.rewrite assertionrewrite @pytest_arr r$ matplotlibrr%seedobjectrr_rrr/rrrs  "N1N1N1N1N1&N1N1N1bt t t t t t t t nB444444r