A geometric interpretation of the covariance matrixWord下载.docx
- 文档编号:17345520
- 上传时间:2022-12-01
- 格式:DOCX
- 页数:14
- 大小:310.20KB
A geometric interpretation of the covariance matrixWord下载.docx
《A geometric interpretation of the covariance matrixWord下载.docx》由会员分享,可在线阅读,更多相关《A geometric interpretation of the covariance matrixWord下载.docx(14页珍藏版)》请在冰豆网上搜索。
Gaussiandensityfunction.Fornormallydistributeddata,68%ofthesamplesfallwithintheintervaldefinedbythemeanplusandminusthestandarddeviation.
Weshowedthatanunbiasedestimatorofthesamplevariancecanbeobtainedby:
(1)
However,variancecanonlybeusedtoexplainthespreadofthedatainthedirectionsparalleltotheaxesofthefeaturespace.Considerthe2Dfeaturespaceshownbyfigure2:
Figure2.
Thediagnoalspreadofthedataiscapturedbythecovariance.
Forthisdata,wecouldcalculatethevariance
inthex-directionandthevariance
inthey-direction.However,thehorizontalspreadandtheverticalspreadofthedatadoesnotexplainthecleardiagonalcorrelation.Figure2clearlyshowsthatonaverage,ifthex-valueofadatapointincreases,thenalsothey-valueincreases,resultinginapositivecorrelation.Thiscorrelationcanbecapturedbyextendingthenotionofvariancetowhatiscalledthe‘covariance’ofthedata:
(2)
For2Ddata,wethusobtain
and
.Thesefourvaluescanbesummarizedinamatrix,calledthecovariancematrix:
(3)
Ifxispositivelycorrelatedwithy,yisalsopositivelycorrelatedwithx.Inotherwords,wecanstatethat
.Therefore,thecovariancematrixisalwaysasymmetricmatrixwiththevariancesonitsdiagonalandthecovariancesoff-diagonal.Two-dimensionalnormallydistributeddataisexplainedcompletelybyitsmeanandits
covariancematrix.Similarly,a
covariancematrixisusedtocapturethespreadofthree-dimensionaldata,anda
covariancematrixcapturesthespreadofN-dimensionaldata.
Figure3illustrateshowtheoverallshapeofthedatadefinesthecovariancematrix:
Figure3.
Thecovariancematrixdefinestheshapeofthedata.Diagonalspreadiscapturedbythecovariance,whileaxis-alignedspreadiscapturedbythevariance.
Inthenextsection,wewilldiscusshowthecovariancematrixcanbeinterpretedasalinearoperatorthattransformswhitedataintothedataweobserved.However,beforedivingintothetechnicaldetails,itisimportanttogainanintuitiveunderstandingofhoweigenvectorsandeigenvaluesuniquelydefinethecovariancematrix,andthereforetheshapeofourdata.
Aswesawinfigure3,thecovariancematrixdefinesboththespread(variance),andtheorientation(covariance)ofourdata.So,ifwewouldliketorepresentthecovariancematrixwithavectoranditsmagnitude,weshouldsimplytrytofindthevectorthatpointsintothedirectionofthelargestspreadofthedata,andwhosemagnitudeequalsthespread(variance)inthisdirection.
Ifwedefinethisvectoras
thentheprojectionofourdata
ontothisvectorisobtainedas
andthevarianceoftheprojecteddatais
.Sincewearelookingforthevector
thatpointsintothedirectionofthelargestvariance,weshouldchooseitscomponentssuchthatthecovariancematrix
oftheprojecteddataisaslargeaspossible.Maximizinganyfunctionoftheform
withrespectto
where
isanormalizedunitvector,canbeformulatedasasocalled
RayleighQuotient.ThemaximumofsuchaRayleighQuotientisobtainedbysetting
equaltothelargesteigenvectorofmatrix
.
Inotherwords,thelargesteigenvectorofthecovariancematrixalwayspointsintothedirectionofthelargestvarianceofthedata,andthemagnitudeofthisvectorequalsthecorrespondingeigenvalue.Thesecondlargesteigenvectorisalwaysorthogonaltothelargesteigenvector,andpointsintothedirectionofthesecondlargestspreadofthedata.
Nowlet’shavealookatsomeexamples.Inanearlierarticlewesawthatalineartransformationmatrix
iscompletelydefinedbyits
eigenvectorsandeigenvalues.Appliedtothecovariancematrix,thismeansthat:
(4)
where
isaneigenvectorof
and
isthecorrespondingeigenvalue.
Ifthecovariancematrixofourdataisadiagonalmatrix,suchthatthecovariancesarezero,thenthismeansthatthevariancesmustbeequaltotheeigenvalues
.Thisisillustratedbyfigure4,wheretheeigenvectorsareshowningreenandmagenta,andwheretheeigenvaluesclearlyequalthevariancecomponentsofthecovariancematrix.
Figure4.
Eigenvectorsofacovariancematrix
However,ifthecovariancematrixisnotdiagonal,suchthatthecovariancesarenotzero,thenthesituationisalittlemorecomplicated.Theeigenvaluesstillrepresentthevariancemagnitudeinthedirectionofthelargestspreadofthedata,andthevariancecomponentsofthecovariancematrixstillrepresentthevariancemagnitudeinthedirectionofthex-axisandy-axis.Butsincethedataisnotaxisaligned,thesevaluesarenotthesameanymoreasshownbyfigure5.
Figure5.
Eigenvaluesversusvariance
Bycomparingfigure5withfigure4,itbecomesclearthattheeigenvaluesrepresentthevarianceofthedataalongtheeigenvectordirections,whereasthevariancecomponentsofthecovariancematrixrepresentthespreadalongtheaxes.Iftherearenocovariances,thenbothvaluesareequal.
Nowlet’sforgetaboutcovariancematricesforamoment.Eachoftheexamplesinfigure3cansimplybeconsideredtobealinearlytransformedinstanceoffigure6:
Figure6.
Datawithunitcovariancematrixiscalledwhitedata.
Letthedatashownbyfigure6be
theneachoftheexamplesshownbyfigure3canbeobtainedbylinearlytransforming
:
(5)
isatransformationmatrixconsistingofarotationmatrix
andascalingmatrix
(6)
Thesematricesaredefinedas:
(7)
istherotationangle,and:
(8)
arethescalingfactorsinthexdirectionandtheydirectionrespectively.
Inthefollowingparagraphs,wewilldiscusstherelationbetweenthecovariancematrix
andthelineartransformationmatrix
Let’sstartwithunscaled(scaleequals1)andunrotateddata.Instatisticsthisisoftenreferedtoas‘whitedata’becauseitssamplesaredrawnfromastandardnormaldistributionandthereforecorrespondtowhite(uncorrelated)noise:
Figure7.
Whitedataisdatawithaunitcovariancematrix.
Thecovariancematrixofthis‘white’dataequalstheidentitymatrix,suchthatthevariancesandstandarddeviationsequal1andthecovarianceequalszero:
(9)
Nowlet’sscalethedatainthex-directionwithafactor4:
(10)
Thedata
nowlooksasfollows:
Figure8.
Varianceinthex-directionresultsinahorizontalscaling.
Thecovariancematrix
of
isnow:
(11)
Thus,thecovariancematrix
oftheresultingdata
isrelatedtothelineartransformation
thatisappliedtotheoriginaldataasfollows:
where
(12)
However,althoughequation(12)holdswhenthedataisscaledinthexandydirection,thequestionrisesifitalsoholdswhenarotationisapplied.Toinvestigatetherelationbetweenthelineartransformationmatrix
andthecovariancematrix
inthegeneralcase,wewillthereforetrytodecomposethecovariancematrixintotheproductofrotationandscalingmatrices.
Aswesawearlier,wecanrepresentthecovariancematrixbyitseigenvectorsandeigenvalues:
(13)
Equation(13)holdsforeacheigenvector-eigenvaluepairofmatrix
.Inthe2Dcase,weobtaintwoeigenvectorsandtwoeigenvalues.Thesystemoftwoequationsdefinedbyequation(13)canberepresentedefficientlyusingmatrixnotation:
(14)
isthematrixwhosecolumnsaretheeigenvectorsof
isthediagonalmatrixwhosenon-zeroelementsarethecorrespondingeigenvalues.
Thismeansthatwecanrepresentthecovariancematrixasafunctionofitseigenvectorsandeigenvalues:
(15)
Equation(15)iscalledtheeigendecompositionofthecovariancematrixandcanbeobtainedusinga
SingularValueDecompositionalgorithm.Whereastheeigenvectorsrepresentthedirectionsofthelargestvarianceofthedata,theeigenvaluesrepresentthemagnitudeofthisvarianceinthosedirections.Inotherwords,
representsarotationmatrix,while
representsascalingmatrix.Thecovariancematrixcanthusbedecomposedfurtheras:
(16)
isarotationmatrixand
isascalingmatrix.
Inequation(6)wedefinedalineartransformation
.Since
isadiagonalscalingmatrix,
.Furthermore,since
isanorthogonalmatrix,
.Therefore,
.Thecovariancematrixcanthusbewrittenas:
(17)
Inotherwords,ifweapplythelineartransformationdefinedby
totheoriginalwhitedata
shownbyfigure7,weobtaintherotatedandscaleddata
withcovariancematrix
.Thisisillustratedbyfigure10:
Figure10.
Thecovariancematrixrepresentsalineartransformationof
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- geometric interpretation of the covariance matrix
![提示](https://static.bdocx.com/images/bang_tan.gif)
链接地址:https://www.bdocx.com/doc/17345520.html