Coupling and Cohesion Measures for Evaluation of Component Reusability This paper provides an account of new measures of coupling and cohesion developed to assess the reusability of Java components retrieved from the internet by a search engine . These measures differ from the majority of established metrics in two respects : they reflect the degree to which entities are coupled or resemble each other , and they take account of indirect couplings or similarities . An empirical comparison of the new measures with eight established metrics shows the new measures are consistently superior at ranking components according to their reusability . D. 2.8.3 -LSB- Metrics -RSB- : Complexity measures . Measurement , Experimentation . The work reported in this paper arose as part of a project that retrieves Java components from the internet -LSB- 1 -RSB- . However , components retrieved from the internet are notoriously variable in quality . It seems highly desirable that the search engine should also provide an indication of both how reliable the component is and how readily it may be adapted in a larger software system . A well designed component , in which the functionality has been appropriately distributed to its various subcomponents , is more likely to be fault free and easier to adapt . Appropriate distribution of function underlies two key concepts : coupling and cohesion . Coupling is the extent to which the various subcomponents interact . If they are highly interdependent then changes to one are likely to have significant effects on others . Hence loose coupling is desirable . Cohesion is the extent to which the functions performed by a subsystem are related . If a subcomponent is responsible for a number of unrelated functions then the functionality has been poorly distributed to subcomponents . Hence high cohesion is a characteristic of a well designed subcomponent . We decided that the component search engine should provide the quality rankings of retrieved components based on measures of their coupling and cohesion . There is a substantial literature on coupling and cohesion metrics which is surveyed in the next section . We then describe in detail the metrics we have developed which attempt to address some of the limitations of existing metrics . In particular , we consider both the strength and transitivity of dependencies . The following section describes an empirical comparison of our proposed metrics and several popular alternatives as predictors of reusability . Section 5 presents an analysis of the results which demonstrate that our proposed metrics consistently outperform the others . The paper concludes with a discussion of the implications of the research . Cohesion is a measure of the extent to which the various functions performed by an entity are related to one another . Most metrics assess this by considering whether the methods of a class access similar sets of instance variables . Coupling is the degree of interaction between classes . Many researches have been done on software metrics -LSB- 8 -RSB- , the most important ones are selected used in our comparative study . Table 1 and Table 2 summarize the characteristics of these cohesion and coupling metrics . Table 1 . Coupling metrics Name Definition CBO -LSB- 4 -RSB- -LSB- 5 -RSB- -LSB- 11 -RSB- Classes are coupled if methods or instance variables in one class are used by the other . CBO for a class is number of other classes coupled with it . RFC -LSB- 4 -RSB- -LSB- 5 -RSB- Count of all methods in the class plus all methods called in other classes . CF -LSB- 3 -RSB- -LSB- 6 -RSB- Classes are coupled if methods or instance variables in one class are used by the other . CF for a software system is number of coupled class pairs divided by total number of class pairs . DAC -LSB- 9 -RSB- The number of attributes having other classes as their types . Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page . To copy otherwise , or republish , to post on servers or to redistribute to lists , requires prior specific permission and/or a fee . MSR ' 06 , May 22-23 , 2006 , Shanghai , China . Copyright 2006 ACM 1-59593-085-X / 06/0005 ... $ 5.00 . 18 Table 2 . Cohesion metrics Name Definition LCOM -LSB- 5 -RSB- Number of non-similar method pairs in a class of pairs . LCOM3 -LSB- 7 -RSB- -LSB- 9 -RSB- Number of connected components in graph whose vertices are methods and whose edges link similar methods . RLCOM -LSB- 10 -RSB- Ratio of number of non-similar method pairs to total number of method pairs in the class . TCC -LSB- 2 -RSB- Ratio of number of similar method pairs to total number of method pairs in the class . All of these measures have two important features in common . First , they treat relationship between a pair of classes or methods as a binary quantity ; second , they treat coupling and cohesion as an intransitive relation ; that is no account is taken of the indirect coupling and cohesion , although two of cohesion -LRB- LCOM3 -LSB- 7 -RSB- -LSB- 9 -RSB- and TCC -LSB- 2 -RSB- -RRB- have suggested extensions to incorporate indirect relationships between methods . In cohesion metrics , it should be noted that three of them -LRB- LCOM , LCOM3 and RLCOM -RRB- are in fact measures of lack of cohesion . TCC -LSB- 2 -RSB- , in contrast to the other three metrics , measures cohesion rather than its absence . In other respects it is similar to RLCOM , being the number of similar method pairs divided by the total number of method pairs . The study suggested that none of these measures was very effective in ranking the reusability of Java components . We therefore decided to develop alternative coupling and cohesion metrics in the hope of achieving superior performance . One obvious step was to develop measures that reflected the extent to which a pair of classes was coupled or a pair of methods resembled each other . Because none of the measures treated coupling or similarity as transitive relations , we decided that such indirect dependencies should be incorporated into our metrics . 3.1 Cohesion We develop a cohesion metric that takes account of both the degree of cohesion and transitive -LRB- i.e indirect -RRB- cohesion between methods . Methods are said to be similar if the sets of instance variables that they access overlap . We adopt a graph theoretical approach . The methods of the class are the vertices . Suppose a class has a set of method members M -LCB- M 1 , M 2 , ... M m -RCB- and let . V j -LCB- V j ,1 , V j ,2 , ... . V j , n -RCB- be the instance variables accessed by method M j . Then the edge from M j to M i exists if and only if V j V i is not null . Thus an edge of the graph reflects the similarity of the methods in that they have at least one instance variable in common . The similarity graph is undirected because intersection is a symmetric relation . The next step is to associate a number with each edge that reflects the extent to which the two methods have instance variables in common . We therefore define SimD -LRB- i , j -RRB- , our measure of direct similarity of two methods , M i and M j , as -LRB- -RRB- j i j i V V V V j i SimD = , where i j -LRB- SimD -LRB- j , j -RRB- is defined to be zero -RRB- . Note that 1 SimD -LRB- i , j -RRB- 0 . The extension of the measure to include indirect similarity proceeds along the same lines as we employed for indirect coupling . The strength of similarity provided by a path between two methods is the product of the SimD values of the edges that make up the path . Thus we define SimT -LRB- i , j , -RRB- , the transitive similarity between methods M i and M j due to a specific path , as -LRB- -RRB- = = t s t s e t s t s e V V V V t s SimD j i SimT , , , -RRB- , , -LRB- where e s , t denotes the edge between vertices s and t . As in the case of coupling , the path with the highest SimT value is selected to define the similarity of the two methods , Sim -LRB- i , j -RRB- . -RRB- , , -LRB- -RRB- , -LRB- max j i SimT j i Sim = where and is the set of all paths from M i to M j . This measure is used to provide a measure of the cohesion of the class , ClassCoh , by summing the similarities of all method pairs and dividing by the total number of such pairs : -RRB- , , -LRB- max arg -RRB- , -LRB- max j i SimT j i = m m j i Sim ClassCoh m j i = = 2 1 , -RRB- , -LRB- where m is the number of methods in the class . Finally , the weighted transitive cohesion of the complete software system , WTCoh , is defined as the mean cohesion of all the classes of which it is comprised : n ClassCoh WTCoh n j j = = 1 where n is the number of classes in the system . 3.2 Coupling As with cohesion measure , we regard software system as a directed graph , in which the vertices are the classes comprising the system . Suppose such a system comprises a set of classes C -LCB- C 1 , C 2 , ... C m -RCB- . Let M j -LCB- M j ,1 , M j ,2 , ... . M j , n -RCB- be the methods of the class C j , and R j , i the set of methods and instance variables in class C i invoked by class C j for j i -LRB- R j , j is defined to be null -RRB- . Then the edge from C j to C i exists if and only if R j , j is not null . Thus an edge of the graph reflects the direct coupling of one class to another . The graph is directed since R j , i is not necessarily equal to Ri , j . The next step is to associate a number with each edge that reflects the extent of direct coupling from one class to another . We define CoupD -LRB- i , j -RRB- , as the ratio of the number of methods in class j invoked by class I to the total number of methods in class I , which indicates the impact of class j to class i. -LRB- -RRB- i i j i M R R j i CoupD + = , , Then the indirect coupling between classes is included . Suppose that CoupD -LRB- i , j -RRB- and CoupD -LRB- j , k -RRB- have finite values but that CoupD -LRB- i , k -RRB- is zero . Thus although there is no direct coupling between classes C i and C k , there is a dependency because C i invokes methods in C j which in turn invokes methods in C k . The strength of this dependency depends on the two direct couplings of which it is composed , a reasonable measure is defined as : 19 CoupD -LRB- i , j -RRB- CoupD -LRB- j , k -RRB- . This notion is readily generalised . A coupling between two classes exists if there is a path from one to the other made up edges whose CoupD values are all non-zero . Thus we define CoupT -LRB- i , j , -RRB- , the transitive coupling between classes C i and C j due to a specific path , as -LRB- -RRB- + = = t s t s e s s t s e M R R t s CoupD j i CoupT , , , , -RRB- , , -LRB- e s , t denotes the edge between vertices s and t. Note first that CoupT includes the direct coupling , which corresponds to path of length one , and second that , because the CoupD values are necessarily less than one , transitive couplings due to longer paths will typically have lower values . In general there may be more than one path having a non-zero CoupT value between any two classes . We simply select the path with largest CoupT value and hence define Coup -LRB- i , j -RRB- , the strength of coupling between the two classes , C i and C j to be : -RRB- , , -LRB- -RRB- , -LRB- max j i CoupT j i Coup = where -RRB- , , -LRB- max arg -RRB- , -LRB- max j i CPT j i = and is the set of all paths from C i to C j . The final step is to use measure between each pair of classes as a basis for a measure of the total coupling of a software system . The weighted transitive coupling -LRB- WTCoup -RRB- of a system is thus defined m m j i Coup WTCoup m j i = = 2 1 , -RRB- , -LRB- where m is the number of classes in the system . In our study , the metrics are used for a specific purpose : predicting how much effort would be required to reuse a component within a larger system . We therefore chose to measure reusability as simply the number of lines of code that were added , modified or deleted -LRB- NLOC -RRB- in order to extend its functionality in a prescribed way . The more lines required , the lower the reusability . This appears to us to be a crude but reasonable measure of the effort that would be required to adapt a component for use within a larger system . Three case studies were carried out : Case 1 HTML Parser : The original components analysed HTML documents , eliminated tags and comments and output the text . The required extension was to count and output the number of tags found during parsing . Case 2 Lexical Tokenizer : The original components tokenized a text document using user supplied token rules and output the tokens on a web interface . The required extension was to count and output the number of tokens retrieved . Case 3 Barcode : The original components accepted a sequence of alphanumeric characters and generated the corresponding barcode . The required extension was to count the number of letters . For each case , 20 Java components were retrieved from a repository of about 10,000 Java components retrieved form the internet . The requisite extensions were then implemented by a very experienced Java programmer and NLOC counted . Despite the relative simplicity of the extensions , there was considerable variation in the quantity of extra code required . We then proceeded to investigate how successful the various measures of coupling and cohesion are in predicting this quantity . Our proposed metrics are compared with all the metrics reviewed in section 2 . In order to present the results on the same graph , those measures that do not produce values in the range -LRB- 0,1 -RRB- -LRB- i.e. CBO , RFC , DAC , LCOM and LCOM3 -RRB- were divided by 100 . Two approaches were used to evaluate the performance of the various measures in predicting reusability : linear regression and rank correlation . 5.1 Linear Regression The regression lines obtained for the five cohesion measures when applied to the HTML parser components are shown in Figure 1 . The results for the other two sets of components were similar . It is clear that some measures provide much more consistent predictors than others . There are no obvious systematic departures from linearity so the use of simple regression appears reasonable . The regression lines obtained for coupling measures demonstrate the same situation . The coefficient of determination , R 2 , provides a measure of how much of the variation in NLOC is accounted for by the measures . Table 3 and Table 4 display the values of R 2 obtained for each of the coupling and cohesion measures on all three sets of components . In each case , our proposed new measure , WTCoup and WTCoh gave the largest value of R 2 , indicating that it was the best linear predictor of reusability . The remaining measures produced at least one R 2 value so low as to indicate that that the correlation was not significantly above chance at the 5 % level . Figure 1 . Regression of cohesion measures against reusability Table 3 . R 2 values for coupling measure regression lines . Cases WTCoup CF CBO RFC DAC HTML Parser .846 .621 .259 .793 .254 Lexical Token . .836 .098 .004 .729 .738 Barcode Gen. .958 .693 .121 534 .507 20 Table 4 . R 2 values for cohesion measure regression lines . Cases WTCoh RLCOM LCOM3 LCOM TCC H. Parser .847 .319 .259 .564 .178 L. Token . .838 .783 .002 .709 .646 B. Gen. .892 .702 .177 .101 .785 5.2 Spearman Rank Correlation Although these results provide a strong indication that the proposed new measures are better predictors of reusability than the alternatives , our primary purpose is simply to rank a set of components retrieved from the repository . We therefore also computed the Spearman rank correlation coefficients between the rankings determined by NLOC and those produced by the various coupling and cohesion measures -LRB- Tables 5 and 6 -RRB- . Table 5 . Rank correlations values for coupling measures . Cases WTCoup CF CBO RFC DAC HTML Parser .975 .882 .465 .896 .507 Lexical Token . .952 .291 .117 .822 .817 Barcode Gen. .974 .758 .485 .656 .800 Table 6 . Rank correlations values for cohesion measures . Cases WTCoh RLCOM LCOM3 LCOM TCC H. Parser -.993 .522 .218 .564 -.343 L. Token . .838 .783 .002 .709 .646 Bar . Gen. .892 .702 .177 .101 .785 The relative performance of the various measures is consistent with the regression studies . In all cases , the two proposed measures , WTCoup and WTCoh , produced the highest rank correlations . They are in fact extremely high ; no value was lower than 0.95 . These results clearly demonstrate that our proposed metrics for coupling and cohesion are very good predictors of the number of lines of code required to make simple modifications to Java components retrieved from the internet and are superior to other measures . The majority of coupling and cohesion metrics treat coupling and similarity as simple binary quantities and ignore the transitive relationship . Both our proposed measures concern these issues : First , they are weighted ; that is , they use a numeric measure of the degree of coupling or similarity between entities rather than a binary quantity . Second they are transitive ; that is , they include indirect coupling or similarity mediated by intervening entities . It is reasonable to enquire whether both these characteristics are necessary to achieve good prediction performance . In fact our investigations suggest that both contribute to the performance . Although both WTCoup and WTCoh are good predictors , it is worth considering whether a linear combination might not produce even better results . Multiple regression for the Lexical Tokenizer components produced an R 2 of 0.981 ; the ranking produced using the regression coefficients to weight the terms had a Spearman correlation of 0.986 . These are superior to the results produced by each metric alone but not by a great margin simply because there original results leave only modest scope for improvement . Developing such a composite quality measure would entail assuming the relative weighting of the two metrics should be the same for all types of component . This work arose from , and is intended primarily as a contribution to , search engine technology . Nevertheless , we believe it may be of interest to a wider body of researchers : in particular , those involved in developing and evaluating software metrics . We are grateful to the four UK higher education funding bodies -LRB- for England , Scotland , Wales and Northern Ireland -RRB- for an Overseas Research Studentship -LRB- ORS/2002015010 -RRB- awarded to G. Gui . -LSB- 1 -RSB- Gui , G. and Scott , P. D. Vector Space Based on Hierarchical Weighting : A Component Ranking Approach to Component Retrieval . In Proceedings of the 6th International Workshop on Advanced Parallel Processing Technologies -LRB- APPT ' 05 -RRB- -LSB- 2 -RSB- Bieman , J. M. and Kang , B-Y . Cohesion and Reuse in an Object-Oriented System . In Proc . ACM Symposium on Software Reusability -LRB- SSR ' 95 -RRB- . -LRB- April 1995 -RRB- 259-262 . -LSB- 3 -RSB- Briand , L. , Devanbu , P. and Melo , W . An investigation into coupling measures for C++ . Proceedings of ICSE 1997 . -LSB- 4 -RSB- Brito e Abreu , F. and Melo , W. Evaluating the impact of OO Design on Software Quality . Proc . Third International Software Metrics Symposium . -LRB- Berin 1996 -RRB- . -LSB- 5 -RSB- Chidamber , S. R. and Kemerer , C. K . A Metrics Suite for Object Oriented Design . IEEE Transactions on Software Engineering , Vol . 20 -LRB- June 1994 -RRB- , 476-493 . -LSB- 6 -RSB- Harrison , R. , S.J.Counsell , & R.V.Nith . An Evaluation of the MOOD Set of Object-Oriented Software Metrics . IEEE Transactions on Software Engineering , Vol . 24 -LRB- June 1998 -RRB- , 491-496 . -LSB- 7 -RSB- Hitz , M. and Montazeri , B. Measuring coupling and cohesion in object-oriented systems . Proceedings of International Symposium on Applied Corporate Computing . -LRB- Monterrey , Mexico , 1995 -RRB- . -LSB- 8 -RSB- Kanmani , S. , Uthariraj , R. , Sankaranarayanan , V. and Thambidurai , P. Investigation into the Exploitation of Object-Oriented Features . ACM Sigsoft , Software Engineering Notes , Vol . 29 -LRB- March 2004 -RRB- . -LSB- 9 -RSB- Li , W. & Henry , S. Object-Oriented metrics that predict maintainability . Journal of Systems and Software . 23 -LRB- 2 -RRB- 1993 111-122 . -LSB- 10 -RSB- Li , X. , Liu , Z. Pan , B. & Xing , B . A Measurement Tool for Object Oriented Software and Measurement Experiments with It . In Proc . IWSM 2000 , 44-54 . -LSB- 11 -RSB- Subramanyam , R. & Krishnan , M. S. Empirical Analysis of CK Metrics for Object-Oriented Design Complexity : Implications for Software Defects . IEEE Transactions on Software Engineering , Vol . 29 -LRB- April 2003 -RRB- , 297-310 . 21

B O B O O O O O B O O O O O O O O O B O B O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B I I O O O O O O O O O O O O O B O B O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O B O O B I O O O O O O O O O O B O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

coupling
intransitive relation
linear regression
spearman rank correlation
search engine technology
reusability
cohesion
component search engine
binary quantity
java components

experimentary comparsion
cohesion metric


Evaluating

Similarity

Measures

A

Large-Scale

Study

in

the

Orkut

Social

Network

Online

information

services

have

grown

too

large

for

users

to

navigate

without

the

help

of

automated

tools

such

as

collaborative

filtering

which

makes

recommendations

to

users

based

on

their

collective

past

behavior

While

many

similarity

measures

have

been

proposed

and

individually

evaluated

they

have

not

been

evaluated

relative

to

each

other

in

a

large

real-world

environment

We

present

an

extensive

empirical

comparison

of

six

distinct

measures

of

similarity

for

recommending

online

communities

to

members

of

the

Orkut

social

network

We

determine

the

usefulness

of

the

different

recommendations

by

actually

measuring

users

propensity

to

visit

and

join

recommended

communities

We

also

examine

how

the

ordering

of

recommendations

influenced

user

selection

as

well

as

interesting

social

issues

that

arise

in

recommending

communities

within

a

real

social

network

H.

-LSB-

Database

Management

-RSB-

Database

Applications-data

mining

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Online

Information

Services

I.

-LSB-

Computing

Methodologies

-RSB-

Pattern

Recognition

Algorithms

measurement

human

factors

The

amount

of

information

available

online

grows

far

faster

than

an

individual

's

ability

to

assimilate

it

For

example

consider

communities

-LRB-

user-created

discussion

groups

-RRB-

within

Orkut

a

social-networking

website

-LRB-

-RRB-

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

KDD

August

Chicago

Illinois

USA

Copyright

ACM

affiliated

with

Google

The

original

mechanisms

for

users

to

find

communities

were

labor-intensive

including

searching

for

keywords

in

community

titles

and

descriptions

or

browsing

other

users

memberships

Four

months

after

its

January

debut

Orkut

had

over

communities

providing

the

necessity

and

opportunity

for

data-mining

for

automated

recommendations

There

are

now

-LRB-

May

-RRB-

over

communities

While

there

are

many

forms

of

recommender

systems

-LSB-

-RSB-

we

chose

a

collaborative

filtering

approach

-LSB-

-RSB-

based

on

overlapping

membership

of

pairs

of

communities

We

did

not

make

use

of

semantic

information

such

as

the

description

of

or

messages

in

a

community

-LRB-

although

this

may

be

an

area

of

future

work

-RRB-

Our

recommendations

were

on

a

per-community

rather

than

a

per-user

basis

that

is

all

members

of

a

given

community

would

see

the

same

recommendations

when

visiting

that

community

's

page

We

chose

this

approach

out

of

the

belief

which

was

confirmed

that

community

memberships

were

rich

enough

to

make

very

useful

recommendations

without

having

to

perform

more

compu-tationally

intensive

operations

such

as

clustering

of

users

or

communities

or

computing

nearest

neighbor

relations

among

users

Indeed

Sarwar

et

al.

have

found

such

item-based

algorithms

to

be

both

more

efficient

and

successful

than

user-based

algorithms

-LSB-

-RSB-

By

measuring

user

acceptance

of

recommendations

we

were

able

to

evaluate

the

absolute

and

relative

utility

of

six

different

similarity

measures

on

a

large

volume

of

data

The

input

data

came

from

the

membership

relation

M

-LCB-

-LRB-

u

c

-RRB-

u

U

c

C

-RCB-

where

C

is

the

set

of

communities

with

at

least

members

and

U

the

set

of

users

belonging

to

at

least

one

such

community

When

we

began

our

experiment

in

May

C

U

and

M

Table

summarizes

the

distribution

All

of

our

measures

of

community

similarity

involve

the

overlap

between

two

communities

i.e.

the

number

of

com-Table

Distribution

of

community

memberships

min

max

median

Users

per

community

Communities

per

user

Research

Track

Poster

mon

users

If

a

base

community

b

and

a

-LRB-

potentially

-RRB-

related

community

r

are

considered

as

sets

of

users

the

overlap

is

B

R

where

we

use

capital

letters

to

represent

the

set

containing

a

community

's

members

Note

that

overlap

can

not

be

the

sole

factor

in

relatedness

as

the

size

of

communities

varies

greatly

If

we

only

considered

overlap

practically

every

community

would

be

considered

related

to

the

Linux

community

which

was

the

most

popular

with

members

The

similarity

measures

in

the

next

section

normalize

the

overlap

in

different

ways

Similarity

Measure

Functions

Each

similarity

measure

we

consider

is

presented

as

a

-LRB-

possibly

asymmetric

-RRB-

function

of

b

and

r

indicating

how

appropriate

the

related

community

r

is

as

a

recommendation

for

the

base

community

b

We

do

not

use

the

result

of

the

function

as

an

absolute

measure

of

similarity

only

to

rank

recommendations

for

a

given

base

community

If

we

consider

the

base

and

related

communities

to

be

vectors

b

and

r

where

the

i

th

element

of

a

vector

is

if

user

i

is

a

member

and

if

not

we

can

measure

the

overlap

as

the

product

of

their

-LRB-

b

r

-RRB-

b

r

b

r

This

quantity

can

also

be

expressed

in

set

notation

where

we

use

a

capital

letter

to

represent

the

set

containing

a

community

's

members

-LRB-

B

R

-RRB-

B

R

B

R

Note

that

this

evaluates

to

the

overlap

between

the

two

groups

divided

by

the

product

of

their

sizes

When

the

base

community

is

held

constant

-LRB-

as

when

we

determine

the

base

community

's

recommendations

-RRB-

this

evaluates

to

the

overlap

divided

by

the

size

of

the

related

community

favoring

small

communities

Kitts

et

al.

-LSB-

-RSB-

reported

this

to

be

a

successful

measure

of

similarity

in

their

recommender

system

Similarly

we

can

measure

the

overlap

with

the

product

of

the

-LRB-

cosine

distance

-LSB-

-RSB-

-RRB-

of

b

and

r

-LRB-

b

r

-RRB-

b

r

b

r

In

set

notation

-LRB-

B

R

-RRB-

B

R

B

R

Note

that

the

square-root

in

the

denominator

causes

to

penalize

large

communities

less

severely

than

Observe

that

the

presented

here

is

equivalent

to

the

widely

used

cosine

coefficient

applied

to

binary

data

Moreover

while

Pearson

correlation

has

been

used

previ-ously

in

recommender

systems

where

ranking

data

is

available

we

did

not

use

this

measure

here

since

it

is

generally

considered

inappropriate

for

binary

data

Pointwise

Mutual-Information

positive

correlations

-LRB-

-RRB-

Information

theory

motivates

other

measures

of

correlation

such

as

mutual

information

-LSB-

-RSB-

We

chose

pointwise

mutual

information

where

we

only

count

positive

correlations

-LRB-

membership

in

both

B

and

R

-RRB-

Such

a

formulation

essentially

focuses

on

how

membership

in

one

group

is

pre-dictive

of

membership

in

another

-LRB-

without

considering

how

base

non-membership

in

a

group

effects

membership

in

another

group

-RRB-

yielding

M

-LRB-

b

r

-RRB-

P

-LRB-

r

b

-RRB-

lg

P

-LRB-

r

b

-RRB-

P

-LRB-

r

-RRB-

P

-LRB-

b

-RRB-

Pointwise

Mutual-Information

positive

and

negative

correlations

-LRB-

-RRB-

Similarly

we

can

compute

the

pointwise

mutual

information

with

both

positive

and

negative

correlations

-LRB-

e.g.

membership

in

both

B

and

R

or

non-membership

in

both

groups

-RRB-

Again

we

do

n't

compute

the

full

expected

mutual

information

since

we

believe

cross-correlations

-LRB-

e.g.

how

membership

in

B

affects

non-membership

in

R

-RRB-

tend

to

be

distortive

with

the

recommendation

task

since

such

cross-correlations

are

plentiful

but

not

very

informative

This

yields

M

-LRB-

b

r

-RRB-

P

-LRB-

r

b

-RRB-

lg

P

-LRB-

r

b

-RRB-

P

-LRB-

r

-RRB-

P

-LRB-

b

-RRB-

P

-LRB-

r

b

-RRB-

lg

P

-LRB-

r

b

-RRB-

P

-LRB-

r

-RRB-

P

-LRB-

b

-RRB-

Salton

-LRB-

IDF

-RRB-

Salton

proposed

a

measure

of

similarity

based

on

inverse

document

frequency

scaling

-LRB-

tf-idf

-RRB-

-LSB-

-RSB-

IDF

-LRB-

b

r

-RRB-

P

-LRB-

r

b

-RRB-

-LRB-

lg

P

-LRB-

r

-RRB-

-RRB-

IDF

-LRB-

B

R

-RRB-

B

R

B

-LRB-

lg

R

U

-RRB-

Log-Odds

We

first

considered

the

standard

log-odds

function

which

measures

the

relative

likelihood

that

presence

or

absence

in

a

base

community

predicts

membership

in

a

related

community

-LRB-

b

r

-RRB-

lg

P

-LRB-

r

b

-RRB-

P

-LRB-

r

b

-RRB-

Empirically

we

found

this

generated

the

exact

same

rankings

as

using

the

which

makes

sense

because

Logarithm

is

monotonic

and

while

affecting

scores

does

not

affect

rankings

Constant

factors

such

as

B

do

not

affect

rankings

For

B

U

P

-LRB-

r

b

-RRB-

P

-LRB-

r

-RRB-

We

formulated

a

different

log-odds

metric

which

measures

whether

membership

in

the

base

community

is

likelier

to

predict

membership

or

absence

in

the

related

community

LogOdds

-LRB-

b

r

-RRB-

lg

P

-LRB-

r

b

-RRB-

P

-LRB-

r

b

-RRB-

Research

Track

Poster

Table

Average

size

of

top-ranked

community

for

each

measure

measure

Average

size

rank

rank

rank

IDF

LogOdds

Table

Agreement

in

top-ranked

results

between

measures

For

example

and

IDF

rank

the

same

related

community

first

for

of

base

communities

Correlations

greater

than

are

in

bold

IDF

LogOdds

Discussion

For

a

given

measure

we

refer

to

the

related

community

yielding

the

highest

value

to

be

the

top-ranked

related

community

relative

to

a

base

community

The

average

size

of

top-ranked

communities

for

each

measure

which

varies

greatly

is

shown

in

Table

Table

shows

how

often

two

functions

yield

the

same

top-ranking

result

Table

shows

the

top

recommendations

for

the

I

love

wine

community

Note

that

and

IDF

favor

very

large

communities

while

and

LogOdds

favor

small

communities

Note

that

in

addition

to

the

obvious

correlations

between

the

two

mutual-information

functions

-LRB-

-RRB-

there

is

a

very

strong

correlation

between

IDF

and

the

mutual-information

functions

-LRB-

-RRB-

Manipulation

of

the

formulas

for

and

IDF

shows

M

-LRB-

b

r

-RRB-

P

-LRB-

r

b

-RRB-

lg

P

-LRB-

r

b

-RRB-

P

-LRB-

r

-RRB-

P

-LRB-

b

-RRB-

P

-LRB-

r

b

-RRB-

P

-LRB-

b

-RRB-

lg

P

-LRB-

r

b

-RRB-

P

-LRB-

r

b

-RRB-

P

-LRB-

b

-RRB-

lg

P

-LRB-

r

-RRB-

P

-LRB-

r

b

-RRB-

P

-LRB-

b

-RRB-

lg

P

-LRB-

r

b

-RRB-

P

-LRB-

r

b

-RRB-

-LSB-

P

-LRB-

b

-RRB-

-RSB-

lg

P

-LRB-

r

-RRB-

P

-LRB-

r

b

-RRB-

-LSB-

P

-LRB-

b

-RRB-

lg

P

-LRB-

r

b

-RRB-

P

-LRB-

r

b

-RRB-

P

-LRB-

b

-RRB-

lg

P

-LRB-

r

-RRB-

-RSB-

P

-LRB-

r

b

-RRB-

lg

P

-LRB-

r

-RRB-

Substituting

IDF

-LRB-

b

r

-RRB-

P

-LRB-

r

b

-RRB-

lg

P

-LRB-

r

-RRB-

we

get

M

-LRB-

b

r

-RRB-

P

-LRB-

r

b

-RRB-

P

-LRB-

b

-RRB-

lg

P

-LRB-

r

b

-RRB-

P

-LRB-

b

-RRB-

lg

P

-LRB-

r

-RRB-

IDF

-LRB-

b

r

-RRB-

Since

for

virtually

all

communities

b

P

-LRB-

b

-RRB-

P

-LRB-

b

-RRB-

we

can

approximate

M

-LRB-

b

r

-RRB-

IDF

-LRB-

b

r

-RRB-

P

-LRB-

r

b

-RRB-

P

-LRB-

b

-RRB-

lg

P

-LRB-

r

-RRB-

Thus

yields

a

ranking

that

can

be

thought

of

as

starting

with

the

ranking

of

IDF

and

perturbing

the

score

of

each

element

in

the

ranking

by

P

-LRB-

r

b

-RRB-

P

-LRB-

b

-RRB-

lg

P

-LRB-

r

-RRB-

which

generally

is

not

great

enough

to

change

the

relative

ranking

of

the

top

scores

leading

to

and

IDF

often

giving

the

same

ranking

to

top-scoring

communities

-LRB-

Note

that

this

perturbation

quantity

is

given

only

to

explain

the

high

correlation

between

and

IDF

Statistically

it

is

meaningless

since

b

and

b

can

not

simultaneously

hold

-RRB-

We

designed

an

experiment

to

determine

the

relative

value

of

the

recommendations

produced

by

each

similarity

measure

This

involved

interleaving

different

pairs

of

recommendations

and

tracking

user

clicks

Specifically

we

measured

the

efficacy

of

different

similarity

measures

using

pair-wise

binomial

sign

tests

on

click-through

data

rather

than

using

traditional

supervised

learning

measures

such

as

preci-sion/recall

or

accuracy

since

there

is

no

true

labeled

data

for

this

task

-LRB-

i.e.

we

do

not

know

what

are

the

correct

communities

that

should

be

recommended

to

a

user

-RRB-

Rather

we

focused

on

the

task

of

determining

which

of

the

similarity

measures

performs

best

on

a

relative

performance

scale

with

regard

to

acceptance

by

users

Combination

When

a

user

viewed

a

community

page

we

hashed

the

combined

user

and

community

identifiers

to

one

of

values

specifying

an

ordered

pair

of

similarity

measures

to

compare

Let

S

and

T

be

the

ordered

lists

of

recommendations

for

the

two

measures

where

S

-LRB-

s

s

s

S

-RRB-

and

T

-LRB-

t

t

t

T

-RRB-

and

S

T

The

recommendations

of

each

measure

are

combined

by

Joachims

Combined

Rank-ing

algorithm

-LSB-

-RSB-

restated

in

Figure

The

resulting

list

is

guaranteed

to

contain

the

top

k

S

and

k

T

recommendations

for

each

measure

where

k

T

k

S

k

T

-LSB-

Theorem

-RSB-

Measurements

Whenever

a

user

visited

a

community

two

measures

were

chosen

and

their

recommendations

interleaved

as

discussed

above

This

was

done

in

a

deterministic

manner

so

that

a

given

user

always

saw

the

same

recommendations

for

a

given

community

To

minimize

feedback

effects

we

did

not

regenerate

recommendations

after

the

experiment

began

A

user

who

views

a

base

community

-LRB-

e.g.

I

love

wine

-RRB-

is

either

a

member

-LRB-

denoted

by

M

-RRB-

or

non-member

-LRB-

denoted

by

n

-RRB-

-LRB-

We

capitalize

M

but

not

n

to

make

them

eas-ier

to

visually

distinguish

-RRB-

In

either

case

recommendations

are

shown

When

a

user

clicks

on

a

recommendation

there

are

three

possibilities

-LRB-

-RRB-

the

user

is

already

a

member

of

the

recommended

community

-LRB-

M

-RRB-

-LRB-

-RRB-

the

user

joins

the

recommended

community

-LRB-

j

-RRB-

or

-LRB-

-RRB-

the

user

visits

but

does

not

join

the

recommended

community

-LRB-

n

-RRB-

The

combination

of

base

and

related

community

memberships

can

be

combined

in

six

different

ways

For

example

Mj

denotes

a

click

where

a

member

of

the

base

community

clicks

on

a

recommendation

to

another

community

to

which

she

does

not

belong

and

joins

that

community

Traditionally

analyses

of

recommender

systems

focus

on

Mj

also

known

informally

as

if

you

like

this

you

'll

like

that

or

formally

as

similarity

or

conversion

Mn

recommendations

are

considered

distracters

having

negative

utility

since

they

waste

a

user

's

time

with

an

item

not

of

interest

Before

running

the

experiment

we

decided

that

the

measures

should

be

judged

on

their

Mj

performance

Other

interpretations

are

possible

Mn

links

could

be

considered

to

have

positive

utility

for

any

of

the

following

Research

Track

Poster

Table

Top

recommendations

for

each

measure

for

the

I

love

wine

community

with

each

recommended

community

's

overlap

with

the

base

community

and

size

The

size

of

I

love

wine

is

IDF

LogOdds

Ice

Wine

-LRB-

Eiswein

-RRB-

-LRB-

-RRB-

Red

Wine

-LRB-

-RRB-

Japanese

Food/Sushi

Lovers

-LRB-

-RRB-

Japanese

Food/Sushi

Lovers

-LRB-

-RRB-

Japanese

Food/Sushi

Lovers

-LRB-

-RRB-

Japanese

Food/Sushi

Lovers

-LRB-

-RRB-

California

Pinot

Noir

-LRB-

-RRB-

Cheeses

of

the

World

-LRB-

-RRB-

Red

Wine

-LRB-

-RRB-

Red

Wine

-LRB-

-RRB-

Photography

-LRB-

-RRB-

Photography

-LRB-

-RRB-

Winery

Visitor

Worldwide

-LRB-

-RRB-

I

love

red

wine

-LRB-

-RRB-

Cheeses

of

the

World

-LRB-

-RRB-

Cheeses

of

the

World

-LRB-

-RRB-

Red

Wine

-LRB-

-RRB-

Linux

-LRB-

-RRB-

Figure

Joachims

Combine

Rankings

algorithm

-LSB-

-RSB-

Input

ordered

recommendation

lists

S

-LRB-

s

s

s

S

-RRB-

and

T

-LRB-

t

t

t

T

-RRB-

where

S

T

Call

combine

-LRB-

S

T

-RRB-

Output

combined

ordered

recommendation

list

D

combine

-LRB-

S

T

k

s

k

t

D

-RRB-

-LCB-

if

-LRB-

k

s

S

k

t

T

-RRB-

if

-LRB-

k

s

k

t

-RRB-

-LCB-

if

-LRB-

S

-LSB-

k

s

-RSB-

D

-RRB-

-LCB-

D

D

S

-LSB-

k

s

-RSB-

-RCB-

combine

-LRB-

S

T

k

s

k

t

D

-RRB-

-RCB-

else

-LCB-

if

-LRB-

T

-LSB-

k

t

-RSB-

D

-RRB-

-LCB-

D

D

T

-LSB-

k

t

-RSB-

-RCB-

combine

-LRB-

S

T

k

s

k

t

D

-RRB-

-RCB-

-RCB-

-RCB-

Table

Clicks

on

recommendations

by

membership

status

in

the

base

and

recommended

communities

as

counts

and

as

percentages

of

total

clicks

The

last

column

shows

the

conversion

rate

defined

as

the

percentage

of

non-members

clicking

on

a

related

community

who

then

joined

it

-LRB-

j

n

j

-RRB-

membership

in

base

community

membership

in

recommended

community

M

-LRB-

member

-RRB-

n

-LRB-

non-member

-RRB-

j

-LRB-

join

-RRB-

total

conversion

rate

M

-LRB-

member

-RRB-

number

of

clicks

percent

of

total

clicks

n

-LRB-

non-member

-RRB-

number

of

clicks

percent

of

total

clicks

total

number

of

clicks

percent

of

total

clicks

Research

Track

Poster

reasons

As

the

user

found

the

link

sufficiently

interesting

to

click

on

it

was

of

more

utility

than

a

link

not

eliciting

a

click

The

user

is

genuinely

interested

in

the

related

community

but

does

not

want

to

proclaim

her

interest

as

membership

information

is

public

and

some

communities

focus

on

taboo

or

embarrassing

topics

For

example

a

recommendation

given

for

the

popular

Choco-late

community

is

PMS

Note

that

this

effect

is

specific

to

social

networks

and

not

for

example

Usenet

groups

where

the

user

's

list

of

communities

is

not

revealed

to

other

users

Similarly

it

is

unclear

how

to

value

clicks

from

a

base

community

that

the

user

does

not

belong

to

Does

an

nj

click

indicate

failure

since

the

base

community

was

not

joined

by

the

user

but

the

recommended

community

was

indicating

a

degree

of

dissimilarity

Or

is

it

of

positive

utility

since

it

helped

a

user

find

a

community

of

interest

For

these

reasons

we

tracked

all

clicks

recording

the

user

's

membership

status

in

the

base

and

recommended

communities

for

later

analysis

-LRB-

We

did

not

track

whether

users

returned

to

communities

in

the

future

because

of

the

logging

overhead

that

would

be

required

-RRB-

User

Interface

On

community

pages

our

recommendations

were

provided

in

a

table

each

cell

of

which

contained

a

recommended

community

's

name

optional

picture

and

link

-LRB-

Figure

-RRB-

Recommendations

were

shown

by

decreasing

rank

from

left

to

right

top

to

bottom

in

up

to

rows

of

For

aesthetic

reasons

we

only

showed

entire

rows

thus

no

recommendations

were

displayed

if

there

were

fewer

than

We

also

provided

a

control

that

allowed

users

to

send

us

comments

on

the

recommendations

We

analyzed

all

accesses

between

July

to

July

of

users

who

joined

Orkut

during

that

period

The

system

served

community

pages

with

recommendations

which

provides

a

lower

bound

on

the

number

of

views

-LRB-

Unfortunately

we

could

not

determine

the

total

number

of

views

due

to

browser

caching

-RRB-

There

were

clicks

on

recommendations

by

members

of

the

base

community

by

non-members

-LRB-

Table

-RRB-

Clicks

to

related

communities

to

which

the

user

already

belonged

were

rare

accounting

for

only

of

clicks

The

most

common

case

was

for

a

non-member

of

a

base

community

to

click

through

and

not

join

a

related

community

-LRB-

-RRB-

We

defined

conversion

rate

-LRB-

also

called

precision

-RRB-

as

the

percentage

of

non-members

who

clicked

through

to

a

community

who

then

joined

it

The

conversion

rate

was

three

times

as

high

-LRB-

-RRB-

when

the

member

belonged

to

the

base

community

-LRB-

from

which

the

recommendation

came

-RRB-

than

not

-LRB-

-RRB-

Relative

performance

of

different

measures

We

compared

each

measure

pairwise

against

every

other

measure

by

analyzing

clicks

of

their

merged

recommendations

If

the

click

was

on

a

recommendation

ranked

higher

by

measure

than

measure

for

example

we

considered

it

a

win

for

and

a

loss

for

If

both

measures

ranked

it

equally

the

result

was

considered

to

be

a

tie

Table

shows

the

outcomes

of

all

clicks

with

conversions

by

members

-LRB-

Mj

-RRB-

and

non-members

of

the

base

community

-LRB-

nj

-RRB-

broken

out

separately

We

say

that

a

measure

dominates

another

if

in

their

pairwise

comparison

the

former

has

more

wins

For

example

dominates

This

definition

combined

with

the

data

in

Table

yielded

a

total

order

-LRB-

to

our

surprise

-RRB-

among

the

measures

IDF

LogOdds

The

same

total

order

occurred

if

only

nj

clicks

were

considered

The

order

was

different

if

all

clicks

were

considered

IDF

LogOdds

Conversion

rates

There

was

great

variance

in

conversion

rate

by

recommended

community

We

examined

the

recommended

communities

that

were

clicked

through

to

more

than

times

Unsurprisingly

the

ten

with

the

lowest

conversion

rate

all

were

about

sex

-LRB-

e.g.

Amateur

Porn

-RRB-

Note

that

members

of

the

base

community

were

far

more

willing

than

non-members

to

join

perhaps

because

they

had

already

shown

their

willingness

to

join

a

sex-related

community

At

the

other

extreme

of

the

ten

with

the

highest

conversion

rate

were

sexual

-LRB-

e.g.

Flamenco

-RRB-

Table

provides

selected

data

by

each

membership

combination

Unsurprisingly

for

all

base

communities

members

were

more

likely

than

non-members

to

join

the

recommended

community

User

comments

Users

were

also

able

to

submit

feedback

on

related

communities

Most

of

the

feedback

was

from

users

who

wanted

recommendations

added

or

removed

Some

complained

about

inappropriate

recommendations

of

sexual

or

political

communities

especially

if

they

found

the

displayed

image

offensive

A

few

objected

to

our

generating

related

community

recommendations

at

all

instead

of

allowing

community

creators

to

specify

them

In

one

case

poor

recommendations

destroyed

a

community

The

creator

of

a

feminist

sexuality

community

disbanded

it

both

because

of

the

prurient

recommendations

that

appeared

on

her

page

and

the

disruptive

new

members

who

joined

as

a

result

of

recommendations

from

such

communities

We

agreed

with

her

that

the

recommendations

were

problematic

and

offered

to

remove

them

While

anecdotal

this

example

illustrates

how

a

recommendation

can

have

unanticipated

consequences

that

can

not

be

captured

in

simple

statistical

measures

-LRB-

An

informal

discussion

of

users

behavior

when

we

allowed

them

to

choose

related

communities

can

be

found

elsewhere

-LSB-

-RSB-

-RRB-

During

the

above

experiment

we

became

curious

how

the

relative

placement

of

recommendations

affected

users

selections

and

performed

a

second

experiment

Design

After

determining

that

was

the

best

measure

of

similarity

we

recomputed

the

recommendations

and

studied

the

effect

of

position

on

click-through

While

in

our

original

experiment

we

displayed

up

to

recommendations

in

decreasing

rank

for

this

experiment

we

displayed

up

to

recommendations

in

random

order

again

ensuring

that

each

Research

Track

Poster

Table

The

relative

performance

of

each

measure

in

pairwise

combination

on

clicks

leading

to

joins

divided

by

base

community

membership

status

and

on

all

clicks

Except

where

numbers

appear

in

italics

the

superioriority

of

one

measure

over

another

was

statistically

significant

-LRB-

p

-RRB-

using

a

binomial

sign

test

-LSB-

-RSB-

measures

M

j

n

j

all

clicks

win

equal

loss

win

equal

loss

win

equal

loss

IDF

LogOdds

IDF

LogOdds

IDF

LogOdds

IDF

IDF

LogOdds

LogOdds

Table

Conversion

rates

by

status

of

membership

in

base

community

for

communities

to

which

more

than

clicks

on

recommendations

occurred

member

of

base

community

non-member

of

base

community

Related

community

MM

Mj

Mj

conversion

rate

nM

nn

nM

conversion

rate

communities

with

highest

conversion

rates

communities

with

lowest

conversion

rates

all

communities

user

always

saw

the

same

ordering

of

recommendations

for

a

given

community

By

randomizing

the

position

of

recommendations

we

sought

to

measure

ordering

primacy

effects

in

the

recommendations

as

opposed

to

their

ranked

quality

Results

We

measured

all

clicks

on

related

community

recommendations

from

September

through

October

Table

shows

the

relative

likelihood

of

clicks

on

each

position

When

there

was

only

a

single

row

the

middle

recommendation

was

clicked

most

followed

by

the

leftmost

then

rightmost

recommendations

although

the

differences

were

not

statistically

significant

When

there

were

two

or

three

rows

the

differences

were

very

significant

-LRB-

p

-RRB-

with

preferences

for

higher

rows

P-values

were

computed

using

a

Chi-Squared

test

comparing

the

observed

click-through

rates

with

a

uniform

distribution

over

all

positions

-LSB-

-RSB-

Orkut

's

large

number

of

community

memberships

and

users

allowed

us

to

evaluate

the

relative

performance

of

six

different

measures

of

similarity

in

a

large-scale

real-world

study

We

are

not

aware

of

any

comparable

published

large-scale

experiments

We

were

surprised

that

a

total

order

emerged

among

the

similarity

measures

and

that

vector

normalization

showed

the

best

empirical

results

despite

other

measures

such

as

log-odds

and

pointwise

mutual

information

which

we

found

more

intuitive

For

future

work

we

would

like

to

see

how

recommendations

handpicked

by

community

owners

compare

Just

as

we

can

estimate

communities

similarity

through

common

users

we

can

estimate

users

similarity

through

common

community

memberships

i.e.

user

A

might

be

similar

to

user

B

because

they

belong

to

n

of

the

same

communities

It

will

be

interesting

to

see

whether

also

proves

superior

in

such

a

domain

We

could

also

take

advantage

of

Orkut

's

being

a

social

network

-LSB-

-RSB-

i.e.

containing

information

on

social

connections

between

pairs

of

users

In

addition

to

considering

common

community

memberships

we

could

consider

distance

between

users

in

the

friendship

graph

Users

close

to

each

other

-LRB-

e.g.

friends

or

friends-of

friends

-RRB-

might

be

judged

more

likely

to

be

similar

than

distant

strangers

although

some

users

might

prefer

the

latter

type

of

link

since

it

would

introduce

them

to

someone

they

would

be

unlikely

to

meet

otherwise

perhaps

from

a

different

country

or

culture

Similarly

friendship

graph

information

can

be

taken

into

account

when

making

community

recommendations

which

would

require

that

recommendations

be

computed

on

a

per-user

-LRB-

or

per-clique

-RRB-

rather

than

per-community

basis

In

such

a

setting

we

could

make

community

recommendations

based

on

weighted

community

overlap

vectors

where

weights

are

determined

based

on

the

graph

distances

of

other

community

members

to

a

given

user

This

is

a

fertile

area

for

future

work

and

yet

another

example

of

how

the

interaction

Research

Track

Poster

Figure

Displays

of

recommendations

for

three

different

communities

Table

The

relative

likelihood

of

clicks

on

link

by

position

when

there

are

-LRB-

a

-RRB-

one

-LRB-

b

-RRB-

two

or

-LRB-

c

-RRB-

three

rows

of

three

recommendations

-LRB-

a

-RRB-

n

p

-LRB-

b

-RRB-

n

p

-LRB-

c

-RRB-

n

p

of

data

mining

and

social

networks

is

becoming

an

exciting

new

research

area

-LSB-

-RSB-

-LSB-

-RSB-

This

work

was

performed

while

Ellen

Spertus

was

on

sabbatical

from

Mills

College

and

a

visiting

scientist

at

Google

We

are

grateful

to

Patrick

Barry

Alex

Drobychev

Jen

Fitz-patrick

Julie

Jalalpour

Dave

Jeske

Katherine

Lew

Marissa

Mayer

Tom

Nielsen

Seva

Petrov

Greg

Reshko

Catherine

Rondeau

Adam

Sawyer

Eric

Sachs

and

Lauren

Simpson

for

their

help

on

the

Orkut

project

and

to

Corey

Anderson

Alex

Drobychev

Oren

Etzioni

Alan

Eustace

Keith

Golden

Carrie

Grimes

Pedram

Keyani

John

Lamping

Tom

Nielsen

Peter

Norvig

Kerry

Rodden

Gavin

Tachibana

and

Yonatan

Zunger

for

their

help

on

this

research

or

its

exposition

-LSB-

-RSB-

Breese

J.

Heckerman

D.

Kadie

C.

Empirical

Analysis

of

Predictive

Algorithms

for

Collaborative

Filtering

In

Proceedings

of

the

Fourteenth

Conference

on

Uncertainty

in

Artificial

Intelligence

-LRB-

Madison

Wisconsin

-RRB-

Morgan

Kaufmann

-LSB-

-RSB-

Cover

T.M.

and

Thomas

J.A.

Elements

of

Information

Theory

Wiley

New

York

-LSB-

-RSB-

Deshpande

M.

and

Karypis

G.

Item-Based

Top-N

Recommendation

Algorithms

ACM

Transactions

on

Information

Systems

-LRB-

-RRB-

-LRB-

January

-RRB-

-LSB-

-RSB-

Domingos

P.

Prospects

and

Challenges

for

Multi-Relational

Data

Mining

ACM

SIGKDD

Exploration

Newsletter

-LRB-

-RRB-

-LRB-

July

-RRB-

-LSB-

-RSB-

Dumais

S.

Joachims

T.

Bharat

K.

Weigend

A.

SIGIR

Workshop

Report

Implicit

Measures

of

User

Interests

and

Preferences

SIGIR

Forum

-LRB-

-RRB-

-LRB-

Fall

-RRB-

-LSB-

-RSB-

Harman

D.

Ranking

Algorithms

In

W.

B.

Frakes

and

R.

Baeza-Yates

-LRB-

ed

-RRB-

Information

Retrieval

Data

Structures

Algorithms

-LRB-

chapter

-RRB-

Upper

Saddle

River

NJ

USA

Prentice

Hall

-LSB-

-RSB-

Joachims

T.

Evaluating

Retrieval

Performance

Using

Clickthrough

Data

In

Proceedings

of

the

SIGIR

Workshop

on

Mathematical/Formal

Methods

in

Information

Retrieval

-LRB-

-RRB-

ACM

Press

New

York

NY

-LSB-

-RSB-

Kautz

H.

Selman

Bart

Shah

M.

Referral

Web

Combining

Social

Networks

and

Collaborative

Filtering

Communications

of

the

ACM

-LRB-

-RRB-

-LRB-

March

-RRB-

-LSB-

-RSB-

Kitts

B.

Freed

D.

Vrieze

M.

Cross-Sell

A

Fast

Promotion-Tunable

Customer-Item

Recommendation

Method

based

on

Conditionally

Independent

Probabilities

In

Proceedings

of

the

Sixth

ACM

SIGKDD

International

Conference

on

Knowledge

Discovery

and

Data

Mining

-LRB-

Boston

-RRB-

ACM

Press

New

York

NY

-LSB-

-RSB-

Lehmann

E.L.

Testing

Statistical

Hypotheses

-LRB-

second

edition

-RRB-

Springer-Verlag

-LSB-

-RSB-

Raghavan

P.

Social

Networks

and

the

Web

-LRB-

Invited

Talk

-RRB-

In

Advances

in

Web

Intelligence

Proceedings

of

the

Second

International

Atlantic

Web

Intelligence

Conference

May

Springer-Verlag

Heidelberg

-LSB-

-RSB-

Salton

G.

Automatic

Text

Processing

The

Transformation

Analysis

and

Retrieval

of

Information

by

Computer

Addison

Wesley

Reading

MA

-LSB-

-RSB-

Sarwar

B.

Karypis

G.

Konstan

J.

Reidl

J.

Item-Based

Collaborative

Filtering

Recommendation

Algorithms

In

Proceedings

of

the

Tenth

International

Conference

on

the

World

Wide

Web

-LRB-

-RRB-

-LRB-

Hong

Kong

-RRB-

ACM

Press

New

York

NY

-LSB-

-RSB-

Spertus

Ellen

Too

Much

Information

Orkut

Media

Selections

January

Available

online

at

Research

Track

Poster

O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

online communities
social networks
social network
similarity measure
collaborative filtering
recommender system
community
data mining