A Similarity Measure for Motion Stream Segmentation and Recognition Recognition of motion streams such as data streams generated by different sign languages or various captured human body motions requires a high performance similarity measure . The motion streams have multiple attributes , and motion patterns in the streams can have different lengths from those of isolated motion patterns and different attributes can have different temporal shifts and variations . To address these issues , this paper proposes a similarity measure based on singular value decomposition -LRB- SVD -RRB- of motion matrices . Eigenvector differences weighed by the corresponding eigenvalues are considered for the proposed similarity measure . Experiments with general hand gestures and human motion streams show that the proposed similarity measure gives good performance for recognizing motion patterns in the motion streams in real time . H. 2.8 -LSB- Database Management -RSB- : Database Applications Data Mining Algorithm Motion streams can be generated by continuously performed sign language words -LSB- 14 -RSB- or captured human body motions such as various dances . Captured human motions can be applied to the movie and computer game industries by reconstructing various motions from video sequences -LSB- 10 -RSB- or images -LSB- 15 -RSB- or from motions captured by motion capture systems -LSB- 4 -RSB- . Recognizing motion patterns in the streams with unsupervised methods requires no training process , and is very convenient when new motions are expected to be added to the known pattern pools . A similarity measure with good performance is thus necessary for segmenting and recognizing the motion streams . Such a similarity measure needs to address some new challenges posed by real world Work supported partially by the National Science Foundation under Grant No. 0237954 for the project CAREER : Animation Databases . Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page . To copy otherwise , to republish , to post on servers or to redistribute to lists , requires prior specific permission and/or a fee . Copyright 200X ACM X-XXXXX-XX-X / XX/XX ... $ 5.00 . motion streams : first , the motion patterns have dozens of attributes , and similar patterns can have different lengths due to different motion durations ; second , different attributes of similar motions have different variations and different temporal shifts due to motion variations ; and finally , motion streams are continuous , and there are no obvious `` pauses '' between neighboring motions in a stream . A good similarity measure not only needs to capture the similarity of complete motion patterns , but also needs to capture the differences between complete motion patterns and incomplete motion patterns or sub-patterns in order to segment a stream for motion recognition . As the main contribution of this paper , we propose a similarity measure to address the above issues . The proposed similarity measure is defined based on singular value decomposition of the motion matrices . The first few eigenvectors are compared for capturing the similarity of two matrices , and the inner products of the eigenvectors are given different weights for their different contributions . We propose to use only the eigenvalues corresponding to the involved eigenvectors of the two motion matrices as weights . This simple and intuitive weighing strategy gives the same importance to eigenvalues of the two matrices . We also show that the 95 % variance rule for choosing the number of eigenvectors -LSB- 13 -RSB- is not sufficient for recognizing both isolated patterns and motion streams . Our experiments demonstrate that at least the first 6 eigenvectors need to be considered for motion streams of either 22 attribute or 54 attributes , and the first 6 eigenvalues accounts for more than 99.5 % of the total variance in the motion matrices . Multi-attribute pattern similarity search , especially in continuous motion streams , has been widely studied for sign language recognition and for motion synthesis in computer animation . The recognition methods usually include template matching by distance measures and hidden Markov models -LRB- HMM -RRB- . Template matching by using similarity/distance measures has been employed for multi-attribute pattern recognition . Joint angles are extracted in -LSB- 11 -RSB- as features to represent different human body static poses for the Mahalanobis distance measure of two joint angle features . Similarly , momentum , kinetic energy and force are constructed in -LSB- 2 , 5 -RSB- as activity measure and prediction of gesture boundaries for various segments of the human body , and the Mahalanobis distance function of two composite features are solved by dynamic programming . 89 Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page . To copy otherwise , to republish , to post on servers or to redistribute to lists , requires prior specific permission and/or a fee . MDM/KDD 2005 Chicago , August 21 , Chicago , Illinois , USA Copyright 2005 ACM -- MDM 2005 - 1-59593-216-X ... $ 5.00 . Similarity measures are defined for multi-attribute data in -LSB- 6 , 12 , 16 -RSB- based on principal component analysis -LRB- PCA -RRB- . Inner products or angular differences of principal components -LRB- PCs -RRB- are considered for similarity measure definitions , with different weighted strategies for different PCs . Equal weights are considered for different combinations of PCs in -LSB- 6 -RSB- , giving different PCs equal contributions to the similarity measure . The similarity measure in -LSB- 12 -RSB- takes the minimum of two weighted sums of PC inner products , and the two sums are respectively weighted by different weights . A global weight vector is obtained by taking into account all available isolated motion patterns in -LSB- 16 -RSB- , and this weight vector is used for specifying different contributions from different PC inner products to the similarity measure Eros . The dominating first PC and a normalized eigenvalue vector are considered in -LSB- 7 , 8 -RSB- for pattern recognition . In contrast , this paper propose to consider the first few PCs , and the angular differences or inner products of different PCs are weighted by different weights which depends on the data variances along the corresponding PCs . The HMM technique has been widely used for sign language recognition , and different recognition rates have been reported for different sign languages and different feature selection approaches . Starner et al. -LSB- 14 -RSB- achieved 92 % and 98 % word accuracy respectively for two systems , the first of the systems used a camera mounted on a desk and the second one used a camera in a user 's cap for extracting features as the input of HMM . Similarly Liang and Ouhyoung -LSB- 9 -RSB- used HMM for postures , orientations and motion primitives as features extracted from continuous Taiwan sign language streams and an average 80.4 % recognition rate was achieved . In contrast , the approach proposed in this paper is an unsupervised approach , and no training as required for HMM recognizers is needed . The joint positional coordinates or joint angular values of a subject in motion can be represented by a matrix : the columns or attributes of the matrix are for different joints , and the rows or frames of the matrix are for different time instants . Similarity of two motions is the similarity of the resulting motion matrices , which have the same number of attributes or columns , and yet can have different number of rows due to different motion durations . To capture the similarity of two matrices of different lengths , we propose to apply singular value decomposition -LRB- SVD -RRB- to the motion matrices in order to capture the similarity of the matrix geometric structures . Hence we briefly present SVD and its associated properties below before proposing the similarity measure based on SVD in this section . 3.1 Singular Value Decomposition The geometric structure of a matrix can be revealed by the SVD of the matrix . As shown in -LSB- 3 -RSB- , any real m n matrix A can be decomposed into A = U V T , where U = -LSB- u 1 , u 2 , ... , u m -RSB- R m m and V = -LSB- v 1 , v 2 , ... , v n -RSB- R n n are two orthogonal matrices , and is a diagonal matrix with diagonal entries being the singular values of A : 1 2 ... min -LRB- m , n -RRB- 0 . Column vectors u i and v i are the i th left and right singular vectors of A , respectively . It can be shown that the right singular vectors of the sym-metric n n matrix M = A T A are identical to the corresponding right singular vectors of A , referred to as eigenvectors of M . The singular values of M , or eigenvalues of M , are squares of the corresponding singular values of A . The eigenvector with the largest eigenvalue gives the first principal component . The eigenvector with the second largest eigenvalue is the second principal component and so on . 3.2 Similarity Measure Since SVD exposes the geometric structure of a matrix , it can be used for capturing the similarity of two matrices . We can compute the SVD of M = A T A instead of computing the SVD of A to save computational time . The reasons are that the eigenvectors of M are identical to the corresponding right singular vectors of A , the eigenvalues of M are the squares of the corresponding singular values of A , and SVD takes O -LRB- n 3 -RRB- time for the n n M and takes O -LRB- mn 2 -RRB- time with a large constant for the m n A , and usually m > n. Ideally , if two motions are similar , their corresponding eigenvectors should be parallel to each other , and their corresponding eigenvalues should also be proportional to each other . This is because the eigenvectors are the corresponding principal components , and the eigenvalues reflect the variances of the matrix data along the corresponding principal components . But due to motion variations , all corresponding eigenvectors can not be parallel as shown in Figure 1 . The parallelness or angular differences of two eigenvectors u and v can be described by the absolute value of their inner products : | cos | = | u v | / -LRB- | u | | v | -RRB- = | u v | , where | u | = | v | = 1 . We consider the absolute value of the inner products because eigenvectors can have different signs as shown in -LSB- 8 -RSB- . Since eigenvalues are numerically related to the variances of the matrix data along the associated eigenvectors , the importance of the eigenvector parallelness can be described by the corresponding eigenvalues . Hence , eigenvalues are to be used to give different weights to different eigenvector pairs . Figure 2 shows that the first eigenvalues are the dominating components of all the eigenvalues , and other eigenvalues become smaller and smaller and approach zero . As the eigenvalues are close to zero , their corresponding eigenvectors can be very different even if two matrices are similar . Hence not all the eigenvectors need to be incorporated into the similarity measure . Since two matrices have two eigenvalues for the corresponding eigenvector pair , these two eigenvalues should have equal contributions or weights to the eigenvector parallelness . In addition , the similarity measure of two matrices should be independent to other matrices , hence only eigenvectors and eigenvalues of the two matrices should be considered . Based on the above discussions , we propose the following similarity measure for two matrices Q and P : -LRB- Q , P -RRB- = 1 2 k i = 1 -LRB- -LRB- i / n i = 1 i + i / n i = 1 i -RRB- | u i v i | -RRB- where i and i are the i th eigenvalues corresponding to the i th eigenvectors u i and v i of square matrices of Q and P , respectively , and 1 < k < n. Integer k determines how many eigenvectors are considered and it depends on the number of attributes n of motion matrices . Experiments with hand gesture motions -LRB- n = 22 -RRB- and human body motions -LRB- n = 90 2 4 6 8 10 12 14 16 18 20 22 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 Component of First Eigenvector Component Value of First Eigenvector Motion341 Motion342 2 4 6 8 10 12 14 16 18 20 22 -0.4 -0.2 0 0.2 0.4 0.6 Component Value of Second Eigenvector Component of Second Eigenvector Motion341 Motion342 Figure 1 : Eigenvectors of similar patterns . The first eigenvectors are similar to each other , while other eigenvectors , such as the second vectors shown in the bottom , can be quite different . 54 -RRB- in Section 4 show that k = 6 is large enough without loss of pattern recognition accuracy in streams . We refer to this non-metric similarity measure as k Weighted Angular Similarity -LRB- kWAS -RRB- , which captures the angular similarities of the first k corresponding eigenvector pairs weighted by the corresponding eigenvalues . It can be easily verified that the value of kWAS ranges over -LSB- 0,1 -RSB- . When all corresponding eigenvectors are normal to each other , the similarity measure will be zero , and when two matrices are identical , the similarity measure approaches the maximum value one if k approaches n. 3.3 Stream Segmentation Algorithm In order to recognize motion streams , we assume one motion in a stream has a minimum length l and a maximum length L . The following steps can be applied to incremen-tally segment a stream for motion recognition : 1 . SVD is applied to all isolated motion patterns P to obtain their eigenvectors and eigenvalues . Let be the incremented stream length for segmentation , and let L be the location for segmentation . Initially L = l. 2 . Starting from the beginning of the stream or the end of the previously recognized motion , segment the stream at location L. Compute the eigenvectors and eigenvalues of the motion segment Q. 3 . Compute kWAS between Q and all motion patterns P . Update max to be the highest similarity after the previous motion 's recognition . 4 . If L + < L , update L = L + and go to step 2 . Otherwise , the segment corresponding to max is recognized to be the motion pattern which gives the highest similarity max , update L = l starting from the end of the last recognized motion pattern and go to step 2 . 1 2 3 4 5 6 7 8 85 87 89 91 93 95 97 99 100 Number of Eigenvalues Accumulated Eigenvalue Percentage -LRB- % -RRB- CyberGlove Data MoCap Data Figure 2 : Accumulated eigenvalue percentages in total eigenvalues for CyberGlove data and captured human body motion data . There are 22 eigenvalues for the CyberGlove data and 54 eigenvalues for the captured motion data . The sum of the first 2 eigenvalues is more than 95 % of the corresponding total eigenvalues , and the sum of the first 6 eigenvalues is almost 100 % of the total eigenvalues . This section evaluates experimentally the performances of the similarity measure kWAS proposed in this paper . It has been shown in -LSB- 16 -RSB- that Eros -LSB- 16 -RSB- outperforms other similarity measures mentioned in Section 2 except MAS -LSB- 8 -RSB- . Hence in this section , we compare the performances of the proposed kWAS with Eros and MAS for recognizing similar isolated motion patterns and for segmenting and recognizing motion streams from hand gesture capturing CyberGlove and human body motion capture system . 4.1 Data Generation A similarity measure should be able to be used not only for recognizing isolated patterns with high accuracy , but also for recognizing patterns in continuous motions or motion streams . Recognizing motion streams is more challenging than recognizing isolated patterns . This is because many very similar motion segments or sub-patterns needs to be compared in order to find appropriate segmentation locations , and a similarity measure should capture the difference between a complete motion or pattern and its sub-patterns . Hence , both isolated motion patterns and motion streams were generated for evaluating the performance of kWAS . Two data sources are considered for data generation : a CyberGlove for capturing hand gestures and a Vicon motion capture system for capturing human body motions . 4.1.1 CyberGlove Data A CyberGlove is a fully instrumented data glove that provides 22 sensors for measuring hand joint angular values to capture motions of a hand , such as American Sign Language -LRB- ASL -RRB- words for hearing impaired . The data for a hand gesture contain 22 angular values for each time instant/frame , one value for a joint of one degree of freedom . The motion data are extracted at around 120 frames per second . Data matrices thus have 22 attributes for the CyberGlove motions . One hundred and ten different isolated motions were generated as motion patterns , and each motion was repeated for three times , resulting in 330 isolated hand gesture motions . Some motions have semantic meanings . For example , 91 the motion for BUS as shown in Table 1 is for the ASL sign `` bus '' . Yet for segmentation and recognition , we only require that each individual motion be different from others , and thus some motions are general motions , and do not have any particular semantic meanings , such as the THUMBUP motion in Table 1 . The following 18 motions shown in Table 1 were used to generate continuous motions or streams . Twenty four different motion streams were generated for segmentation and recognition purpose . There are 5 to 10 motions in a stream and 150 motions in total in 24 streams , with 6.25 motions in a stream on average . It should be noted that variable-length transitional noises occur between successive motions in the generated streams . Table 1 : Individual motions used for streams 35 60 70 80 90 BUS GOODBYE HALF IDIOM JAR JUICE KENNEL KNEE MILK TV SCISSOR SPREAD THUMBUP 4.1.2 Motion Capture Data The motion capture data come from various motions captured collectively by using 16 Vicon cameras and the Vicon iQ Workstation software . A dancer wears a suit of non-reflective material and 44 markers are attached to the body suit . After system calibration and subject calibration , global coordinates and rotation angles of 19 joints/segments can be obtained at about 120 frames per second for any motion . Similarity of patterns with global 3D positional data can be disguised by different locations , orientations or different paths of motion execution as illustrated in Figure 3 -LRB- a -RRB- . Since two patterns are similar to each other because of similar relative positions of corresponding body segments at corresponding time , and the relative positions of different segments are independent of locations or orientations of the body , we can transform the global position data into local position data as follows . Let X p , Y p , Z p be the global coordinates of one point on pelvis , the selected origin of the `` moving '' local coordinate system , and , , be the rotation angles of the pelvis segment relative to the global coordinate system axes , respectively . The translation matrix is T as follows : T = 1 0 0 0 0 1 0 0 0 0 1 0 - X p - Y p - Z p 1 The rotation matrix R = R x R y R z , where R x = 1 0 0 0 0 cos - sin 0 0 sin cos 0 0 0 0 1 R y = cos 0 sin 0 0 1 0 0 - sin 0 cos 0 0 0 0 1 0 50 100 150 200 250 300 350 400 450 -1500 -1000 -500 0 500 1000 1500 0 50 100 150 200 250 300 350 400 450 0 500 1000 1500 2000 Motion Capture Frames Global Coordinates of Joints -LRB- mm -RRB- Global Coordinates of Joints -LRB- mm -RRB- -LRB- a -RRB- 0 50 100 150 200 250 300 350 400 450 -1000 -500 0 500 1000 Transformed Coordinates of Joints -LRB- mm -RRB- 0 50 100 150 200 250 300 350 400 450 -1000 -500 0 500 1000 Motion Capture Frames Transformed Coordinates of Joints -LRB- mm -RRB- -LRB- b -RRB- Figure 3 : 3D motion capture data for similar motions executed at different locations and in different orientations : -LRB- a -RRB- before transformation ; -LRB- b -RRB- after transformation . R z = cos - sin 0 0 sin cos 0 0 0 0 1 0 0 0 0 1 Let X , Y , Z be the global coordinates of one point on any segments , and x , y , z be the corresponding transformed local coordinates . x , y and z can be computed as follows : -LSB- x y z 1 -RSB- = -LSB- X Y Z 1 -RSB- T R The transformed data are positions of different segments relative to a moving coordinate system with the origin at some fixed point of the body , for example the pelvis . The moving coordinate system is not necessarily aligned with the global system , and it can rotate with the body . So data transformation includes both translation and rotation , and the transformed data would be translation and rotation invariant as shown in Figure 3 -LRB- b -RRB- . The coordinates of the origin pelvis are not included , thus the transformed matrices have 54 columns . Sixty two isolated motions including Taiqi , Indian dances , and western dances were performed for generating motion capture data , and each motion was repeated 5 times , yielding 310 isolated human motions . Every repeated motion has a different location and different durations , and can face different orientations . Twenty three motion streams were generated for segmentation . There are 3 to 5 motions in a stream , and 93 motions in total in 23 streams , with 4.0 motions in a stream on average . 4.2 Performance of k WAS for Capturing Similarities and Segmenting Streams We first apply kWAS to isolated motion patterns to show that the proposed similarity measure kWAS can capture the similarities of isolated motion patterns . Then kWAS is applied to motion streams for segmenting streams and recognizing motion patterns in the streams . We experimented with different k values in order to find out the smallest k without loss of good performance . Figure 2 shows the accumulated eigenvalue percentages averaged on 330 hand gestures and 310 human motions , respectively . Although the first two eigenvalues account for 92 1 2 90 91 92 93 94 95 96 97 98 99 100 Number of Nearest Neighbors -LRB- Most Similar Patterns -RRB- Pattern Recognition Rate -LRB- % -RRB- kWAS -LRB- k = 22 -RRB- kWAS -LRB- k = 5 -RRB- kWAS -LRB- k = 3 -RRB- kWAS -LRB- k = 2 -RRB- MAS EROS Figure 4 : Recognition rate of similar CyberGlove motion patterns . When k is 3 , kWAS can find the most similar motions for about 99.7 % of 330 motions , and can find the second most similar motions for 97.5 % of the them . 1 2 3 4 95 95.5 96 96.5 97 97.5 98 98.5 99 99.5 100 Number of Nearest Neighbors -LRB- Most Similar Patterns -RRB- Pattern Recognition Rate -LRB- % -RRB- kWAS -LRB- k = 54 -RRB- kWAS -LRB- k = 5 -RRB- kWAS -LRB- k = 4 -RRB- kWAS -LRB- k = 3 -RRB- MAS EROS Figure 5 : Recognition rate of similar captured motion patterns . When k is 5 , by using kWAS , the most similar motions of all 310 motions can be found , and the second most similar motions of 99.8 % of the 310 motions can also be found . more than 95 % of the respective sums of all eigenvalues , considering only the first two eigenvectors for kWAS is not sufficient as shown in Figure 4 and Figure 5 . For CyberGlove data with 22 attributes , kWAS with k = 3 gives the same performance as kWAS with k = 22 , and for motion capture data with 54 attributes , kWAS with k = 5 gives the same performance as kWAS with k = 54 . Figure 4 and Figure 5 illustrate that kWAS can be used for finding similar motion patterns and outperforms MAS and Eros for both hand gesture and human body motion data . The steps in Section 3.3 are used for segmenting streams and recognizing motions in streams . The recognition accuracy as defined in -LSB- 14 -RSB- is used for motion stream recognition . The motion recognition accuracies are shown in Table 2 . For both CyberGlove motion and captured motion data , k = 6 is used for kWAS , which gives the same accuracy as k = 22 for CyberGlove data and k = 54 for motion capture data , respectively . Figure 6 shows the time taken for updating the candidate segment , including updating the matrix , computing the SVD of the updated matrix , and computing the similarities of the segment and all motion patterns . The code implemented in C++ was run on one 2.70 GHz Intel processor of a GenuineIntel Linux box . There are 22 attributes for the CyberGlove streams , and 54 attributes for the captured CyberGlove Streams Motion Capture Streams 0 2 4 6 8 10 12 14 16 18 20 Time -LRB- milliseconds -RRB- MAS kWAS -LRB- k = 6 -RRB- EROS Figure 6 : Computation time for stream segment update and similarity computation . Table 2 : Stream Pattern Recognition Accuracy -LRB- % -RRB- Similarity CyberGlove Motion Capture Measures Streams Streams Eros 68.7 78.5 MAS 93.3 78.5 kWAS -LRB- k = 6 -RRB- 94.0 94.6 motion streams . Hence updating captured motion segments takes longer than updating CyberGlove motion segments as shown in Figure 6 . The time required by kWAS is close to the time required by MAS , and is less than half of the time taken by using Eros . 4.3 Discussions k WAS captures the similarity of square matrices of two matrices P and Q , yet the temporal order of pattern execution is not revealed in the square matrices . As shown in -LSB- 7 -RSB- , two matrices with the identical row vectors in different orders have identical eigenvectors and identical eigenvalues . If different temporal orders of pattern execution yield patterns with different semantic meanings , we need to further consider the temporal execution order , which is not reflected in the eigenvectors and eigenvalues and has not been considered previously in -LSB- 6 , 12 , 16 -RSB- . Since the first eigenvectors are close or parallel for similar patterns , we can project pattern A onto its first eigenvector u 1 by Au 1 . Then similar patterns would have similar projections -LRB- called projection vectors hereafter -RRB- , showing similar temporal execution orders while the projection variations for each pattern can be maximized . The pattern projection vectors can be compared by computing their dynamic time warping -LRB- DTW -RRB- distances , for DTW can align sequences of different lengths and can be solved easily by dynamic programming -LSB- 1 -RSB- . Incorporating temporal order information into the similarity measure can be done as for MAS in -LSB- 7 -RSB- if motion temporal execution orders cause motion pattern ambiguity to kWAS . This paper has proposed a similarity measure kWAS for motion stream segmentation and motion pattern recognition . kWAS considers the first few k eigenvectors and computes their angular similarities/differences , and weighs contributions of different eigenvector pairs by their correspond-93 ing eigenvalues . Eigenvalues from two motion matrices are given equal importance to the weights . Experiments with CyberGlove hand gesture streams and captured human body motions such as Taiqi and dances show that kWAS can recognize 100 % most similar isolated patterns and can recognize 94 % motion patterns in continuous motion streams . -LSB- 1 -RSB- D. Berndt and J. Clifford . Using dynamic time warping to find patterns in time series . In AAAI-94 Workshop on Knowledge Discovery in Databases , pages 229 248 , 1994 . -LSB- 2 -RSB- V. M. Dyaberi , H. Sundaram , J. James , and G. Qian . Phrase structure detection in dance . In Proceedings of the ACM Multimedia Conference 2004 , pages 332 335 , Oct. 2004 . -LSB- 3 -RSB- G. H. Golub and C. F. V. Loan . Matrix Computations . The Johns Hopkins University Press , Baltimore , Maryland , 1996 . -LSB- 4 -RSB- L. Ikemoto and D. A. Forsyth . Enriching a motion collection by transplanting limbs . In Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation , pages 99 108 , 2004 . -LSB- 5 -RSB- K. Kahol , P. Tripathi , S. Panchanathan , and T. Rikakis . Gesture segmentation in complex motion sequences . In Proceedings of IEEE International Conference on Image Processing , pages II 105 108 , Sept. 2003 . -LSB- 6 -RSB- W. Krzanowski . Between-groups comparison of principal components . J. Amer . Stat . Assoc. , 74 -LRB- 367 -RRB- : 703 707 , 1979 . -LSB- 7 -RSB- C. Li , B. Prabhakaran , and S. Zheng . Similarity measure for multi-attribute data . In Proceedings of the 2005 IEEE International Conference on Acoustics , Speach , and Signal Processing -LRB- ICASSP -RRB- , Mar. 2005 . -LSB- 8 -RSB- C. Li , P. Zhai , S.-Q . Zheng , and B. Prabhakaran . Segmentation and recognition of multi-attribute motion sequences . In Proceedings of the ACM Multimedia Conference 2004 , pages 836 843 , Oct. 2004 . -LSB- 9 -RSB- R. H. Liang and M. Ouhyoung . A real-time continuous gesture recognition system for sign language . In Proceedings of the 3rd . International Conference on Face and Gesture Recognition , pages 558 565 , 1998 . -LSB- 10 -RSB- K. Pullen and C. Bregler . Motion capture assisted animation : texturing and synthesis . In SIGGRAPH , pages 501 508 , 2002 . -LSB- 11 -RSB- G. Qian , F. Guo , T. Ingalls , L. Olson , J. James , and T. Rikakis . A gesture-driven multimodal interactive dance system . In Proceedings of IEEE International Conference on Multimedia and Expo , June 2004 . -LSB- 12 -RSB- C. Shahabi and D. Yan . Real-time pattern isolation and recognition over immersive sensor data streams . In Proceedings of the 9th International Conference on Multi-Media Modeling , pages 93 113 , Jan 2003 . -LSB- 13 -RSB- A. Singhal and D. E. Seborg . Clustering of multivariate time-series data . In Proceedings of the American Control Conference , pages 3931 3936 , 2002 . -LSB- 14 -RSB- T. Starner , J. Weaver , and A. Pentland . Real-time american sign language recognition using desk and wearable computer based video . IEEE Transactions on Pattern Analysis and Machine Intelligence , 20 -LRB- 12 -RRB- : 1371 1375 , 1998 . -LSB- 15 -RSB- C. J. Taylor . Reconstruction of articulated objects from point correspondences in a single image . Computer Vision and Image Understanding , 80 -LRB- 3 -RRB- : 349 363 , 2000 . -LSB- 16 -RSB- K. Yang and C. Shahabi . A PCA-based similarity measure for multivariate time series . In Proceedings of the Second ACM International Workshop on Multimedia Databases , pages 65 74 , Nov. 2004 . 94

O B I O B I B O B B O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I I O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O O B I O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O B O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I B O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O

pattern recognition
segmentation
motion stream
gesture
data streams
singular value decomposition
similarity measure
recognition
eigenvalue
eigenvector


The Maximum Entropy Method for Analyzing Retrieval Measures We present a model , based on the maximum entropy method , for analyzing various measures of retrieval performance such as average precision , R-precision , and precision-at-cutoffs . Our methodology treats the value of such a measure as a constraint on the distribution of relevant documents in an unknown list , and the maximum entropy distribution can be determined subject to these constraints . For good measures of overall performance -LRB- such as average precision -RRB- , the resulting maximum entropy distributions are highly correlated with actual distributions of relevant documents in lists as demonstrated through TREC data ; for poor measures of overall performance , the correlation is weaker . As such , the maximum entropy method can be used to quantify the overall quality of a retrieval measure . Furthermore , for good measures of overall performance -LRB- such as average precision -RRB- , we show that the corresponding maximum entropy distributions can be used to accurately infer precision-recall curves and the values of other measures of performance , and we demonstrate that the quality of these inferences far exceeds that predicted by simple retrieval measure correlation , as demonstrated through TREC data . H. 3.4 -LSB- Information Storage and Retrieval -RSB- : Systems and Software Performance evaluation Theory , Measurement , Experimentation We gratefully acknowledge the support provided by NSF grant CCF-0418390 . Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page . To copy otherwise , to republish , to post on servers or to redistribute to lists , requires prior specific permission and/or a fee . SIGIR ' 05 , August 15 19 , 2005 , Salvador , Brazil . Copyright 2005 ACM 1-59593-034-5 / 05/0008 ... $ 5.00 . The efficacy of retrieval systems is evaluated by a number of performance measures such as average precision , R-precision , and precisions at standard cutoffs . Broadly speaking , these measures can be classified as either system-oriented measures of overall performance -LRB- e.g. , average precision and R-precision -RRB- or user-oriented measures of specific performance -LRB- e.g. , precision-at-cutoff 10 -RRB- -LSB- 3 , 12 , 5 -RSB- . Different measures evaluate different aspects of retrieval performance , and much thought and analysis has been devoted to analyzing the quality of various different performance measures -LSB- 10 , 2 , 17 -RSB- . We consider the problem of analyzing the quality of various measures of retrieval performance and propose a model based on the maximum entropy method for evaluating the quality of a performance measure . While measures such as average precision at relevant documents , R-precision , and 11 pt average precision are known to be good measures of overall performance , other measures such as precisions at specific cutoffs are not . Our goal in this work is to develop a model within which one can numerically assess the overall quality of a given measure based on the reduction in uncertainty of a system 's performance one gains by learning the value of the measure . As such , our evaluation model is primarily concerned with assessing the relative merits of system-oriented measures , but it can be applied to other classes of measures as well . We begin with the premise that the quality of a list of documents retrieved in response to a given query is strictly a function of the sequence of relevant and non-relevant documents retrieved within that list -LRB- as well as R , the total number of relevant documents for the given query -RRB- . Most standard measures of retrieval performance satisfy this premise . Our thesis is then that given the assessed value of a `` good '' overall measure of performance , one 's uncertainty about the sequence of relevant and non-relevant documents in an unknown list should be greatly reduced . Suppose , for example , one were told that a list of 1,000 documents retrieved in response to a query with 200 total relevant documents contained 100 relevant documents . What could one reasonably infer about the sequence of relevant and non-relevant documents in the unknown list ? From this information alone , one could only reasonably conclude that the likelihood of seeing a relevant document at any rank level is uniformly 1/10 . Now suppose that one were additionally told that the average precision of the list was 0.4 -LRB- the maximum possi-27 ble in this circumstance is 0.5 -RRB- . Now one could reasonably conclude that the likelihood of seeing relevant documents at low numerical ranks is much greater than the likelihood of seeing relevant documents at high numerical ranks . One 's uncertainty about the sequence of relevant and non-relevant documents in the unknown list is greatly reduced as a consequence of the strong constraint that such an average precision places on lists in this situation . Thus , average precision is highly informative . On the other hand , suppose that one were instead told that the precision of the documents in the rank range -LSB- 100 , 110 -RSB- was 0.4 . One 's uncertainty about the sequence of relevant and non-relevant documents in the unknown list is not appreciably reduced as a consequence of the relatively weak constraint that such a measurement places on lists . Thus , precision in the range -LSB- 100 , 110 -RSB- is not a highly informative measure . In what follows , we develop a model within which one can quantify how informative a measure is . We consider two questions : -LRB- 1 -RRB- What can reasonably be inferred about an unknown list given the value of a measurement taken over this list ? -LRB- 2 -RRB- How accurately do these inferences reflect reality ? We argue that the former question is properly answered by considering the maximum entropy distributions subject to the measured value as a constraint , and we demonstrate that such maximum entropy models corresponding to good overall measures of performance such as average precision yield accurate inferences about underlying lists seen in practice -LRB- as demonstrated through TREC data -RRB- . More specifically , we develop a framework based on the maximum entropy method which allows one to infer the most `` reasonable '' model for the sequence of relevant and non-relevant documents in a list given a measured constraint . From this model , we show how one can infer the most `` reasonable '' model for the unknown list 's entire precision-recall curve . We demonstrate through the use of TREC data that for `` good '' overall measures of performance -LRB- such as average precision -RRB- , these inferred precision-recall curves are accurate approximations of actual precision-recall curves ; however , for `` poor '' overall measures of performance , these inferred precision-recall curves do not accurately approximate actual precision-recall curves . Thus , maximum entropy modeling can be used to quantify the quality of a measure of overall performance . We further demonstrate through the use of TREC data that the maximum entropy models corresponding to `` good '' measures of overall performance can be used to make accurate predictions of other measurements . While it is well known that `` good '' overall measures such as average precision are well correlated with other measures of performance , and thus average precision could be used to reasonably predict other measures of performance , we demonstrate that the maximum entropy models corresponding to average precision yield inferences of other measures even more highly correlated with their actual values , thus validating both average precision and maximum entropy modeling . In the sections that follow , we first describe the maximum entropy method and discuss how maximum entropy modeling can be used to analyze measures of retrieval performance . We then describe the results of applying our methodology using TREC data , and we conclude with a summary and future work . The concept of entropy as a measure of information was first introduced by Shannon -LSB- 20 -RSB- , and the Principle of Maximum Entropy was introduced by Jaynes -LSB- 7 , 8 , 9 -RSB- . Since its introduction , the Maximum Entropy Method has been applied in many areas of science and technology -LSB- 21 -RSB- including natural language processing -LSB- 1 -RSB- , ambiguity resolution -LSB- 18 -RSB- , text classification -LSB- 14 -RSB- , machine learning -LSB- 15 , 16 -RSB- , and information retrieval -LSB- 6 , 11 -RSB- , to name but a few examples . In what follows , we introduce the maximum entropy method through a classic example , and we then describe how the maximum entropy method can be used to evaluate measures of retrieval performance . Suppose you are given an unknown and possibly biased six-sided die and were asked the probability of obtaining any particular die face in a given roll . What would your answer be ? This problem is under-constrained and the most seemingly `` reasonable '' answer is a uniform distribution over all faces . Suppose now you are also given the information that the average die roll is 3.5 . The most seemingly `` reasonable '' answer is still a uniform distribution . What if you are told that the average die roll is 4.5 ? There are many distributions over the faces such that the average die roll is 4.5 ; how can you find the most seemingly `` reasonable '' distribution ? Finally , what would your answer be if you were told that the average die roll is 5.5 ? Clearly , the belief in getting a 6 increases as the expected value of the die rolls increases . But there are many distributions satisfying this constraint ; which distribution would you choose ? The `` Maximum Entropy Method '' -LRB- MEM -RRB- dictates the most `` reasonable '' distribution satisfying the given constraints . The `` Principle of Maximal Ignorance '' forms the intuition behind the MEM ; it states that one should choose the distribution which is least predictable -LRB- most random -RRB- subject to the given constraints . Jaynes and others have derived numerous entropy concentration theorems which show that the vast majority of all empirical frequency distributions -LRB- e.g. , those corresponding to sequences of die rolls -RRB- satisfying the given constraints have associated empirical probabilities and entropies very close to those probabilities satisfying the constraints whose associated entropy is maximal -LSB- 7 -RSB- . Thus , the MEM dictates the most random distribution satisfying the given constraints , using the entropy of the probability distribution as a measure of randomness . The entropy of a probability distribution p = -LCB- p 1 , p 2 , ... , p n -RCB- is a measure of the uncertainty -LRB- randomness -RRB- inherent in the distribution and is defined as follows H -LRB- p -RRB- = n X i = 1 p i lg p i . Thus , maximum entropy distributions are probability distributions making no additional assumptions apart from the given constraints . In addition to its mathematical justification , the MEM tends to produce solutions one often sees in nature . For example , it is known that given the temperature of a gas , the actual distribution of velocities in the gas is the maximum entropy distribution under the temperature constraint . We can apply the MEM to our die problem as follows . Let the probability distribution over the die faces be p = -LCB- p 1 , ... , p 6 -RCB- . Mathematically , finding the maximum entropy distribution over die faces such that the expected die roll is 28 1 2 3 4 5 6 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 die face probability 1 2 3 4 5 6 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 die face probability 1 2 3 4 5 6 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 die face probability Figure 1 : Maximum entropy die distributions with mean die rolls of 3.5 , 4.5 , and 5.5 , respectively . d corresponds to the following optimization problem : Maximize : H -LRB- p -RRB- Subject to : 1 . 6 P i = 1 p i = 1 2 . 6 P i = 1 i p i = d The first constraint ensures that the solution forms a distribution over the die faces , and the second constraint ensures that this distribution has the appropriate expectation . This is a constrained optimization problem which can be solved using the method of Lagrange multipliers . Figure 1 shows three different maximum entropy distributions over the die faces such that the expected die roll is 3.5 , 4.5 , and 5.5 , respectively . 2.1 Application of the Maximum Entropy Method to Analyzing Retrieval Measures Suppose that you were given a list of length N corresponding to the output of a retrieval system for a given query , and suppose that you were asked to predict the probability of seeing any one of the 2 N possible patterns of relevant documents in that list . In the absence of any information about the query , any performance information for the system , or any a priori modeling of the behavior of retrieval systems , the most `` reasonable '' answer you could give would be that all lists of length N are equally likely . Suppose now that you are also given the information that the expected number of relevant documents over all lists of length N is R ret . Your `` reasonable '' answer might then be a uniform distribution over all ` N R ret different possible lists with R ret relevant documents . But what if apart from the constraint on the number of relevant documents retrieved , you were also given the constraint that the expected value of average precision is ap ? If the average precision value is high , then of all the ` N R ret lists with R ret relevant documents , the lists in which the relevant documents are retrieved at low numerical ranks should have higher probabilities . But how can you determine the most `` reasonable '' such distribution ? The maximum entropy method essentially dictates the most reasonable distribution as a solution to the following constrained optimization problem . Let p -LRB- r 1 , ... , r N -RRB- be a probability distribution over the relevances associated with document lists of length N , let rel -LRB- r 1 , ... , r N -RRB- be the number of relevant documents in a list , and let ap -LRB- r 1 , ... , r N -RRB- be the average precision of a list . Then the maximum entropy method can be mathematically formulated as follows : Maximize : H -LRB- p -RRB- Subject to : 1 . P r 1 , ... , r N p -LRB- r 1 , ... , r N -RRB- = 1 2 . P r 1 , ... , r N ap -LRB- r 1 , ... , r N -RRB- p -LRB- r 1 , ... , r N -RRB- = ap 3 . P r 1 , ... , r N rel -LRB- r 1 , ... , r N -RRB- p -LRB- r 1 , ... , r N -RRB- = R ret Note that the solution to this optimization problem is a distribution over possible lists , where this distribution effectively gives one 's a posteriori belief in any list given the measured constraint . The previous problem can be formulated in a slightly different manner yielding another interpretation of the problem and a mathematical solution . Suppose that you were given a list of length N corresponding to output of a retrieval system for a given a query , and suppose that you were asked to predict the probability of seeing a relevant document at some rank . Since there are no constraints , all possible lists of length N are equally likely , and hence the probability of seeing a relevant document at any rank is 1/2 . Suppose now that you are also given the information that the expected number of relevant documents over all lists of length N is R ret . The most natural answer would be a R ret / N uniform probability for each rank . Finally , suppose that you are given the additional constraint that the expected average precision is ap . Under the assumption that our distribution over lists is a product distribution -LRB- this is effectively a fairly standard independence assumption -RRB- , we may solve this problem as follows . Let p -LRB- r 1 , ... , r N -RRB- = p -LRB- r 1 -RRB- p -LRB- r 2 -RRB- p -LRB- r N -RRB- where p -LRB- r i -RRB- is the probability that the document at rank i is relevant . We can then solve the problem of calculating the probability of seeing a relevant document at any rank using the MEM . For notational convenience , we will refer to this product distribution as the probability-at-rank distribution and the probability of seeing a relevant document at rank i , p -LRB- r i -RRB- , as p i . Standard results from information theory -LSB- 4 -RSB- dictate that if p -LRB- r 1 , ... , r N -RRB- is a product distribution , then H -LRB- p -LRB- r 1 , ... , r N -RRB- -RRB- = N X i = 1 H -LRB- p i -RRB- where H -LRB- p i -RRB- is the binary entropy H -LRB- p i -RRB- = - p i lg p i - -LRB- 1 - p i -RRB- lg -LRB- 1 - p i -RRB- . Furthermore , it can be shown that given a product distribution p -LRB- r 1 , ... , r N -RRB- over the relevances associated with docu-29 Maximize : P N i = 1 H -LRB- p i -RRB- Subject to : 1 . 1 R N P i = 1 ` p i i ` 1 + i-1 P j = 1 p j = ap 2 . N P i = 1 p i = R ret Figure 2 : Maximum entropy setup for average precision . Maximize : P N i = 1 H -LRB- p i -RRB- Subject to : 1 . 1 R R P i = 1 p i = rp 2 . N P i = 1 p i = R ret Figure 3 : Maximum entropy setup for R-precision . Maximize : P N i = 1 H -LRB- p i -RRB- Subject to : 1 . 1 k k P i = 1 p i = PC -LRB- k -RRB- 2 . N P i = 1 p i = R ret Figure 4 : Maximum entropy setup for precision-at-cutoff . ment lists of length N , the expected value of average precision is 1 R N X i = 1 p i i 1 + i-1 X j = 1 p j !! . -LRB- 1 -RRB- -LRB- The derivation of this formula is omitted due to space constraints . -RRB- Furthermore , since p i is the probability of seeing a relevant document at rank i , the expected number of relevant documents retrieved until rank N is P N i = 1 p i . Now , if one were given some list of length N , one were told that the expected number of relevant documents is R ret , one were further informed that the expected average precision is ap , and one were asked the probability of seeing a relevant document at any rank under the independence assumption stated , one could apply the MEM as shown in Figure 2 . Note that one now solves for the maximum entropy product distribution over lists , which is equivalent to a maximum entropy probability-at-rank distribution . Applying the same ideas to R-precision and precision-at-cutoff k , one obtains analogous formulations as shown in Figures 3 and 4 , respectively . All of these formulations are constrained optimization problems , and the method of Lagrange multipliers can be used to find an analytical solution , in principle . When analytical solutions can not be determined , numerical optimization methods can be employed . The maximum entropy distributions for R-precision and precision-at-cutoff k can be obtained analytically using the method of Lagrange multipliers . However , numerical optimization methods are required to determine the maximum entropy distribution for average precision . In Figure 5 , examples of maximum entropy probability-at-rank curves corresponding to the measures average precision , R-precision , and precision-at-cutoff 10 for a run in TREC8 can be seen . Note that the probability-at - rank curves are step functions for the precision-at-cutoff and R-precision constraints ; this is as expected since , for example , given a precision-at-cutoff 10 of 0.3 , one can only reasonably conclude a uniform probability of 0.3 for seeing a relevant document at any of the first 10 ranks . Note , however , that the probability-at-rank curve corresponding to average precision is smooth and strictly decreasing . Using the maximum entropy probability-at-rank distribution of a list , we can infer the maximum entropy precision-recall curve for the list . Given a probability-at-rank distribution p , the number of relevant documents retrieved until rank i is REL -LRB- i -RRB- = P i j = 1 p j . Therefore , the precision and recall at rank i are PC -LRB- i -RRB- = REL -LRB- i -RRB- / i and REC -LRB- i -RRB- = REL -LRB- i -RRB- / R. Hence , using the maximum entropy probability-0 100 200 300 400 500 600 700 800 900 1000 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 rank probability TREC8 System fub99a Query 435 AP = 0.1433 ap maxent dist . rp maxent dist . pc-10 maxent dist . Figure 5 : Probability-at-rank distributions . at-rank distribution for each measure , we can generate the maximum entropy precision-recall curve of the list . If a measure provides a great deal of information about the underlying list , then the maximum entropy precision-recall curve should approximate the precision-recall curve of the actual list . However , if a measure is not particularly informative , then the maximum entropy precision-recall curve need not approximate the actual precision-recall curve . Therefore , noting how closely the maximum entropy precision-recall curve corresponding to a measure approximates the precision-recall curve of the actual list , we can calculate how much information a measure contains about the actual list , and hence how `` informative '' a measure is . Thus , we have a methodology for evaluating the evaluation measures themselves . Using the maximum entropy precision-recall curve of a measure , we can also predict the values of other measures . For example , using the maximum entropy precision-recall curve corresponding to average precision , we can predict the precision-at-cutoff 10 . For highly informative measures , these predictions should be very close to reality . Hence , we have a second way of evaluating evaluation measures . We tested the performance of the evaluation measures average precision , R-precision , and precision-at-cutoffs 5 , 10 , 15 , 20 , 30 , 100 , 200 , 500 and 1000 using data from TRECs 3 , 5 , 6 , 7 , 8 and 9 . For any TREC and any query , we chose those systems whose number of relevant documents retrieved was at least 10 in order to have a sufficient number of points on the precision-recall curve . We then calculated the maximum entropy precision-recall curve subject to the given measured constraint , as described above . The maximum entropy precision-recall curve corresponding to an average precision 30 constraint can not be determined analytically ; therefore , we used numerical optimization 1 to find the maximum entropy distribution corresponding to average precision . We shall refer to the execution of a retrieval system on a particular query as a run . Figure 6 shows examples of maximum entropy precision-recall curves corresponding to average precision , R-precision , and precision-at-cutoff 10 for three different runs , together with the actual precision-recall curves . We focused on these three measures since they are perhaps the most commonly cited measures in IR . We also provide results for precision-at-cutoff 100 in later plots and detailed results for all measures in a later table . As can be seen in Figure 6 , using average precision as a constraint , one can generate the actual precision-recall curve of a run with relatively high accuracy . In order to quantify how good an evaluation measure is in generating the precision-recall curve of an actual list , we consider two different error measures : the root mean squared error -LRB- RMS -RRB- and the mean absolute error -LRB- MAE -RRB- . Let -LCB- 1 , 2 , ... , R ret -RCB- be the precisions at the recall levels -LCB- 1/R , 2/R , ... , R ret / R -RCB- where R ret is the number of relevant documents retrieved by a system and R is the number of documents relevant to the query , and let -LCB- m 1 , m 2 , ... , m R ret -RCB- be the estimated precisions at the corresponding recall levels for a maximum entropy distribution corresponding to a measure . Then the MAE and RMS errors are calculated as follows . RMS = v u u t 1 R ret R ret X i = 1 -LRB- i - m i -RRB- 2 MAE = 1 R ret R ret X i = 1 | i - m i | The points after recall R ret / R on the precision-recall curve are not considered in the evaluation of the MAE and RMS errors since , by TREC convention , the precisions at these recall levels are assumed to be 0 . In order to evaluate how good a measure is at inferring actual precision-recall curves , we calculated the MAE and RMS errors of the maximum entropy precision-recall curves corresponding to the measures in question , averaged over all runs for each TREC . Figure 7 shows how the MAE and RMS errors for average precision , R-precision , precision-at-cutoff 10 , and precision-at-cutoff 100 compare with each other for each TREC . The MAE and RMS errors follow the same pattern over all TRECs . Both errors are consistently and significantly lower for average precision than for the other measures in question , while the errors for R-precision are consistently lower than for precision-at-cutoffs 10 and 100 . Table 1 shows the actual values of the RMS errors for all measures over all TRECs . In our experiments , MAE and RMS errors follow a very similar pattern , and we therefore omit MAE results due to space considerations . From this table , it can be seen that average precision has consistently lower RMS errors when compared to the other measures . The penultimate column of the table shows the average RMS errors per measure averaged over all TRECs . On average , R-precision has the second lowest RMS error after average precision , and precision-at-cutoff 30 is the third best measure in terms of RMS error . The last column of the table 1 We used the TOMLAB Optimization Environment for Matlab . shows the percent increase in the average RMS error of a measure when compared to the RMS error of average precision . As can be seen , the average RMS errors for the other measures are substantially greater than the average RMS error for average precision . We now consider a second method for evaluating how informative a measure is . A highly informative measure should properly reduce one 's uncertainty about the distribution of relevant and non-relevant documents in a list ; thus , in our maximum entropy formulation , the probability-at-rank distribution should closely correspond to the pattern of relevant and non-relevant documents present in the list . One should then be able to accurately predict the values of other measures from this probability-at-rank distribution . Given a probability-at-rank distribution p 1 , p 2 , ... , p N , we can predict average precision , R-precision and precision-at-cutoff k values as follows : ap = 1 R N X i = 1 p i i 1 + i-1 X j = 1 p j !! rp = 1 R R X i = 1 p i PC -LRB- k -RRB- = 1 k k X i = 1 p i The plots in the top row of Figures 8 and 9 show how average precision is actually correlated with R-precision , precision-at - cutoff 10 , and precision-at-cutoff 100 for TRECs 6 and 8 , respectively . Each point in the plot corresponds to a system and the values of the measures are averaged over all queries . Using these plots as a baseline for comparison , the plots in the bottom row of the figures show the correlation between the actual measures and the measures predicted using the average precision maximum entropy probability-at - rank distribution . Consider predicting precision-at-cutoff 10 values using the average precision maximum entropy distributions in TREC 6 . Without applying the maximum entropy method , Figure 8 shows that the two measures are correlated with a Kendall 's value of 0.671 . However , the precision-at-cutoff 10 values inferred from the average precision maximum entropy distribution have a Kendall 's value of 0.871 when compared to actual precisions-at-cutoff 10 . Hence , the predicted precision-at-cutoff 10 and actual precision-at-cutoff 10 values are much more correlated than the actual average precision and actual precision-at-cutoff 10 values . Using a similar approach for predicting R-precision and precision-at-cutoff 100 , it can be seen in Figures 8 and 9 that the measured values predicted by using average precision maximum entropy distributions are highly correlated with actual measured values . We conducted similar experiments using the maximum entropy distributions corresponding to other measures , but since these measures are less informative , we obtained much smaller increases -LRB- and sometimes even decreases -RRB- in inferred correlations . -LRB- These results are omitted due to space considerations . -RRB- Table 2 summarizes the correlation improvements possible using the maximum entropy distribution corresponding to average precision . The row labeled act gives the actual Kendall 's correlation between average precision and the measure in the corresponding column . The row labeled inf gives the Kendall 's correlation between the 31 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 recall precision TREC8 System fub99a Query 435 AP = 0.1433 actual prec-recall ap maxent prec-recall rp maxent prec-recall pc-10 maxent prec-recall 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 recall precision TREC8 System MITSLStd Query 404 AP = 0.2305 actual prec-recall ap maxent prec-recall rp maxent prec-recall pc-10 maxent prec-recall 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 recall precision TREC8 System pir9At0 Query 446 AP = 0.4754 actual prec-recall ap maxent prec-recall rp maxent prec-recall pc-10 maxent prec-recall Figure 6 : Inferred precision-recall curves and actual precision-recall curve for three runs in TREC8 . TREC3 TREC5 TREC6 TREC7 TREC8 TREC9 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 Mean Absolute Error ap maxent prec-recall rp maxent prec-recall pc-10 maxent prec-recall pc-100 maxent prec-recall TREC3 TREC5 TREC6 TREC7 TREC8 TREC9 0.1 0.15 0.2 0.25 RMS Error ap maxent prec-recall rp maxent prec-recall pc-10 maxent prec-recall pc-100 maxent prec-recall Figure 7 : MAE and RMS errors for inferred precision-recall curves over all TRECs . TREC3 TREC5 TREC6 TREC7 TREC8 TREC9 AVERAGE % INC AP 0.1185 0.1220 0.1191 0.1299 0.1390 0.1505 0.1298 RP 0.1767 0.1711 0.1877 0.2016 0.1878 0.1630 0.1813 39.7 PC-5 0.2724 0.2242 0.2451 0.2639 0.2651 0.2029 0.2456 89.2 PC-10 0.2474 0.2029 0.2183 0.2321 0.2318 0.1851 0.2196 69.1 PC-15 0.2320 0.1890 0.2063 0.2132 0.2137 0.1747 0.2048 57.8 PC-20 0.2210 0.1806 0.2005 0.2020 0.2068 0.1701 0.1968 51.6 PC-30 0.2051 0.1711 0.1950 0.1946 0.2032 0.1694 0.1897 46.1 PC-100 0.1787 0.1777 0.2084 0.2239 0.2222 0.1849 0.1993 53.5 PC-200 0.1976 0.2053 0.2435 0.2576 0.2548 0.2057 0.2274 75.2 PC-500 0.2641 0.2488 0.2884 0.3042 0.3027 0.2400 0.2747 111.6 PC-1000 0.3164 0.2763 0.3134 0.3313 0.3323 0.2608 0.3051 135.0 Table 1 : RMS error values for each TREC . TREC3 TREC5 TREC6 RP PC-10 PC-100 RP PC-10 PC-100 RP PC-10 PC-100 act 0.921 0.815 0.833 0.939 0.762 0.868 0.913 0.671 0.807 inf 0.941 0.863 0.954 0.948 0.870 0.941 0.927 0.871 0.955 % Inc 2.2 5.9 14.5 1.0 14.2 8.4 1.5 29.8 18.3 TREC7 TREC8 TREC9 RP PC-10 PC-100 RP PC-10 PC-100 RP PC-10 PC-100 act 0.917 0.745 0.891 0.925 0.818 0.873 0.903 0.622 0.836 inf 0.934 0.877 0.926 0.932 0.859 0.944 0.908 0.757 0.881 % Inc 1.9 17.7 3.9 0.8 5.0 8.1 0.6 21.7 5.4 Table 2 : Kendall 's correlations and percent improvements for all TRECs . 32 0 0.1 0.2 0.3 0.4 0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Actual RP Actual AP TREC 6 Actual RP vs Actual AP Kendall 's = 0.913 0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Actual PC-10 Actual AP TREC 6 Actual PC-10 vs Actual AP Kendall 's = 0.671 0 0.1 0.2 0.3 0.4 0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Actual PC-100 Actual AP TREC 6 Actual PC-100 vs Actual AP Kendall 's = 0.807 0 0.1 0.2 0.3 0.4 0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Actual RP Inferred RP TREC 6 Actual RP vs Inferred RP Kendall 's = 0.927 0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Actual PC-10 Inferred PC-10 TREC 6 Actual PC-10 vs Inferred PC-10 Kendall 's = 0.871 0 0.1 0.2 0.3 0.4 0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Actual PC-100 Inferred PC-100 TREC 6 Actual PC-100 vs Inferred PC-100 Kendall 's = 0.955 Figure 8 : Correlation improvements , TREC6 . measure inferred from the maximum entropy distribution corresponding to average precision and the measure in the corresponding column . The row labeled % Inc gives the percent increase in correlation due to maximum entropy modeling . As can be seen , maximum entropy modeling yields great improvements in the predictions of precision-at-cutoff values . The improvements in predicting R-precision are no-ticeably smaller , though this is largely due to the fact that average precision and R-precision are quite correlated to begin with . We have described a methodology for analyzing measures of retrieval performance based on the maximum entropy method , and we have demonstrated that the maximum entropy models corresponding to `` good '' measures of overall performance such as average precision accurately reflect underlying retrieval performance -LRB- as measured by precision-recall curves -RRB- and can be used to accurately predict the values of other measures of performance , well beyond the levels dictated by simple correlations . The maximum entropy method can be used to analyze other measures of retrieval performance , and we are presently conducting such studies . More interestingly , the maximum entropy method could perhaps be used to help develop and gain insight into potential new measures of retrieval performance . Finally , the predictive quality of maximum entropy models corresponding to average precision suggest that if one were to estimate some measure of performance using an incomplete judgment set , that measure should be average precision -- from the maximum entropy model corresponding to that measure alone , one could accurately infer other measures of performance . Note that the concept of a `` good '' measure depends on the purpose of evaluation . In this paper , we evaluate measures based on how much information they provide about the overall performance of a system -LRB- a system-oriented evaluation -RRB- . However , in different contexts , different measures may be more valuable and useful , such as precision-at-cutoff 10 in web search -LRB- a user-oriented evaluation -RRB- . R-precision and average precision are system-oriented measures , whereas precision-at-cutoff k is typically a user-oriented measure . Another important conclusion of our work is that one can accurately infer user-oriented measures from system-oriented measures , but the opposite is not true . Apart from evaluating the information captured by a single measure , we could use the MEM to evaluate the information contained in combinations of measures . How much does knowing the value of precision-at-cutoff 10 increase one 's knowledge of a system 's performance beyond simply knowing the system 's average precision ? Which is more informative : knowing R-precision and precision-at-cutoff 30 , or knowing average precision and precision-at-cutoff 100 ? Such questions can be answered , in principle , using the MEM . Adding the values of one or more measures simply adds one or more constraints to the maximum entropy model , and one can then assess the informativeness of the combination . Note that TREC reports many different measures . Using the MEM , one might reasonably be able to conclude which are the most informative combinations of measures . -LSB- 1 -RSB- A. L. Berger , V. D. Pietra , and S. D. Pietra . A maximum entropy approach to natural language processing . Comput . Linguist. , 22:39 71 , 1996 . -LSB- 2 -RSB- C. Buckley and E. Voorhees . Evaluating evaluation measure stability . In SIGIR ' 00 : Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval , pages 33 40 . ACM Press , 2000 . -LSB- 3 -RSB- W. S. Cooper . On selecting a measure of retrieval effectiveness . part i . In Readings in information retrieval , pages 191 204 . Morgan Kaufmann Publishers Inc. , 1997 . 33 0 0.1 0.2 0.3 0.4 0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Actual RP Actual AP TREC 8 Actual RP vs Actual AP Kendall 's = 0.925 0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Actual PC-10 Actual AP TREC 8 Actual PC-10 vs Actual AP Kendall 's = 0.818 0 0.1 0.2 0.3 0.4 0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Actual PC-100 Actual AP TREC 8 Actual PC-100 vs Actual AP Kendall 's = 0.873 0 0.1 0.2 0.3 0.4 0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Actual RP Inferred RP TREC 8 Actual RP vs Inferred RP Kendall 's = 0.932 0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Actual PC-10 Inferred PC-10 TREC 8 Actual PC-10 vs Inferred PC-10 Kendall 's = 0.859 0 0.1 0.2 0.3 0.4 0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Actual PC-100 Inferred PC-100 TREC 8 Actual PC-100 vs Inferred PC-100 Kendall 's = 0.944 Figure 9 : Correlation improvements , TREC8 . -LSB- 4 -RSB- T. M. Cover and J. Thomas . Elements of Information Theory . John Wiley & sons , 1991 . -LSB- 5 -RSB- B. Dervin and M. S. Nilan . Information needs and use . In Annual Review of Information Science and Technology , volume 21 , pages 3 33 , 1986 . -LSB- 6 -RSB- W. R. Greiff and J. Ponte . The maximum entropy approach and probabilistic ir models . ACM Trans . Inf . Syst. , 18 -LRB- 3 -RRB- : 246 287 , 2000 . -LSB- 7 -RSB- E. Jaynes . On the rationale of maximum entropy methods . In Proc.IEEE , volume 70 , pages 939 952 , 1982 . -LSB- 8 -RSB- E. T. Jaynes . Information theory and statistical mechanics : Part i. Physical Review 106 , pages 620 630 , 1957a . -LSB- 9 -RSB- E. T. Jaynes . Information theory and statistical mechanics : Part ii . Physical Review 108 , page 171 , 1957b . -LSB- 10 -RSB- Y. Kagolovsky and J. R. Moehr . Current status of the evaluation of information retrieval . J. Med . Syst. , 27 -LRB- 5 -RRB- : 409 424 , 2003 . -LSB- 11 -RSB- P. B. Kantor and J. Lee . The maximum entropy principle in information retrieval . In SIGIR ' 86 : Proceedings of the 9th annual international ACM SIGIR conference on Research and development in information retrieval , pages 269 274 . ACM Press , 1986 . -LSB- 12 -RSB- D. D. Lewis . Evaluating and optimizing autonomous text classification systems . In SIGIR ' 95 : Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval , pages 246 254 . ACM Press , 1995 . -LSB- 13 -RSB- R. M. Losee . When information retrieval measures agree about the relative quality of document rankings . J. Am . Soc . Inf . Sci. , 51 -LRB- 9 -RRB- : 834 840 , 2000 . -LSB- 14 -RSB- K. Nigam , J. Lafferty , and A. McCallum . Using maximum entropy for text classification . In IJCAI-99 Workshop on Machine Learning for Information Filtering , pages 61 67 , 1999 . -LSB- 15 -RSB- D. Pavlov , A. Popescul , D. M. Pennock , and L. H. Ungar . Mixtures of conditional maximum entropy models . In T. Fawcett and N. Mishra , editors , ICML , pages 584 591 . AAAI Press , 2003 . -LSB- 16 -RSB- S. J. Phillips , M. Dudik , and R. E. Schapire . A maximum entropy approach to species distribution modeling . In ICML ' 04 : Twenty-first international conference on Machine learning , New York , NY , USA , 2004 . ACM Press . -LSB- 17 -RSB- V. Raghavan , P. Bollmann , and G. S. Jung . A critical investigation of recall and precision as measures of retrieval system performance . ACM Trans . Inf . Syst. , 7 -LRB- 3 -RRB- : 205 229 , 1989 . -LSB- 18 -RSB- A. Ratnaparkhi and M. P. Marcus . Maximum entropy models for natural language ambiguity resolution , 1998 . -LSB- 19 -RSB- T. Saracevic . Evaluation of evaluation in information retrieval . In SIGIR ' 95 : Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval , pages 138 146 . ACM Press , 1995 . -LSB- 20 -RSB- C. E. Shannon . A mathematical theory of communication . The Bell System Technical Journal 27 , pages 379 423 & 623 656 , 1948 . -LSB- 21 -RSB- N. Wu . The Maximum Entropy Method . Springer , New York , 1997 . 34

O B I O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O B I B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O

maximum entropy
average precision
evaluation