Catenaccio : Interactive Information Retrieval System through Drawing The Catenaccio system integrates information retrieval with sketch manipulations . The system is designed especially for pen-based computing and allows users to retrieve information by simple pen manipulations such as drawing a picture . When a user draws a circle and writes a keyword , information nodes related to the keyword are collected automatically inside the circle . In addition , the user can create a Venn diagram by repeatedly drawing circles and keywords to form more complex queries . Thus , the user can retrieve information both interactively and visually without complex manipulations . Moreover , the sketch interaction is so simple that it is possible to combine it with other types of data such as images and real-world information for information retrieval . In this paper , we describe our Catenaccio system and how it can be effectively applied . CR Categories : I. 3.6 -LSB- Computer Graphics -RSB- : Methodology and Techniques Interaction Techniques ; I. 3.4 -LSB- Computer Graphics -RSB- : Graphics Utilities Software support Pen-based computers , such as personal digital assistants -LRB- PDA -RRB- and tablet PCs , have been developed . These computers are characterized by simple sketch interfaces similar to drawing a picture on paper in the real world . This drawing manipulation is not especially useful for communicating details , but is effective for general use . It is especially useful for creative activities , so there have been a number of research reports on improving sketch manipulation -LSB- 1 , 2 , 3 -RSB- . In addition , some game devices -LRB- e.g. , Nintendo DS -LSB- 4 -RSB- -RRB- support such kinds of interactions and provide many types of game content . In these systems , a user can use the entire system window as a workspace and create 3D CG from 2D drawings . However , as the original applications may not support information retrieval , the user has to use conventional retrieval applications along with pen-based input styles . Considerable research has been done to support the use of information visualization for retrieving information -LSB- 5 -RSB- . Technical visualization methods such as zooming and scaling can be used to effectively display huge amounts of data -LSB- 6 , 7 , 8 -RSB- . However , existing visualization systems focus on mouse manipulation -LRB- e.g. , click and drag -RRB- , so they are not effectively designed for pen-based interactions such as a drawing . The most popular method of retrieving information is no doubt keyword searching . Search engines via the Web -LRB- e.g. , Google and Yahoo -RRB- have been generally used for keyword searching -LSB- 12 , 13 -RSB- , and people feel that they can not live without such search engines . Generally , keyword searching requires users to input one or more keywords . In these systems , users can retrieve information related to the keywords with Boolean operations -LRB- e.g. , AND , OR and NOT -RRB- . However , the systems are based on conventional input methods . Users of pen-based computers have to write a query into a fixed dialog box with a stylus or pen . Therefore , we have been developing an information retrieval system based on simple sketch manipulations . Our goal is to devise an effective and simple information retrieval system that works on pen-based computers , so we integrated a keyword searching that is one of the most usual methods with sketch manipulation that is one of the simple interactions . In our system , users retrieve information by drawing a Venn diagram instead of inputting keywords to a dialog box . Because the Venn diagram can be used to display Boolean operations -LRB- e.g. , AND , OR , and NOT -RRB- visually and create some relationships at the same time , users can recognize the relationships at a glance . Moreover , the system allows users to use other types of data as elements in a Venn diagram -LRB- Fig. 1 -RRB- . In this paper , we describe our Catenaccio system that integrates information retrieval with sketch manipulations , and explain how it can be effectively applied for information retrieval . Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page . To copy otherwise , or republish , to post on servers or to redistribute to lists , requires prior specific permission and/or a fee . AVI ' 06 , May 23 26 , 2006 , Venezia , Italy . Copyright 2006 ACM 1-59593-353-0 / 06/0005 ... $ 5.00 . 79 Figure 1 : Venn diagram : Venn diagram can be used to display Boolean operations and some relationships at the same time -LRB- top -RRB- . The user can create an original Venn diagram -LRB- bottom -RRB- . Figure 2 . Basic manipulation : A user of the Catenaccio system draws a circle and writes a keyword inside that circle . Information nodes related to the keyword are then collected within the circled area . A wide variety of information visualization systems are used for information retrieval -LSB- 5 -RSB- . Treating information as visualized nodes -LRB- e.g. , images and simple shapes -RRB- allows users to interact with the information space visually . Moreover , several techniques -LRB- e.g. , scaling , zooming , focus and context -RRB- are used to display a huge amount of information more effectively -LSB- 6 , 7 , 8 , 19 -RSB- . Especially , spring model -LSB- 11 -RSB- provides useful ways to recognize the relationships between nodes . In these systems , related nodes move when the user clicks and drags a node . That is , node positions are dynamically changed through the user manipulations , so users retrieve information interactively . InfoCrystal -LSB- 9 -RSB- is also a visual tool focused on information retrieval . The system uses Venn diagrams to treat huge amounts of information effectively . However , conventional systems are designed for mouse interactions -LRB- e.g. , click and drag -RRB- and their layouts are predefined , so they are not suitable for Pen-based computing , especially drawing or writing by hand . There are also several sketch interfaces focusing on pen-based computing -LSB- 1 , 2 , 3 -RSB- . Most of them enhance drawing manipulations and focus on 3D creations performed with 2D manipulation . Characteristically , the manipulations required for these systems are simple and are similar to drawing a stroke on a piece of paper with a pen . Sketch -LSB- 1 -RSB- users can draw 3D curves by performing 2D manipulations . This system calculates a 3D curve by combining a 2D stroke and a shadow stroke . Users of Harold -LSB- 2 -RSB- and Tolba -LSB- 3 -RSB- can create flat models in a 3D space by using sketch-based manipulation , effectively creating a 2.5 D scene in a 3D space . Figure 3 . Drawing a Venn diagram : By repeatedly drawing circles and keywords , users create Venn diagrams , and can then retrieve information by forming complex queries . Information nodes related with both `` CG '' and `` 3D '' are collected . The Catenaccio system is focused on pen-based computing and provides an interactive and visual information retrieval environment of using drawing manipulations . 3.1 Drawing Circles and Writing Keywords A user of the Catenaccio system draws a circle and writes a keyword inside that circle . The system automatically recognizes both the circle area and the keyword . Information nodes related to the keyword are then collected within the circled area . By making a continuous series of simple drawings , the user can create a Venn diagram to form a more complex query . Since the entire window is both a search area and a drawing canvas , the user can use the workspace freely . Since all the manipulations required for information retrieval are based on sketch manipulations , users can design an original Venn diagram related their interests . Thus , users can freely exploit the whole application window as both an input and a search area and retrieve information without complex GUIs . The circle provides an area where information nodes related to the keyword will be collected , and the keyword provides a query to search for related information nodes from a database . By continuing to use simple drawings , users can form more complex queries . The related information nodes are moved with a force that depends on the distance between the node position and the center of the circle -LRB- Fig. 2 -RRB- . Figure 3 shows an example of creating a Venn diagram by continuing to draw circles and keywords . In the example , when a user retrieves information that has two keywords `` CG '' and `` 3D '' , the user first draws a circle and writes `` CG '' -LRB- Fig. 3 -LRB- 1 -RRB- -RRB- , and then draws another circle and writes `` 3D '' -LRB- Fig. 3 -LRB- 2 -RRB- -RRB- . Information nodes related to both keywords appear in the shared area of the Venn diagram -LRB- Fig. 3 -LRB- 3 and 4 -RRB- -RRB- . Moreover , in the Venn diagram , the user can view four areas at a glance -LRB- Fig. 3 -LRB- 3 -RRB- -RRB- . 80 Figure 4 : Venn diagram of a drawing and an image , and Venn diagram of a drawing and an image that contains character information . Figure 5 : Combination with real-world information : Capturing real-world information as a picture -LRB- 1 , 2 -RRB- and drawing a circle around the keyword brings up related information nodes -LRB- 3 , 4 -RRB- . 3.2 Combination with Other Types of Data Users can now easily take pictures using digital cameras and cell phones that contain CCD cameras . As a result , they may have a huge amount of original image data in their computers . These data include some information such as name , time , or place , so we considered using them for information retrieval . We have developed prototype applications to explore the potential of Catenaccio . A Venn diagram is basically constructed by combining keywords and areas , so it is possible to combine that diagram with other types of data such as images and real-world information . Images contain name , time , or place information , and that information becomes a good trigger for retrieving other information and it can be used for queries . In addition , the image data has a rectangular shape that is useful for setting an area by controlling its position and size . The example in Figure 4 shows how users can use images to create Venn diagrams by combining drawings with image data . In the example , a file named `` Mr. Tobita '' becomes a query for the Venn diagram , so information related to `` VR '' AND `` Mr. Tobita '' is collected . In this case , even if users have forgotten someone 's name , they can still retrieve related information through image contents . Figure 5 shows an example a Venn diagram with real-world information . Generally , using captured data for an interaction trigger is a common technique in AR systems -LSB- 20 -RSB- . We also use it for elements of Venn diagrams . Users first capture real-world information through digital cameras attached to their computers -LRB- Fig. 5 -LRB- 1 , 2 -RRB- -RRB- , and then draw circles around keywords on the captured data and another circle to collect information nodes -LRB- Fig . 5 -LRB- 3 -RRB- -RRB- . As the system recognizes the keyword inside the first circle , related information nodes appear inside the second curve -LRB- Fig. 5 -LRB- 4 -RRB- -RRB- . Figure 6 : Recognition of user drawings : The system labels the user drawing area -LRB- 1 , 2 -RRB- . The system recognizes written keywords by using an OCR library -LRB- 3 -RRB- Previously , we have proposed a similar information retrieval system -LSB- 10 -RSB- . That system provides natural interactions , however , it completely depends on real-world objects . However , Catenaccio provides not only real-world information , but also drawing manipulations . Thus , the user can retrieve information even if there are not enough real-world objects . The entire workspace is bitmapped as in conventional 2D paint systems , and user drawing manipulations are reflected in the bitmap . The system supports two types of drawing , writing keywords and area drawings . The keyword writing is displayed as green , and the area drawing is displayed as blue . To set an area , the system labels the inside of a green area and knows the size and position of the area -LRB- Fig. 6 -LRB- 1 -RRB- -RRB- . Then , the system divides the area into four layers for node animations -LRB- Fig. 6 -LRB- 2 -RRB- -RRB- . For keyword writing , the system sends the result to an OCR library to search for its meaning -LRB- Fig. 6 -LRB- 3 -RRB- -RRB- . Catenaccio is a prototype system now , and the relationship between keywords and information nodes are predefined in a temporary database . The database contains three types of data : node names , keywords , and relationship levels . After the image recognition processes , nodes related to the keyword are selected and start moving until they are in the area appropriate to their relationship . The force is calculated by spring model -LSB- 11 -RSB- . For example , the node with the strongest relationship with the keyword receives a force that takes it to the deepest area . We have had some opportunities to demonstrate our system . Here , we discuss user interactions with Catenaccio based on comments made by visitors to our demonstrations . Also we consider the limitations of the system and our plans for future work . From our demonstrations , the visitors quickly understood the concepts of our system that integrates information retrieval with 81 drawing manipulations . Using Venn diagrams makes recognizing the relationships between information nodes and keywords easy . Most visitors could create simple Venn diagrams and set related nodes into the diagrams after watching a simple demonstration . We observed that some users drew interesting Venn diagrams that resembled pictures . The system facilitates creative activities , so we expect users will be able to create more original , and increasingly effective drawings for information retrieval . Especially , we received good reactions from users regarding the combination of drawing and an image to create a Venn diagram . By exploiting such combinations , the system augments keyword searching , and it is different from conventional search engines -LSB- 12 , 13 -RSB- . Moreover , combining keywords and user-drawn pictures to create Venn diagrams is possible . Our system focuses on information retrieval for pen-based input . However , information retrieval using Venn diagrams is quite rough . We plan to combine our system with other types of sketch-based systems such as VelvetPath -LSB- 16 -RSB- to support more detailed interaction . With such a combination , users can use Catenaccio for general retrieval of information and then use VelvetPath to examine the information in more detail . In this case , all the manipulations would still be based on drawing or handwriting , so a user can handle a large amount of data in a natural way . Drawing manipulation is also useful for finger gestures . Many AR systems support the use of finger gestures as an input method -LSB- 17 , 18 -RSB- . As the system recognizes user finger gestures , users can create Venn diagrams by manipulating real-world objects , drawing circles , and writing keywords . We described the Catenaccio system that is focused on Pen-based computing and allows users to retrieve information by drawing Venn diagrams . The system recognizes user writing and drawings -LRB- keywords and circles -RRB- and places information related to the keywords inside the circles . Using this input , the system provides an interactive and visual information retrieval method . We described some examples of retrieving information through simple drawings . We also provided several examples of unique Venn diagrams created by combining drawings with images and real-world information . We thank Tetsuji Takada and Sinji Daigo for the visible and useful suggestions on this work . -LSB- 1 -RSB- R. C. Zeleznik , K. P. Herndon , and J. F. Hughes . An Interface for Sketching 3D Curves . In Proceedings of ACM SIGGRAPH ' 96 , pp. 163-170 , 1996 . -LSB- 2 -RSB- J. M. Cohen , J. F. Hughes , and R. C. Zeleznik . Harold : A World Made of Drawings . In Proceedings of NPAR2000 -LRB- Symposium on Non-Photorealistic Animation and Rendering -RRB- , pp. 83-90 , 2000 . -LSB- 3 -RSB- O. Tolba , J. Doresey , and L. McMillan . Sketching with Projective 2D Strokes . In Proceedings of ACM UIST ' 99 , pp. 149-157 , 1999 . -LSB- 4 -RSB- Nintendo DS : http://www.nintendo.co.jp/ds/ -LSB- 5 -RSB- S. K. Card , J. D. MacKinlay , and B. Shneiderman . Readings in Information Visualization : Using Vision to Think . Morgan Kaufmann , 1999 . -LSB- 6 -RSB- H. Koike . Fractal views : a fractal-based method for controlling information display . In Proceedings of ACM Transactions on Information Systems , Vol . 13 , No. 3 , pp. 305-323 , July 1995 . -LSB- 7 -RSB- G. W. Furnas . Generalized fisheye views . In Proceedings of the ACM Transactions on Computer-Human Interaction , Vol . 1 , No. 2 , pp. 126-160 , 1994 . -LSB- 8 -RSB- B. B. Bederson , J. D. Hollan , K. Perlin , J. Meyer , D. Bacon , and G. Furnas . Pad + + : A Zoomable Graphical Sketchpad for Exploring Alternate Interface Physics . Journal of Visual Languages and Computing , Vol . 7 , No. 1 , pp. 3-31 , 1996 . -LSB- 9 -RSB- A. Spoerri . Visual tools for information retrieval . In Proceedings of VL ' 93 , pp. 160-168 , 1993 . -LSB- 10 -RSB- H. Koike , Y. Sato , Y. Kobayashi , H. Tobita and M. Kobayashi . Interactive Textbook and Interactive Venn Diagram . In Proceedings of ACM CHI2000 , pp. 121-128 , 2000 . -LSB- 11 -RSB- R. Davidson and D. Harel . Drawing Graphics Nicely Using Simulated Annealing . In Proceedings of ACM Transactions on Graphics , Vol . 15 , No. 4 , pp. 301-331 , 1996 . -LSB- 12 -RSB- Google : http://www.google.com -LSB- 13 -RSB- Yahoo : http://www.yahoo.com -LSB- 14 -RSB- T. Calishain and R. Dornfest . GoogleHack : 100 Industrial-Strength Tips & Tricks . O'RELLY , 2003 . -LSB- 15 -RSB- P. Bausch . AmazonHack : 100 Industrial-Strength Tips & Tools . O'RELLY , 2003 . -LSB- 16 -RSB- H. Tobita . VelvetPath : Layout Design System with Sketch and Paint Manipulations , In Proceedings of EUROGRAPHICS2003 Short Presentations , pp. 137-144 , 2003 . -LSB- 17 -RSB- J. Rekimoto . SmartSkin : An Infrastructure for Freehand Manipulation on Interactive Surfaces , In Proceedings of ACM CHI2002 , 113-120 , 2002 . -LSB- 18 -RSB- X. Chen , H. Koike , Y. Nakanishi , K. Oka , and Y. Sato . Two-handed drawing on augmented desk system , In Proceedings of AVI 2002 , 2002 . -LSB- 19 -RSB- E. Orimo and H. Koike . ZASH : A browsing system for multi-dimensional data . In Proceedings of IEEE VL ' 99 , pp . 266-286 , 1999 . -LSB- 20 -RSB- J. Rekimoto and K. Nagao . The world through the computer : Computer augmented interaction with real world environments . In Proceedings of ACM UIST ' 95 , pp. 29-36 , 1995 . 82

O O O B I O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

information retrieval
venn diagram
image data
sketch manipulations
sketch manipulation
keyword searching

interactive information retrieval system


Efficient

Multi-way

Text

Categorization

via

Generalized

Discriminant

Analysis

Text

categorization

is

an

important

research

area

and

has

been

receiving

much

attention

due

to

the

growth

of

the

on-line

information

and

of

Internet

Automated

text

categorization

is

generally

cast

as

a

multi-class

classification

problem

Much

of

previous

work

focused

on

binary

document

classification

problems

Support

vector

machines

-LRB-

SVMs

-RRB-

excel

in

binary

classification

but

the

elegant

theory

behind

large-margin

hyperplane

can

not

be

easily

extended

to

multi-class

text

classification

In

addition

the

training

time

and

scaling

are

also

important

concerns

On

the

other

hand

other

techniques

naturally

extensible

to

handle

multi-class

classification

are

generally

not

as

accurate

as

SVM

This

paper

presents

a

simple

and

efficient

solution

to

multi-class

text

categorization

Classification

problems

are

first

formulated

as

optimization

via

discriminant

analysis

Text

categorization

is

then

cast

as

the

problem

of

finding

coordinate

transformations

that

reflects

the

inherent

similarity

from

the

data

While

most

of

the

previous

approaches

decompose

a

multiclass

classification

problem

into

multiple

independent

binary

classification

tasks

the

proposed

approach

enables

direct

multi-class

classification

By

using

Generalized

Singular

Value

Decomposition

-LRB-

GSVD

-RRB-

a

coordinate

transformation

that

reflects

the

inherent

class

structure

indicated

by

the

generalized

singular

values

is

identified

Extensive

experiments

demonstrate

the

efficiency

and

effectiveness

of

the

proposed

approach

H.

-LSB-

Information

Storage

and

Retrieval

-RSB-

Information

Search

and

Retrieval

I.

-LSB-

Artificial

Intelligence

-RSB-

Learning

I.

-LSB-

Pattern

Recognition

-RSB-

Applications

Algorithms

Measurement

Performance

Experimentation

Verifi-cation

The

current

affiliation

Amazon.com

Inc.

Permission

to

make

digital

or

hard

copies

of

all

or

part

of

this

work

for

personal

or

classroom

use

is

granted

without

fee

provided

that

copies

are

not

made

or

distributed

for

profit

or

commercial

advantage

and

that

copies

bear

this

notice

and

the

full

citation

on

the

first

page

To

copy

otherwise

to

republish

to

post

on

servers

or

to

redistribute

to

lists

requires

prior

specific

permission

and/or

a

fee

CIKM

November

New

Orleans

Louisiana

USA

Copyright

ACM

With

the

ever-increasing

growth

of

the

on-line

information

and

the

permeation

of

Internet

into

daily

life

methods

that

assist

users

in

organizing

large

volumes

of

documents

are

in

huge

demand

In

particular

automatic

text

categorization

has

been

extensively

studied

recently

This

categorization

problem

is

usually

viewed

as

supervised

learning

where

the

gaol

is

to

assign

predefined

category

labels

to

unlabeled

documents

based

on

the

likelihood

in-ferred

from

the

training

set

of

labeled

documents

Numerous

approaches

have

been

applied

including

Bayesian

probabilistic

approaches

-LSB-

-RSB-

nearest

neighbor

-LSB-

-RSB-

neural

networks

-LSB-

-RSB-

decision

trees

-LSB-

-RSB-

inductive

rule

learning

-LSB-

-RSB-

support

vector

machines

-LSB-

-RSB-

Maximum

Entropy

-LSB-

-RSB-

boosting

-LSB-

-RSB-

and

linear

discriminate

projection

-LSB-

-RSB-

-LRB-

see

-LSB-

-RSB-

for

comparative

studies

of

text

categorization

methods

-RRB-

Although

document

collections

are

likely

to

contain

many

different

categories

most

of

the

previous

work

was

focused

on

binary

document

classification

One

of

the

most

effective

binary

classification

techniques

is

the

support

vector

machines

-LRB-

SVMs

-RRB-

-LSB-

-RSB-

It

has

been

demonstrated

that

the

method

performs

superbly

in

binary

discriminative

text

classification

-LSB-

-RSB-

SVMs

are

accurate

and

robust

and

can

quickly

adapt

to

test

instances

However

the

elegant

theory

behind

the

use

of

large-margin

hyperplanes

can

not

be

easily

extended

to

multi-class

text

categorization

problems

A

number

of

techniques

for

reducing

multi-class

problems

to

binary

problems

have

been

proposed

including

one-versus-the-rest

method

pairwise

comparison

-LSB-

-RSB-

and

error-correcting

output

coding

-LSB-

-RSB-

In

these

approaches

the

original

problems

are

decomposed

into

a

collection

of

binary

problems

where

the

assertions

of

the

binary

classifiers

are

integrated

to

produce

the

final

output

In

practice

which

reduction

method

is

best

suited

is

problem-dependent

so

it

is

a

non-trivial

task

to

select

the

decomposition

method

Indeed

each

reduction

method

has

its

own

merits

and

limitations

-LSB-

-RSB-

In

addition

regardless

of

specific

details

these

reduction

techniques

do

not

appear

to

be

well

suited

for

text

categorization

tasks

with

a

large

number

of

categories

because

training

of

a

single

binary

SVM

requires

O

n

time

for

where

n

is

the

number

of

training

data

-LSB-

-RSB-

Thus

having

to

train

many

classifiers

has

a

significant

impact

on

the

overall

training

time

Also

the

use

of

multiple

classifiers

slows

down

prediction

Thus

despite

its

elegance

and

superiority

the

use

of

SVM

may

not

be

best

suited

for

multi-class

document

classification

However

there

do

not

appear

to

exist

many

alternatives

since

many

other

techniques

that

can

be

naturally

extended

to

handle

multi-class

classification

problems

such

as

neural

networks

and

decision

trees

are

not

so

accurate

as

SVMs

-LSB-

-RSB-

In

statistics

pattern

recognition

literature

discriminant

analysis

approaches

are

well

known

to

be

able

to

learn

discriminative

feature

transformations

-LRB-

see

e.g.

-LSB-

-RSB-

-RRB-

For

example

Fisher

discriminant

analysis

-LSB-

-RSB-

finds

a

discriminative

feature

transformation

as

eigenvectors

associated

with

the

largest

eigenvalues

of

matrix

T

w

b

where

w

is

the

intra-class

covariance

matrix

and

b

is

the

inter-class

covariance

matrix

Intuitively

T

captures

not

only

compactness

of

individual

classes

but

separations

among

them

Thus

eigenvectors

corresponding

to

the

largest

eigenvalues

of

T

are

likely

to

constitute

a

discriminative

feature

transform

However

for

text

categorization

w

is

usually

singular

owing

to

the

large

number

of

terms

Simply

removing

the

space

of

w

would

eliminate

important

discriminant

information

when

the

projections

of

b

along

those

directions

are

not

zeros

-LSB-

-RSB-

This

issue

has

stymied

attempts

to

use

traditional

discriminant

approaches

in

document

analysis

In

this

paper

we

resolve

this

problem

We

extend

discriminant

analysis

and

present

a

simple

efficient

but

effective

solution

to

text

categorization

We

propose

a

new

optimization

criterion

for

classification

and

cast

text

categorization

as

the

problem

of

finding

transformations

to

reflect

the

inherent

similarity

from

the

data

In

this

framework

given

a

document

of

unknown

class

membership

we

compare

the

distance

of

the

new

document

to

the

centroid

of

each

category

in

the

transformed

space

and

assign

it

to

the

class

having

the

smallest

distance

to

it

We

call

this

method

Generalized

Discriminant

Analysis

-LRB-

GDA

-RRB-

since

it

uses

generalized

singular

value

decomposition

to

optimize

transformation

We

show

that

the

transformation

derived

using

GDA

is

equivalent

to

optimization

via

the

trace

or

determinant

ratios

GDA

has

several

favorable

properties

First

it

is

simple

and

can

be

programed

in

a

few

lines

in

MATLAB

Second

it

is

efficient

-LRB-

Most

of

our

experiments

only

took

several

seconds

-RRB-

Third

the

algorithm

does

not

involve

parameter

tuning

Finally

and

probably

the

most

importantly

it

is

very

accurate

We

have

conducted

extensive

experiments

on

various

datasets

to

evaluate

its

performance

The

rest

of

the

paper

is

organized

as

follows

Section

reviews

the

related

work

on

text

categorization

Section

introduces

our

new

criterion

for

discriminant

analysis

Section

introduces

the

basics

of

generalized

singular

value

decomposition

and

gives

the

solution

of

the

optimization

problem

Section

shows

that

the

transformation

derived

using

GDA

can

also

be

obtained

by

optimizing

the

trace

or

determinant

ratios

Section

presents

some

illustrating

examples

Section

shows

experimental

results

Finally

Section

provides

conclusions

and

discussions

Text

categorization

algorithms

can

be

roughly

classified

into

two

types

those

algorithms

that

can

be

naturally

extended

to

handle

multi-class

cases

and

those

require

decomposition

into

binary

classification

problems

The

first

consists

of

such

algorithms

as

Naive

Bayes

-LSB-

-RSB-

neural

networks

-LSB-

-RSB-

K-Nearest

Neighbors

-LSB-

-RSB-

Maximum

Entropy

-LSB-

-RSB-

and

decision

trees

Naive

Bayes

uses

the

joint

distributions

of

words

and

categorizes

to

estimate

the

probabilities

that

an

input

document

belongs

to

each

document

class

and

This

is

equivalent

to

using

eigenvectors

associated

with

the

smallest

eigenvalues

of

matrix

T

b

w

It

indicates

that

traditional

discriminant

analysis

requires

the

non-singularity

of

at

least

one

covariance

matrix

Since

the

rank

of

w

is

usually

greater

than

that

of

b

we

will

base

our

discussion

on

the

eigenvalue-decomposition

of

T

w

b

then

selects

the

most

probable

class

K-Nearest

Neighbor

finds

the

k

nearest

neighbors

among

training

documents

and

uses

the

categories

of

the

k

neighbors

to

determine

the

category

of

the

test

document

The

underlying

principle

of

maximum

entropy

is

that

without

external

knowledge

uniform

distribution

should

be

preferred

Based

on

this

principle

it

estimate

the

conditional

distribution

of

the

class

label

given

a

document

The

reduction

techniques

that

are

used

by

the

second

group

include

one-versus-the-rest

method

-LSB-

-RSB-

error-correcting

output

coding

-LSB-

-RSB-

pairwise

comparison

-LSB-

-RSB-

and

multi-class

objective

functions

where

the

first

two

have

been

applied

to

text

categorization

-LSB-

-RSB-

In

the

one-versus-the-rest

method

a

classifier

separating

between

from

a

class

and

the

rest

is

trained

for

each

class

Multi-class

classification

is

carried

out

by

integrating

prediction

of

these

individual

classifiers

with

a

strategy

for

resolving

conflicts

The

method

is

sometimes

criticizes

for

solving

asymmetric

problems

in

a

symmetrical

manner

and

for

not

considering

correlations

between

classes

Error-correcting

output

coding

-LRB-

ECOC

-RRB-

-LSB-

-RSB-

partitions

the

original

set

of

classes

into

two

sets

in

many

different

ways

A

binary

classifier

is

trained

for

each

partition

The

partitions

are

carefully

chosen

so

that

the

outputs

of

these

classifiers

assign

a

unique

binary

codeword

for

each

class

-LRB-

with

a

large

Hamming

distance

between

any

pair

of

them

-RRB-

The

class

of

an

input

with

unknown

class

membership

is

chosen

by

computing

the

outputs

of

the

classifiers

on

that

input

and

then

finding

the

class

with

the

codeword

closest

to

the

output

codeword

Although

SVMs

are

considered

to

be

very

effective

in

binary

classification

its

large

training

costs

may

make

it

unsuitable

for

multi-class

classification

with

a

large

number

of

classes

if

the

above

decomposition

techniques

are

applied

Also

the

lack

of

a

clear

winner

among

the

above

techniques

makes

the

reduction

task

complicated

Our

GDA

directly

deals

with

multi-class

classification

and

does

not

require

reduction

to

binary

classification

problems

Other

techniques

for

text

categorization

exist

Godbole

et

al.

-LSB-

-RSB-

propose

a

new

multi-class

classification

technique

that

exploits

the

accuracy

of

SVMs

and

the

speed

of

Naive

Bayes

It

uses

a

Naive

Bayes

classifier

to

compute

a

confusion

matrix

quickly

Then

it

uses

this

matrix

to

reduce

both

the

number

and

the

complexity

of

binary

SVMs

to

be

built

Chakrabarti

et

al.

-LSB-

-RSB-

propose

a

fast

text

classification

technique

that

uses

multiple

linear

projections

It

first

projects

training

instances

to

low-dimensional

space

and

then

builds

decision

tree

classifiers

on

the

projected

spaces

Fragoudis

et

al.

-LSB-

-RSB-

propose

a

new

algorithm

that

targets

both

feature

and

instance

selection

for

text

categorization

In

summary

as

pointed

out

in

-LSB-

-RSB-

there

is

no

obvious

winner

in

multi-class

classification

techniques

For

practical

problems

the

choice

of

approach

will

have

to

be

made

depending

on

the

constraints

e.g.

the

desired

accuracy

level

the

time

available

and

the

nature

of

the

problem

Suppose

the

dataset

D

has

m

instances

d

d

m

having

p

features

each

Then

D

can

be

viewed

as

a

subset

of

R

p

as

well

as

a

member

of

R

m

p

Suppose

D

has

L

classes

D

D

L

having

m

m

L

instances

respectively

where

m

L

i

m

i

For

each

i

i

L

let

J

i

be

the

set

of

all

j

j

m

such

that

the

j-th

instance

belongs

to

the

i-th

class

and

let

c

i

be

the

centroid

of

the

i-th

class

i.e.

the

component-wise

average

of

the

m

i

vectors

in

the

class

Let

c

be

the

centroid

of

the

entire

dataset

The

intra-class

scatter

matrix

of

D

w

is

defined

by

w

L

i

j

J

i

d

j

c

i

T

d

j

c

i

and

its

inter-class

scatter

matrix

b

is

defined

by

b

L

i

j

J

i

d

j

c

T

d

j

c

Let

A

w

be

the

m

p

matrix

constructed

by

stacking

D

e

T

c

D

L

e

L

T

c

L

and

let

A

b

be

the

p

m

matrix

whose

columns

are

from

left

to

right

m

c

c

T

m

L

c

L

c

T

Then

w

A

w

A

T

w

and

b

A

b

A

T

b

Although

there

are

ways

-LRB-

such

as

Kernel

tricks

-LSB-

-RSB-

-RRB-

for

utilizing

non-linear

transformation

we

will

focus

on

linear

transformation

Given

a

linear

transformation

the

covariance

matrices

in

the

transformed

space

are

A

b

T

A

b

T

A

T

b

A

b

T

b

and

A

w

T

A

w

T

A

T

w

A

w

T

w

Fisher

's

linear

discriminant

analysis

discriminates

inter-class

distance

and

intra-class

distance

by

using

their

corresponding

covariance

matrices

The

optimal

projection

can

be

obtained

by

solving

the

generalized

eigenvalue

problem

b

w

-LRB-

-RRB-

If

w

is

nonsingular

is

given

by

the

eigenvectors

of

matrix

w

b

As

we

already

pointed

out

the

approach

fails

if

w

is

singular

which

is

often

the

case

in

document

classification

Usually

this

problem

is

overcome

by

using

a

nonsingular

intermediate

space

of

w

obtained

by

removing

the

space

of

w

and

then

computing

eigenvectors

However

the

removal

of

the

space

of

w

possibly

eliminates

some

useful

information

because

some

of

the

most

discriminant

dimensions

may

be

lost

by

the

removal

In

fact

the

space

of

w

is

guaranteed

to

contain

useful

discriminant

information

when

the

projections

of

b

are

not

zeros

along

those

directions

Thus

simple

removal

of

the

space

of

w

is

not

an

effective

resolution

-LSB-

-RSB-

Once

the

transformation

has

been

determined

classification

is

performed

in

the

transformed

space

based

on

a

distance

metrics

such

as

Euclidean

distance

d

x

y

i

x

i

y

i

and

cosine

measure

d

x

y

i

x

i

y

i

i

x

i

i

y

i

A

new

instance

z

it

is

classified

to

argmin

k

d

z

x

k

-LRB-

-RRB-

where

x

k

is

the

centroid

of

k-th

class

In

fact

w

is

nonsingular

only

if

there

are

p

L

samples

This

is

usually

impractical

The

New

Criterion

We

propose

the

use

of

the

following

criterion

for

discriminating

inter-class

and

intra-class

distances

by

inter-class

and

intra-class

covariance

matrices

min

A

b

I

n

F

A

w

F

-LRB-

-RRB-

where

X

F

is

the

Frobenius

norm

of

the

matrix

X

i.e.

i

j

x

i

j

The

criterion

does

not

involve

the

inverse

of

the

intra-class

matrix

and

is

similar

to

Tikhonov

regularization

of

least

squares

problems

Intuitively

the

first

term

of

-LRB-

-RRB-

is

used

to

minimize

the

difference

between

the

projection

of

x

i

x

in

a

new

space

and

the

i-th

unit

vector

of

the

new

space

The

second

term

is

used

to

minimize

the

intra-class

covariance

The

equation

-LRB-

-RRB-

can

be

rewritten

as

min

A

w

A

b

I

n

F

-LRB-

-RRB-

and

this

is

a

least

squares

problem

with

the

solution

A

T

w

A

w

A

T

b

A

b

A

T

b

-LRB-

-RRB-

Here

we

will

show

how

to

use

GSVD

to

compute

efficiently

the

solution

to

the

optimization

problem

formulated

in

Section

and

show

that

the

solution

thus

obtained

is

stable

The

Basics

of

GSVD

Singular

value

decomposition

-LRB-

SVD

-RRB-

is

a

process

of

decomposing

a

rectangular

matrix

into

three

other

matrices

of

a

very

special

form

It

can

be

viewed

as

a

technique

for

deriving

a

set

of

uncor-related

indexing

variables

or

factors

-LSB-

-RSB-

A

Generalized

Singular

Value

Decomposition

-LRB-

GSVD

-RRB-

is

an

SVD

of

a

sequence

of

matrices

GSVD

has

played

a

significant

role

in

signal

processing

and

in

signal

identification

and

has

been

widely

used

in

such

problems

as

source

separation

stochastic

realization

and

generalized

Gauss-Markov

estimation

The

diagonal

form

of

GSVD

shown

below

was

first

introduced

in

-LSB-

-RSB-

T

HEOREM

-LRB-

GSVD

Diagonal

Form

-LSB-

-RSB-

-RRB-

If

A

R

m

p

B

R

n

p

and

rank

A

T

B

T

k

then

there

exist

two

orthogonal

matrices

U

R

m

m

and

V

R

n

n

and

a

non-singular

matrix

R

p

p

such

that

U

T

V

T

A

B

X

C

S

I

k

-LRB-

-RRB-

where

C

and

S

are

nonnegative

diagonal

and

of

dimension

m

k

and

n

k

respectively

S

S

min

n

k

min

n

k

and

C

T

C

S

T

S

I

k

The

generalized

singular

values

are

defined

to

be

the

component-wise

ratios

of

the

diagonal

entries

of

the

two

diagonal

matrices

In

signal

processing

A

is

often

the

signal

matrix

and

B

is

the

noise

matrix

in

which

case

the

generalized

singular

values

are

referred

to

as

signal-noise

ratios

Stable

Solution

By

plugging

the

GSVD

matrices

of

A

w

and

A

b

in

-LRB-

-RRB-

we

have

X

I

k

S

T

V

T

Since

V

is

orthogonal

we

can

drop

it

without

changing

the

squared

distance

So

we

have

X

I

k

S

T

-LRB-

-RRB-

This

derivation

of

holds

even

if

w

is

singular

Thus

by

using

GSVD

to

solve

the

new

criterion

we

can

avoid

removing

space

thereby

keeping

all

the

useful

information

The

degree

of

linear

independence

of

the

original

data

rank

A

T

w

A

T

b

is

equal

to

k

Since

R

p

k

rank

A

w

T

A

b

T

the

degree

of

linear

independence

in

the

transformed

space

is

at

most

k

We

now

state

a

theorem

that

shows

that

the

solution

is

stable

T

HEOREM

-LRB-

GSVD

relative

perturbation

bound

-LSB-

-RSB-

-RRB-

Suppose

A

and

B

be

matrices

with

the

same

number

of

columns

and

B

is

of

full

column

rank

Let

A

A

D

and

B

B

D

such

that

D

and

D

have

full

rank

Let

E

E

D

and

F

F

D

be

perturbations

of

A

and

B

respectively

such

that

for

all

x

there

exist

some

for

which

it

holds

that

E

x

A

x

F

x

B

x

Let

i

and

i

be

the

i-th

generalized

singular

value

of

A

B

and

that

of

A

E

B

F

respectively

Then

either

i

i

or

i

i

i

The

above

theorem

gives

a

bound

on

the

relative

error

of

the

generalized

eigenvalues

-LRB-

C

ii

and

S

ii

-RRB-

if

the

difference

between

the

estimated

covariance

matrices

and

the

genuine

covariance

matrices

is

small

This

guarantees

that

the

relative

error

of

is

bounded

by

the

relative

error

of

estimated

intra

and

inter-class

covariance

matrices

GSVD

also

brings

some

favorable

features

which

might

improve

accuracy

In

particular

computation

of

the

cross

products

A

T

b

A

b

and

A

T

w

A

w

which

causes

roundoff

errors

is

not

required

The

GDA

Algorithm

The

pseudo

codes

of

the

training

and

prediction

procedures

are

described

as

follows

Algorithm

Training

procedure

Train

-LRB-

x

's

-RRB-

Input

the

training

data

x

i

's

Output

the

transformation

begin

Construct

the

matrices

A

w

and

A

b

Perform

GSVD

on

the

matrix

pair

Obtain

as

described

in

equation

Return

end

Algorithm

Prediction

Procedure

T

Predict

-LRB-

x

-RRB-

Input

the

transformation

generated

by

the

training

procedure

and

a

new

instance

x

Output

the

label

T

of

the

new

instance

begin

Perform

Prediction

as

in

equation

Return

T

end

Here

we

show

that

the

above

transformation

derived

using

our

new

criterion

can

also

be

obtained

by

optimizing

the

trace

or

determinant

ratios

Optimizing

the

determinant

ratio

Fisher

's

criterion

is

to

maximize

the

ratio

of

the

determinant

of

the

inter-class

scatter

matrix

of

the

projected

samples

to

the

determinant

of

the

intra-class

scatter

matrix

of

the

projected

samples

J

T

b

T

w

-LRB-

-RRB-

One

way

to

overcome

the

requirements

of

non-singularity

of

Fisher

's

criterion

is

looking

for

solutions

that

simultaneously

maximize

T

b

minimize

T

w

Using

GSVD

A

b

and

A

w

are

decomposed

as

A

w

UC

I

k

X

and

A

b

V

S

I

k

X

To

maximize

J

T

b

should

be

increased

while

decreasing

T

w

Let

C

C

I

k

and

S

S

I

k

Then

we

have

b

A

T

b

A

b

X

S

X

and

w

A

T

w

A

w

XC

X

This

implies

T

b

T

X

S

X

S

X

and

T

w

T

XC

X

C

X

Thus

the

matrix

satisfying

X

I

k

would

simultaneously

maximize

T

b

and

minimize

T

w

-LRB-

since

the

diagonal

of

S

is

decreasing

-RRB-

So

we

have

X

I

k

In

the

case

where

we

must

weight

the

transformation

with

the

generalized

singular

X

I

k

S

T

is

the

transformation

we

want

Optimizing

the

trace

ratio

The

same

transformation

can

also

be

obtained

by

optimizing

the

trace

ratio

Using

GSVD

we

have

trace

T

b

trace

S

S

T

X

T

X

T

trace

S

S

T

GG

T

k

i

S

ii

g

ii

and

trace

T

w

trace

C

C

T

X

T

X

T

trace

C

C

T

GG

T

k

i

C

ii

g

ii

where

G

X

and

g

ii

is

the

ii-th

term

of

G

Since

C

T

C

S

T

S

I

k

we

have

trace

T

b

trace

T

w

k

i

S

ii

g

ii

k

i

C

ii

g

ii

k

i

g

ii

If

we

force

that

trace

T

b

the

optimization

is

formulated

as

minimization

of

trace

T

w

k

i

g

ii

Here

g

ii

's

are

diagonal

elements

of

a

positive

semi-definite

matrix

so

they

are

nonnegative

Also

for

all

i

g

ii

implies

that

for

all

j

g

i

j

g

ji

Note

that

GG

T

is

a

p

p

matrix

Since

only

the

first

k

diagonal

entries

g

ii

ki

appear

in

the

formula

for

trace

T

w

k

i

g

ii

the

quantities

of

other

m

k

diagonal

entries

do

not

affect

the

optimization

Thus

we

may

set

all

of

these

to

thereby

obtaining

X

I

k

In

the

case

when

we

want

to

weight

the

transformation

with

the

generalized

singular

values

we

obtain

X

I

k

S

T

A

well-known

transformation

method

in

information

retrieval

is

Latent

Semantic

Indexing

-LRB-

LSI

-RRB-

-LSB-

-RSB-

which

applies

Singular

Value

Decomposition

-LRB-

SVD

-RRB-

to

the

document-term

matrix

and

computes

eigenvectors

having

largest

eigenvalues

as

the

directions

related

to

the

dominant

combinations

of

the

terms

occurring

in

the

dataset

-LRB-

latent

semantics

-RRB-

A

transformation

matrix

constructed

from

these

eigenvectors

projects

a

document

onto

the

latent

semantic

space

Although

LSI

has

been

proven

extremely

useful

in

information

retrieval

it

is

not

optimal

for

text

categorization

because

LSI

is

com-pletely

unsupervised

In

other

words

LSI

deals

with

the

data

without

paying

any

particular

attention

to

the

underlying

class

structure

It

only

aims

at

optimally

transforming

the

original

data

into

a

lower

dimensional

space

with

respect

to

the

mean

squared

error

which

has

nothing

to

do

with

the

discrimination

of

the

different

classes

Our

GDA

approach

possesses

advantages

of

both

discriminant

analysis

and

of

latent

semantic

analysis

By

explic-itly

taking

the

intra-class

and

inter-class

covariance

matrices

into

the

optimization

criterion

GDA

deals

directly

with

discrimination

between

classes

Furthermore

by

employing

GSVD

to

solve

the

optimization

problem

GDA

tries

to

identify

the

latent

concepts

indicated

by

the

generalized

singular

values

To

illustrate

how

well

GDA

can

perform

we

present

here

two

examples

In

the

first

example

we

compare

GDA

against

LDA

and

LSI

Figure

shows

a

small

dataset

consisting

of

nine

phrases

in

three

topics

user

interaction

graph

theory

and

distributed

systems

No

Class

Phrase

Human

interface

for

user

response

A

survey

of

user

opinion

of

computer

system

response

time

Relation

of

user-perceived

response

time

to

error

measurement

The

generation

of

random

binary

unordered

trees

The

intersection

graph

of

paths

in

trees

Graph

Minors

IV

Widths

of

trees

and

well-quasi-ordering

A

survey

of

distributed

shared

memory

system

RADAR

A

multi-user

distributed

system

Management

interface

tools

for

distributed

computer

system

Figure

Nine

example

sentences

After

removing

words

-LRB-

terms

-RRB-

that

occurs

only

once

we

have

the

document-term

matrix

as

shown

in

Figure

The

first

and

second

samples

in

each

class

are

used

for

training

GDA

LDA

and

LSI

are

run

on

the

training

data

to

obtain

transformation

matrices

Figure

shows

the

plot

of

the

word

No.

a

computer

distributed

for

graph

interface

of

response

survey

system

the

time

trees

user

Figure

Document-term

Matrix

distances/similarities

between

document

pairs

in

the

transformed

space

using

each

of

the

three

methods

-LRB-

a

-RRB-

GDA

-LRB-

b

-RRB-

LDA

-LRB-

c

-RRB-

LSI

Figure

Pairwise

document

similarity

via

GDA

LDA

and

LSI

The

darker

the

close

is

the

more

similar

the

documents

are

GDA

is

a

clear

winner

The

second

example

illustrates

differences

between

GDA

and

LSI

Distinction

among

three

newsgroups

in

are

attempted

by

selecting

from

each

newsgroup

twenty

training

and

twenty

for

testing

Figure

shows

plots

of

the

the

sixty

testing

articles

using

the

two

dominant

directions

as

the

axes

GDA

has

clear

separation

while

the

LSI

plot

shows

an

L-shaped

concentration

of

the

data

points

The

confusion

matrices

of

these

methods

are

shown

in

Table

GDA

clearly

performed

better

than

LSI

prediction

prediction

actual

actual

Table

The

confusion

matrices

Left

GDA

Right

LSI

For

our

experiments

we

used

a

variety

of

datasets

most

of

which

are

frequently

used

in

the

information

retrieval

research

The

range

of

the

number

of

classes

is

from

four

to

and

the

range

of

the

number

of

documents

is

from

to

which

seem

varied

Group

Group

Group

-LRB-

a

-RRB-

GDA

Group

Group

Group

-LRB-

b

-RRB-

LSI

Figure

Document

plots

The

three

groups

are

separated

sig-nificantly

better

with

GDA

than

with

LSI

enough

to

obtain

good

insights

as

to

how

GDA

performs

Table

summarizes

the

characteristics

of

the

datasets

The

-LRB-

-RRB-

dataset

contains

approximately

articles

evenly

divided

among

Usenet

newsgroups

The

raw

text

size

is

MB

All

words

were

stemmed

using

a

porter

stemming

program

all

HTML

tags

were

skipped

and

all

header

fields

except

subject

and

organization

of

the

posted

article

were

ignored

WebKB

The

WebKB

dataset

contains

Web

pages

collected

from

university

computer

science

departments

There

are

approximately

documents

in

the

set

and

they

are

divided

into

seven

categories

student

faculty

staff

course

project

department

and

other

The

raw

text

size

of

the

dataset

is

MB

Among

the

seven

categories

student

faculty

course

and

project

are

the

four

most

populous

The

subset

consisting

only

of

these

categories

is

also

used

here

which

is

called

In

neither

of

the

datasets

we

used

stemming

or

stop

lists

Industry

Sector

The

Industry

Section

dataset

is

based

on

the

data

made

available

by

Market

Guide

Inc.

-LRB-

-RRB-

The

set

consists

of

company

homepages

that

are

categorized

in

a

hierarchy

of

industry

sectors

but

we

disregarded

the

hierarchy

There

were

documents

in

the

dataset

which

were

divided

into

classes

We

tokened

the

documents

by

skipping

all

MIME

and

HTML

headers

and

using

a

standard

stop

list

We

did

not

perform

stemming

Reuters

The

Text

Categorization

Test

Collection

contains

documents

collected

from

the

Reuters

newswire

in

It

is

a

standard

text

categorization

benchmark

and

contains

categories

We

used

its

subsets

one

consisting

of

the

ten

most

frequent

categories

which

we

call

and

the

other

consisting

of

documents

associated

with

a

single

topic

which

we

call

had

approximately

documents

and

categories

is

the

NIST

Topic

Detection

and

Tracking

text

corpus

version

released

in

December

-LSB-

-RSB-

This

corpus

contains

news

data

collected

daily

from

nine

news

sources

in

two

languages

-LRB-

American

English

and

Mandarin

Chinese

-RRB-

over

a

period

of

six

months

-LRB-

January

June

in

-RRB-

We

used

only

the

English

news

texts

which

were

collected

from

New

York

Times

Newswire

Service

Associated

Press

Worldstream

Service

Cable

News

Network

Voice

of

America

American

Broadcasting

Company

and

Public

Radio

International

The

documents

were

manu-ally

annotated

using

target

topics

We

selected

the

documents

having

annotated

topics

and

removed

the

brief

texts

The

resulting

Both

and

WebKB

are

available

at

Available

at

TextLearning/datasets.html

dataset

contained

documents

K-dataset

This

dataset

was

obtained

from

the

WebACE

project

-LSB-

-RSB-

It

contained

documents

consisting

of

news

articles

from

Reuters

News

Service

made

available

on

the

Web

in

October

These

documents

were

divided

into

classes

They

were

processed

by

eliminating

stop

words

and

HTML

tags

stemming

the

remaining

words

using

Porter

's

suffix-stripping

algorithm

CSTR

This

is

the

dataset

of

the

abstracts

of

technical

reports

published

in

the

Department

of

Computer

Science

at

the

University

of

Rochester

between

and

The

dataset

contained

abstracts

which

were

divided

into

four

research

areas

Symbolic-AI

Spatial-AI

Systems

and

Theory

We

processed

the

abstracts

by

removing

stop

words

and

applying

stemming

operations

on

the

remaining

words

Datasets

documents

class

WebKB

Industry

Sector

CSTR

K-dataset

Table

Data

Sets

Descriptions

Data

Preprocessing

In

all

experiments

we

randomly

chose

of

the

documents

for

training

and

assigned

the

rest

for

testing

It

is

suggested

in

-LSB-

-RSB-

that

information

gain

is

effective

for

term

removal

and

it

can

remove

up

to

or

more

of

the

unique

terms

without

performance

degrade

So

we

first

selected

the

top

words

by

information

gain

with

class

labels

The

feature

selection

is

done

with

the

Rainbow

package

-LSB-

-RSB-

Here

we

use

classification

accuracy

for

evaluation

Different

measures

such

as

precision-recall

graphs

and

F

measure

-LSB-

-RSB-

have

been

used

in

the

literature

However

since

the

datasets

used

in

our

experiments

are

relatively

balanced

and

single-labeled

and

our

goal

in

text

categorization

is

to

achieve

low

misclassification

rates

and

high

separation

between

different

classes

on

a

test

set

we

thought

that

accuracy

is

the

best

measure

of

performance

All

of

our

experiments

were

carried

out

on

a

GHz

machine

with

memory

running

Linux

Experimental

Results

Now

we

present

and

discuss

the

experimental

results

Here

we

compare

GDA

against

Naive

Bayes

-LRB-

NB

for

short

-RRB-

K-Nearest

Neighbor

-LRB-

KNN

for

short

-RRB-

Maximum

Entropy

-LRB-

ME

for

short

-RRB-

LDA

and

SVM

on

the

same

datasets

with

the

same

training

and

testing

data

Recall

that

the

first

three

of

the

methods

we

compare

against

are

commonly-used

direct

methods

for

multi-class

classification

-LRB-

in

the

sense

that

they

do

not

require

reduction

to

binary

classification

problems

-RRB-

For

experiments

involving

SVM

we

used

SVMTorch

-LSB-

-RSB-

which

uses

the

one-versus-the-rest

decomposition

Table

and

Figure

show

performance

comparisons

GDA

outperformed

all

the

other

five

methods

on

WebKB

and

Industry

Sector

SVM

performed

the

best

on

The

TRs

are

available

at

Download-able

at

K-dataset

and

GDA

outperformed

LDA

on

all

the

experiments

and

the

improvement

was

significant

-LRB-

more

than

-RRB-

when

the

sample

size

was

relatively

small

-LRB-

in

the

case

of

CSTR

and

K-dataset

-RRB-

On

the

performance

of

GDA

s

which

is

approximately

higher

than

that

of

NB

higher

than

that

of

ME

and

higher

than

that

of

SVM

On

the

dataset

GDA

beats

NB

by

approximately

and

both

ME

and

SVM

by

approximately

On

the

WebKB

dataset

GDA

beats

NB

by

approximately

and

ME

by

The

performance

of

GDA

is

about

higher

than

that

of

NB

and

by

than

that

of

ME

on

the

Industry

Sector

The

results

with

GDA

and

with

SVM

are

almost

the

same

on

WebKB

Industry

Sector

and

CSTR

On

K-dataset

and

SVM

performs

slightly

better

than

GDA

by

ME

achieves

the

best

results

on

the

CSTR

dataset

while

NB

is

the

winner

on

in

terms

of

performance

On

CSTR

the

performance

of

GDA

is

lower

than

that

of

NB

and

lower

than

that

of

ME

On

GDA

is

beaten

by

NB

by

approximately

In

total

the

performance

of

GDA

is

always

either

the

winner

or

very

close

to

the

winner

it

is

ranked

the

first

four

times

ranked

the

second

three

times

and

ranked

the

third

in

the

remaining

two

Although

there

is

no

single

winner

over

all

datasets

GDA

seems

to

outperform

the

rest

on

most

counts

We

can

say

that

GDA

is

a

viable

competitive

algorithm

in

text

categorization

Datasets

GDA

NB

KNN

ME

LDA

SVM

WebKB

Industry

Sector

CSTR

K-dataset

Table

Performance

comparisons

For

KNN

we

set

k

to

ewsg

roup

s

Web

KB

Web

KB

Ind

ustr

y

S

ecto

r

Reu

ters

top

Reu

ters

URCS

K-d

atas

et

TDT

GDA

NB

KNN

ME

LDA

SVM

Figure

Performance

Comparison

GDA

is

also

very

efficient

and

most

experiments

are

done

in

several

seconds

Table

summarizes

the

running

time

for

all

the

experiments

of

GDA

and

SVM

Figure

and

Figure

present

the

comparisons

of

training

and

prediction

time

respectively

The

time

saving

of

GDA

is

very

obvious

In

summary

these

experiments

have

shown

that

GDA

provides

an

alternate

choice

for

fast

and

efficient

text

categorization

GDA

GDA

SVM

SVM

Datasets

Training

Prediction

Training

Prediction

WebKB

Industry

Sector

CSTR

K-dataset

Table

Time

Table

in

seconds

ewsg

rou

ps

We

bKB

We

bKB

Ind

ustry

Sect

or

Reu

ters

top

Reu

ters

CST

R

K-d

ata

set

TDT

Training

Time

GDA

SVM

Figure

Training

Time

Comparisons

ewsg

rou

ps

We

bKB

We

bKB

Ind

ustry

Sect

or

Reu

ters

top

Reu

ters

CST

R

K-d

ata

set

TDT

Prediction

Time

GDA

SVM

Figure

Prediction

Time

Comparisons

In

this

paper

we

presented

GDA

a

simple

efficient

and

yet

accurate

direct

approach

to

multi-class

text

categorization

GDA

utilizes

GSVD

to

transform

the

original

data

into

a

new

space

which

could

reflect

the

inherent

similarities

between

classes

based

on

a

new

optimization

criterion

Extensive

experiments

clearly

demonstrate

its

efficiency

and

effectiveness

Interestingly

enough

although

traditional

discriminant

approaches

have

been

successfully

applied

in

pattern

recognition

little

work

has

been

reported

on

document

analysis

As

we

mentioned

earlier

this

is

partly

because

the

intra-class

covariance

matrix

is

usually

singular

for

document-term

data

and

hence

restrict

the

usage

of

discriminant

Our

new

criterion

avoids

the

problem

while

still

preserving

the

discriminative

power

of

the

covariance

matrix

Another

big

barrier

to

application

of

discriminant

analysis

in

document

classification

is

its

large

computation

cost

As

we

know

traditional

discriminant

analysis

requires

a

large

amount

of

computation

on

matrix

inversion

SVD

and

eigenvalue-analysis

The

costs

of

these

operations

are

extremely

large

in

document

analysis

because

the

matrices

have

thousands

of

dimension

Our

approach

makes

use

of

effective

feature

selection

via

information

gain

with

which

we

can

remove

up

to

or

more

of

the

unique

terms

without

significant

performance

degrade

-LSB-

-RSB-

One

of

our

future

plans

is

to

explore

how

the

performance

correlates

with

different

feature

selection

methods

and

the

number

of

words

selected

There

are

also

other

possible

extensions

such

as

using

random

projection

to

reduce

the

dimensionality

before

applying

discriminant

analysis

-LSB-

-RSB-

This

work

is

supported

in

part

by

NSF

grants

and

and

NIH

grant

-LSB-

-RSB-

Allwein

E.

L.

Schapire

R.

E.

Singer

Y.

-LRB-

-RRB-

Reducing

multiclass

to

binary

A

unifying

approach

for

margin

classifiers

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Apte

C.

Damerau

F.

Weiss

S.

-LRB-

-RRB-

Text

mining

with

decision

rules

and

decision

trees

Proceedings

of

the

Workshop

with

Conference

on

Automated

Learning

and

Discovery

Learning

from

text

and

the

Web

-LSB-

-RSB-

Chakrabarti

S.

Roy

S.

Soundalgekar

M.

V.

-LRB-

-RRB-

Fast

and

accurate

text

classification

via

multiple

linear

discriminant

projections

Proceedings

of

the

International

Conference

on

Very

Large

Databases

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Cohen

W.

W.

Singer

Y.

-LRB-

-RRB-

Context-sensitive

learning

methods

for

text

categorization

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Collobert

R.

Bengio

S.

-LRB-

-RRB-

SVMTorch

Support

vector

machines

for

large-scale

regression

problems

Journal

of

Machine

Learning

Research

-LSB-

-RSB-

Deerwester

S.

C.

Dumais

S.

T.

Landauer

T.

K.

Furnas

G.

W.

Harshman

R.

A.

-LRB-

-RRB-

Indexing

by

latent

semantic

analysis

Journal

of

the

American

Society

of

Information

Science

-LSB-

-RSB-

Demmel

J.

Veseli

c

K.

-LRB-

-RRB-

Jacobi

's

method

is

more

accurate

than

QP

SIAM

Journal

on

Matrix

Analysis

and

Applications

-LSB-

-RSB-

Dietterich

T.

G.

Bakiri

G.

-LRB-

-RRB-

Solving

multiclass

learning

problems

via

error-correcting

output

codes

Journal

of

Artificial

Intelligence

Research

-LSB-

-RSB-

Dumais

S.

Platt

J.

Heckerman

D.

Sahami

M.

-LRB-

-RRB-

Inductive

learning

algorithms

and

representations

for

text

categorization

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Fisher

R.

-LRB-

-RRB-

The

use

of

multiple

measurements

in

taxonomic

problems

Annals

of

Eugenics

-LSB-

-RSB-

Fragoudis

D.

Meretakis

D.

Likothanassis

S.

-LRB-

-RRB-

Integrating

feature

and

instance

selection

for

text

classification

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Fukunaga

K.

-LRB-

-RRB-

Introduction

to

statistical

pattern

recognition

Academic

Press

-LSB-

-RSB-

Ghani

R.

-LRB-

-RRB-

Using

error-correcting

codes

for

text

classification

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Godbole

S.

Sarawagi

S.

Chakrabarti

S.

-LRB-

-RRB-

Scaling

multi-class

support

vector

machine

using

inter-class

confusion

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Han

E.-H.

Boley

D.

Gini

M.

Gross

R.

Hastings

K.

Karypis

G.

Kumar

V.

Mobasher

B.

Moore

J.

-LRB-

-RRB-

WebACE

A

web

agent

for

document

categorization

and

exploration

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Hastie

T.

Tibshirani

R.

-LRB-

-RRB-

Classification

by

pairwise

coupling

Advances

in

Neural

Information

Processing

Systems

The

MIT

Press

-LSB-

-RSB-

Joachims

T.

-LRB-

-RRB-

Making

large-scale

support

vector

machine

learning

practical

In

Advances

in

kernel

methods

Support

vector

machines

-LSB-

-RSB-

Joachims

T.

-LRB-

-RRB-

A

statistical

learning

model

of

text

classification

with

support

vector

machines

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Lam

W.

Ho.

C.

-LRB-

-RRB-

Using

a

generalized

instance

set

for

automatic

text

categorization

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Lewis

D.

D.

-LRB-

-RRB-

Naive

-LRB-

Bayes

-RRB-

at

forty

The

independence

assumption

in

information

retrieval

-LSB-

-RSB-

Loan

C.

V.

-LRB-

-RRB-

Generalizing

the

singular

value

decomposition

SIAM

J.

Num

Anal.

-LSB-

-RSB-

Masand

B.

Linoff

G.

Waltz.

D.

-LRB-

-RRB-

Classifying

news

stories

using

memory

based

reasoning

-LRB-

pp.

-RRB-

-LSB-

-RSB-

McCallum

A.

K.

-LRB-

-RRB-

Bow

A

toolkit

for

statistical

language

modeling

text

retrieval

classification

and

clustering

mccallum/bow

-LSB-

-RSB-

Mika

S.

R

atsch

G.

Weston

J.

Sch

olkopf

B.

M

uller

K.-R

-LRB-

-RRB-

Fisher

discriminant

analysis

with

kernels

Neural

Networks

for

Signal

Processing

IX

-LRB-

pp.

-RRB-

IEEE

-LSB-

-RSB-

Ng

H.

T.

Goh

W.

B.

Low

K.

L.

-LRB-

-RRB-

Feature

selection

perceptron

learning

and

a

usability

case

study

for

text

categorization

Proceedings

of

the

Annual

International

ACM

SIGIR

Conference

on

Research

and

Development

in

Information

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Nigam

K.

Lafferty

J.

McCallum

A.

-LRB-

-RRB-

Using

maximum

entropy

for

text

classification

In

Workshop

on

Machine

Learning

for

Information

Filtering

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Papadimitriou

C.

H.

Tamaki

H.

Raghavan

P.

Vempala

S.

-LRB-

-RRB-

Latent

semantic

indexing

A

probabilistic

analysis

Proceedings

of

the

Symposium

on

Principles

of

Database

Systems

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Schapire

R.

E.

Singer

Y.

-LRB-

-RRB-

Boostexter

A

boosting-based

system

for

text

categorization

Machine

Learning

-LSB-

-RSB-

Scholkopf

B.

J.Smola

A.

-LRB-

-RRB-

Learning

with

kernels

MIT

Press

-LSB-

-RSB-

-LRB-

-RRB-

Nist

topic

detection

and

tracking

corpus

-LSB-

-RSB-

Tzeras

K.

Hartmann

S.

-LRB-

-RRB-

Automatic

indexing

based

on

Bayesian

inference

networks

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Vapnik

V.

N.

-LRB-

-RRB-

Statistical

learning

theory

Wiley

New

York

-LSB-

-RSB-

Wiener

E.

D.

Pedersen

J.

O.

Weigend

A.

S.

-LRB-

-RRB-

A

neural

network

approach

to

topic

spotting

Annual

Symposium

on

Document

Analysis

and

Information

Retrieval

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Yang

Y.

Liu

X.

-LRB-

-RRB-

A

re-examination

of

text

categorization

methods

-LRB-

pp.

-RRB-

-LSB-

-RSB-

Yang

Y.

Pederson

J.

O.

-LRB-

-RRB-

A

comparative

study

on

feature

selection

in

text

categorization

-LRB-

pp.

-RRB-

O O B I O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O B I O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O

text categorization
gsvd
gda
discriminant analysis
svms

multi-class classification
multi-class text categorization