Enforcing Security and Safety Models with an Information Flow Analysis Tool Existing security models require that information of a given security level be prevented from `` leaking '' into lower-security information . High-security applications must be demonstrably free of such leaks , but such demonstration may require substantial manual analysis . Other authors have argued that the natural way to enforce these models automatically is with information-flow analysis , but have not shown this to be practicable for general purpose programming languages in current use . Modern safety-critical systems can contain software components with differing safety integrity levels , potentially operating in the same address space . This case poses problems similar to systems with differing security levels ; failure to show separation of data may require the entire system to be validated at the higher integrity level . In this paper we show how the information flow model enforced by the SPARK Examiner provides support for enforcing these security and safety models . We describe an extension to the SPARK variable annotations which allows the specification of a security or safety level for each state variable , and an extension to the SPARK analysis which automatically enforces a given information flow policy on a SPARK program . D2 .4 -LSB- Software Engineering -RSB- : Software/Program verification ; F3 .1 -LSB- Logics and Meanings of Programs -RSB- : Specifying and verifying and reasoning about programs Design , Measurement , Security , Verification Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page . To copy otherwise , to republish , to post on servers or to redistribute to lists , requires prior specific permission and/or a fee . SIGAda ' 04 , November 14 18 , 2004 , Atlanta , Georgia , USA . Copyright 2004 ACM 1-58113-906-3 / 04/0011 ... $ 5.00 . Software is often used to develop security-critical applications . Some of these applications are required to manage information of different classification levels , ensuring that each user may only access the data for which he or she has adequate authorisation . This requirement has two components ; the developer must produce a design and implementation which supports this model and its constraints , and the implementation must support verification that the constraints are never violated . It is not enough for the implementation to be correct ; for it to be secure , it must be seen to be correct . In this paper we examine the problem of developing and verifying a security-critical application containing this security structure -LRB- often termed multi-level security -RRB- . We will analyse recent work on information flow and security and show how the information flow analysis of the SPARK Examiner tool is appropriate for solving this problem . We will show how it is also important for the efficient analysis of safety-critical systems . We will then describe modifications to the current definition of the SPARK Ada language -LSB- 2 -RSB- and the Examiner which permit complete validation of Ada programs against defined security models such as the Bell-LaPadula model -LSB- 3 -RSB- . We intend that the additional flow analysis features described here will appear in a future commercial Examiner release . Although we anticipate possible minor changes to the syntax and analysis performed , we expect the released version to implement the analysis described here in all substantial aspects . In this section we identify typical standards which will inform the development and verification of a security-critical or safety-critical application . We then analyse a survey paper on information flow and security , and compare its conclusions against the information flow model of the SPARK annotated subset of Ada95 . 2.1 Standards The Common Criteria for IT Security -LSB- 5 -RSB- specify the development and verification activities that are suitable for applications at differing levels of security . These Evaluation Assurance Levels -LRB- EALs -RRB- range from EAL-1 -LRB- lowest -RRB- to EAL-7 -LRB- highest -RRB- . As the EAL number rises , the required activities become more rigorous ; at the higher levels , formal specification and analysis of the software becomes required . 39 For safety-related applications there are similar concepts for the safety criticality of software ; RTCA DO-178B -LSB- 10 -RSB- for civil avionics software defines criticality levels E -LRB- lowest -RRB- through to A -LRB- most critical -RRB- . As one would expect , the required development and verification activities become increasingly more onerous -LRB- and expensive -RRB- with increasing criticality level . As a result , if an avionics system were to contain a `` core '' of critical functionality at Level A and a larger body of utility code at Level D then either the entire system would have to be developed and verified at Level A or a rigorous argument would have to be applied that the Level D code could not in any way affect the integrity of the Level A code . Another notation for safety criticality is the Safety Integrity Level -LRB- SIL -RRB- , described in IEC 61508 -LSB- 8 -RSB- . SIL-1 is the lowest level of safety integrity , and SIL-4 the highest ; DO-178B level A approximates to SIL-3 / SIL-4 when comparing the development and verification activities required . In this paper we assume that we are attempting to validate systems at EAL-5 or greater -LRB- for security -RRB- and RTCA Level B or greater -LRB- for safety -RRB- . We are therefore required to provide a rigorous and comprehensive justification for any statements which we make about the separation of data . Therefore we now look at how such statements may be expressed . 2.2 Information Flow `` Information flow '' as it applies to conventional imperative computer programs has a range of definitions , and is often confused with `` data flow '' ; we shall take the definition as expressed in Barnes -LSB- 2 -RSB- p. 13 : data flow analysis is concerned with the direction of data flow ; whereas information flow analysis also considers the coupling between variables . An example of the difference between data flow and information flow comes from the following -LRB- Ada -RRB- code : while -LRB- X < 4 -RRB- loop A : = A + 2 * B ; X : = X * 2 ; end loop ; Here , data flow analysis would state that data flows from X to X and from A and B to A. Information flow analysis would note additionally that the final value of A is affected by the initial value of X , and hence that there is information flow from X to A. Sabelfeld and Myers -LSB- 11 -RSB- recently surveyed the use of information flow analysis in software . They viewed the fundamental problem of maintaining security as one of tracking the flow of information in computing systems . They examined the various ways that secure information could leak into less secure information , overtly and covertly . They identify in particular the concept of implicit information flows , an example of which is given in the while-loop example above . Their survey characterised language-based information flow as requiring : 1 . semantics-based security , so that rigorous argument could be made about a variation of a high-security value not affecting a low-security value ; and 2 . a security type system , so that a program or expression can be assigned security values -LRB- types -RRB- and the type changes of the program or expression can be characterised . They concluded that `` standard '' security practices do not and can not enforce the end-to-end confidentiality required by common security models . They characterised the existing work in security and information flow analysis , but notably did not address the work of Bergeretti and Carr e -LSB- 4 -RSB- . 2.3 Security models The Bell-LaPadula -LRB- BLP -RRB- model of computer security -LSB- 3 -RSB- enforces two properties : 1 . no process may read data from a higher security level ; and 2 . no process may write data to a lower security level . Such multi-level security has a number of problems ; Anderson -LSB- 1 -RSB- provides a list of them including : `` blind write-up '' : the inability to inform low-security data whether a write to high-security data has happened correctly ; `` downgrading '' : moving information from a high security level to a lower level is sometimes desirable ; and `` TCB bloat '' : a large subset of the operating system may end up in the Trusted Computing Base -LRB- TCB -RRB- . The Dolev-Yao security model -LSB- 6 -RSB- makes secret information indivisible ; it can not be leaked in part but only in total . 2.4 Information Flow in Ada Bergeretti and Carr e wrote a seminal paper -LSB- 4 -RSB- describing a practical implementation of information flow analysis in the SPADE Pascal language -LRB- although the principles were applicable to most conventional imperative programming languages -RRB- . This was managed by the composition of matrices representing information flow dependencies between variable imports and exports . Notably , conditional and infinite loops were permissible and analysable within the framework of such a language ; these were managed by computing the transitive closure of the information flow matrix corresponding to one execution of the loop . This information flow model was implemented in the SPARK annotated Ada subset -LSB- 2 -RSB- . The subset requires each subprogram to be annotated with the required information flow -LRB- `` derives '' annotations -RRB- if information flow analysis is required . The SPARK subset is enforced by the SPARK Examiner tool which checks the required information flow of each subprogram against the actual information flow . The annotations -LRB- and other SPARK rules such as the ban on circular dependency -RRB- are necessary to make this information flow analysis tractable . The result of this is that it is possible , in a fully-analysed SPARK program , to be certain that a given exported variable is independent of a given imported variable . This is a key step towards supporting multi-level security in SPARK Ada , and makes it easier to write demonstrably secure and correct code , but is not a complete solution . In Section 3 we will describe how SPARK Examiner analysis may be extended better to implement these checks . We now examine case studies of the use of SPARK , and the utility of information flow in real applications . 40 2.5 Security and Safety Applications An example of a high-security application which was partly developed in SPARK is the MULTOS CA -LSB- 7 -RSB- . This was developed and verified at a high level of integrity , approximating to the Common Criteria EAL-5 . The delivered system had a defect rate of 0.04 errors per thousand lines of code -LRB- KLOC -RRB- . This showed that SPARK was a practical language for implementing high-security applications . Part of the analysis required during development and verification of the CA was to show that secret data could not leak out directly or indirectly in unclassified outputs . An example of a safety-critical application with mixed integrity levels where information flow analysis was helpful was the SHOLIS information system described by King et al. -LSB- 9 -RSB- . This application mixed SIL-4 and non-critical code , and was written in the SPARK subset of Ada83 . Static verification was used , including information flow analysis and partial program proof , to verify significant properties of SHOLIS . There was a successful argument based partly on information flow analysis that the high-SIL code was not compromised by the low-SIL code ; however , this argument had to be made manually , based on the validated information flow of each subprogram , as the Examiner did not provide such tracing facilities at the program level . The outstanding need in SPARK is to be able to mark state variables in packages with a -LRB- numerical -RRB- level of security and / or safety . This has been implemented by allowing an optional aggregate after a package -LRB- `` own '' -RRB- variable declaration . 3.1 Marking security levels Suppose that a package Classify was defined as follows to create various secrecy levels : package Classify is -- Security levels UNCLASSIFIED : constant : = 0 ; RESTRICTED : constant : = 1 ; CONFIDENTIAL : constant : = 2 ; SECRET : constant : = 3 ; TOPSECRET : constant : = 4 ; end Classify ; Then we define a package KeyStore to store and manage a symmetric encryption key SymmetricKey , designed to mutate after each encryption according to a rotation parameter RotorValue . The key is clearly a high-security data item , with the rotor value requiring less security . KeyStore marks its state variables with the field Integrity 1 thus : -- # inherit Classify ; package KeyStore -- # own SymmetricKey -LRB- Integrity = > Classify.SECRET -RRB- ; -- # RotorValue -LRB- Integrity = > Classify.RESTRICTED -RRB- ; is procedure Rotate ; -- # global in RotorValue ; -- # in out SymmetricKey ; 1 Integrity was chosen to make sense for both security and safety applications -- # derives SymmetricKey from -- # SymmetricKey , RotorValue ; procedure Encrypt -LRB- C : in MessageBlock ; E : out MessageBlock -RRB- ; -- # global in SymmetricKey ; -- # derives E from C , SymmetricKey ; ... end KeyStore ; Any security analysis must show , in the case of this code , that SymmetricKey data can not leak into RotorValue or data ; i.e. there must be no subprogram -LRB- or main program -RRB- whose information annotation shows SymmetricKey as an import to RotorValue . Note that in this case package Classify will need to be inherited by all relevant package , but will never be withed and so never compiled . 3.2 Implementation We now define how the integrity levels are marked in the SPARK language , and how they are enforced by the SPARK Examiner . The extra information flow checking is invoked using the / policy = X command line switch to the Examiner . Current valid policy values are security and safety . 3.2.1 Variable declaration The above Integrity property on package own variables is a static property ; at any point in static analysis of the code , the actual and required integrity level of an own variable is known . Own variables without Integrity levels are taken to have a default integrity level of Natural ` Last -LRB- i.e. very highly classified -RRB- if imported under a security policy , and Natural ` First -LRB- i.e. unclassified -RRB- if exported under a security policy . This gives the most paranoid checking so that data may not leak from an input to an output through an intermediate variable with unspecified integrity . If the analysis policy is safety then the default integrity values for unspecified own variables are reversed . 3.2.2 Integrity checks Information flow is declared explicitly in derives annotations for subprograms ; an example from our cryptographic KeyStore is the Rotate subprogram which changes the key based on the value of RotorValue : procedure Rotate ; -- # global in RotorValue ; -- # in out SymmetricKey ; -- # derives SymmetricKey from -- # SymmetricKey , RotorValue ; Using policy security the Examiner will then check that the integrity level of the export SymmetricKey -LRB- SECRET -RRB- is no less than the integrity levels of any import -LRB- SECRET and RESTRICTED -RRB- . With policy safety the checks would be that the integrity of the export is no more than the integrity levels of any import , i.e. that high-safety data exports can not be contaminated by low-safety data imports . Information flow is also checked at each procedure call by substituting in actual parameters -LRB- which may be own variables -RRB- for formal parameters and rechecking the procedure 's known information flow for integrity flows . As examples we 41 give the procedures which get and set the rotor and key values : procedure GetKey -LRB- This_Key : out Key -RRB- ; -- # global in SymmetricKey ; -- # derives This_Key from SymmetricKey ; procedure GetRotor -LRB- This_Rotor : out Rotor -RRB- ; -- # global in RotorValue ; -- # derives This_Rotor from RotorValue ; procedure SetKey -LRB- New_Key : in Key -RRB- ; -- # global out SymmetricKey ; -- # derives SymmetricKey from New_Key ; procedure SetRotor -LRB- New_Rotor : in Rotor -RRB- ; -- # global out RotorValue ; -- # derives RotorValue from New_Rotor ; Because RotorValue is RESTRICTED and SymmetricKey is SECRET , the analysis requires a check at each invocation of these subprograms that the actual parameters mapped to formal parameters do not violate integrity flow checks . 3.2.3 Analysis techniques not adopted One analysis technique which we considered -LRB- but rejected -RRB- was to track the integrity flows within each subprogram body and raise warnings at each individual statement where a violation occurs . We now explain how this would have worked and why we rejected it . In the following code , the programmer generates a key and rotor , encrypts a message with it , then tries to create a new rotor based on the old key . -- Key and rotor generation R1 : = KeyStore.MakeRotor -LRB- 34,56,22,55 -RRB- ; KeyStore.SetRotor -LRB- R1 -RRB- ; K1 : = KeyStore.MakeKey -LRB- 66,11,2,4 -RRB- ; KeyStore.SetKey -LRB- K1 -RRB- ; -- Encrypt a message KeyStore.Encrypt -LRB- C = > Clear , E = > Encrypted -RRB- ; -- Get a copy of the -LRB- changed -RRB- key and break - it down into data KeyStore.GetKey -LRB- K1 -RRB- ; KeyStore.DecomposeKey -LRB- K1 , I1 , I2 , I3 , I4 -RRB- ; -- Build a new rotor R1 : = KeyStore.MakeRotor -LRB- I1 , I2 , I3 , I4 -RRB- ; KeyStore.SetRotor -LRB- R1 -RRB- ; The statement-by-statement information flow would proceed as shown in Table 1 , where C denotes Clear , RV denotes RotorValue , SK denotes SymmetricKey , and E denotes Encrypted . The final step causes RotorValue to exceed its assigned integrity level , and would generate a static integrity flow error . The problem with this technique arises from the need to track the integrity levels of local variables . It quickly becomes clear that for practical analysis reasons each local variable involved needs to be assigned an integrity level . This is possible , and is done by declaring them as own variables with integrity levels in a package embedded in the subprogram in question , but is cumbersome . It also requires substantial rework of any existing code which we may want to retro-analyse . R1 K1 SK RV I1 C E Instruction 0 0 1 0 0 1 R1 : = 0 0 1 1 SetRotor 0 0 1 1 K1 : = 0 0 3 1 1 SetKey 0 0 3 1 1 3 Encrypt 0 3 3 1 1 3 GetKey 0 3 3 1 3 1 3 Decompose 3 3 3 1 3 1 3 R1 : = 3 3 3 3 3 1 3 SetRotor Table 1 : Information flow for example 3.2.4 Example of checking We placed the code described above into procedure Operate in a package Crypto and annotated the declaration with the correct derives annotation thus : -- # inherit KeyStore , Classify , BitString ; package Crypto -- # own Clear , -- # Encrypted -LRB- Integrity = > Classify.SECRET -RRB- ; is procedure Operate ; -- # global out KeyStore.RotorValue , Encrypted ; -- # in out KeyStore.SymmetricKey ; -- # in Clear ; -- # derives -- # KeyStore.SymmetricKey , -- # KeyStore.RotorValue -- # from -- # KeyStore.SymmetricKey -- # & -- # Encrypted -- # from -- # Clear , -- # KeyStore.SymmetricKey -- # ; end Crypto ; Analysis using / policy = security gives the following static semantic error : Unit name : Crypto Unit type : package specification Unit has been analysed , any errors are listed below 1 error -LRB- s -RRB- or warning -LRB- s -RRB- Line 30 -- # KeyStore.SymmetricKey ^ 1 *** -LRB- 1 -RRB- Semantic Error : 175 : Information flow from KeyStore.SymmetricKey to KeyStore.RotorValue violates the selected information flow policy . . which correctly identifies the potential leak . 3.3 Case study SHOLIS is the Royal Navy 's Ship Helicopter Operating Limits Information System -LSB- 9 -RSB- designed to assist landing of 42 helicopters on Royal Navy Type 23 frigates . Failure of this system could result in the death of helicopter pilots and passengers , loss of a helicopter and damage to the ship . This is intolerable for normal operation , hence SIL-4 reliability is required to give sufficient confidence that such an accident will not happen during the in-service lifetime of the system . SHOLIS is an on-demand system rather than a continuously operating system , and so has a required probability of failure to function on demand of approximately 10 -4 ; a more precise probability would be specified in the system safety case . However , the bulk of the SHOLIS code does not relate to critical system functionality . The code specific to SIL-4 must be analysed at a deep level ; the rest of the code can be analysed less deeply as long as it can be shown not to affect the SIL-4 data adversely . 3.3.1 Original analysis The original SHOLIS code is Ada 83 and consists of 75 source files and shadow files amenable to SPARK analysis ; 26.5 KLoC of non-blank non-comment non-annotation code . It is a substantial program and therefore a suitable test to see if integrity level checking scales . Without any own variables annotated , a full SPARK analysis by an Examiner with policy = safety generated no integrity errors , as we would expect . 3.3.2 Identifying critical outputs To enable easy marking of variable safety criticality we added a single package Safety : package Safety is NSC : constant : = 0 ; SC : constant : = 1 ; end Safety ; We took as an example the safety-critical output Alarm2Z in a package RMR which represents an alarm signal on the front panel . This was annotated : -- # own Alarm2Z : BasicTypes.OkFailT -- # -LRB- Integrity = > Safety.SC -RRB- ; A re-analysis of package RMR raised no integrity errors . The other package that used Alarm2Z was EVENT which was the main event handler . A SPARK of this package raised integrity flow errors in subprogram Sync , where Alarm2Z depended on a large range of other inputs , which had not yet been marked as safety-critical . This was what we would expect so far and confirmed that basic safety integrity analysis was working . 3.3.3 Extending analysis The next phase of work started in the SENSOR package which was near the middle of the package hierarchy . We set the package variables representing current speed , heading , roll , pitch and wind velocity state to be safety-critical and then ran a trial SPARK analysis . This indicated many own variables in this and other packages which caused integrity flow errors since they had no explicit integrity level . For each of these packages in turn , we : 1 . marked all of the package own variables as non-critical -LRB- Safety.NSC -RRB- ; 2 . re-analysed the package specification ; 3 . analysed the package body to ensure that there were no integrity flow errors at subprogram call points ; and 4 . if necessary , transformed NSC variables to SC status and re-ran . Eventually we converged on a stable SC/NSC partition of the variables . 3.3.4 Declassification The DisplayBuffers state variable in I/O package sio5 was a point where safety-critical and non-safety critical data merged . It was necessary during the actual project to produce a manual justification that the buffer would never be filled with non-safety critical data when there was safety-critical data to be added and displayed to a user . We therefore set its integrity to NSC and ignored all integrity errors relating to flows from SIO5.DisplayBuffers . 3.3.5 Results There were 1282 integrity flow errors , but every single one of these referred to a flow from SIO5.DisplayBuffers to Fault.Faults , as expected . Therefore only one manual argument is needed to validate the separation of SC and NSC data at this level . Of the 233 package specification variables which were given integrity levels , 110 were NSC and 123 were SC . 3.3.6 Lessons learned SHOLIS was developed using a now out-of-date version of the SPARK Examiner which did not support proof work involving abstract state ; as a consequence there were many more public own variables than you would expect in a well-designed modern SPARK program . This made the conversion work slower ; at the top level , as noted above , it in-creased the time required beyond what was available for the study . The `` TCB bloat '' problem noted in Section 2.3 did not seem to be a problem . While working up the calling hierarchy there was a small amount of returning to lower levels to make NSC variables into SC , but this did not spread out of control . There is a clear need for a declassification mechanism , as discussed in more detail in Section 4.3 . Being able to suppress the integrity flow errors from DisplayBuffers would have made the transformation process easier . 3.4 Possible tactical extensions Given the preceding work , it is relatively simple to extend the own variable annotation to allow other fields in the aggregate . Within the security domain , there are considerations which mean that security can not be considered on a linear scale . An example is a set of documents on an international military aviation development where markings might include NATO RESTRICTED , UK RESTRICTED and ICELAND RESTRICTED . They are all at the same level of security , but apply to different nationalities in different ways . A UK citizen could receive the first two , a German could receive the first one only , and a Russian could not receive any . The nationality information could be represented by an additional field , which might be an array of booleans mapping each country code to an Allowed/Forbidden code . 43 3.5 Security policies So far we have considered enforcing the Bell-LaPadula security policy . However , there are other policies which may be desirable for enforcement . One example is a policy where information at security level N may only flow into other information at security level N ; there is no concept of ordering on these security levels , they may simply not be mixed . There is further work to be done on investigating whether other information flow policies are desirable and useful for security-critical or safety-critical code . In principle they should not be complicated to enforce . The / policy = X command line switch provides a hook to specify different policies in future . In this section we discuss the limitations of the existing work and examine how the analysis techniques may be extended in the future . 4.1 Difficulties with analysis The concept of `` label creep '' as identified by Sabelfeld and Myers refers to the tendency of data to increase in security level as program execution progresses ; assigning a SECRET value to one field of a large CONFIDENTIAL record will make the entire record SECRET . It remains to be seen how SPARK security programs should be designed in order to minimise label creep . Concurrency is more complex because there is the possibility that security information may leak from the observable scheduling order of the processes . This is addressed to some extent because SPARK analysis of Ravenscar programs does information flow across tasks . 4.2 Wider analysis One extension suggestion that has come from a software development project within the company is the idea of subprogram integrity level . The motivation is similar to that for the SHOLIS analysis ; that only part of a given program is safety-critical , and that verification activities -LRB- proof , unit testing levels , coverage analysis etc. -RRB- may be better focused on the safety-critical parts . Subprograms are a more useful unit of division for these activities than package state . The algorithm for identifying a subprogram 's actual integrity level is to examine its exports . If it only exports own variables then the subprogram integrity level is the maximum of all exported own variable integrity levels . If some exported variables are formal parameters then each invocation of the subprogram must be examined for own variables that may be actual parameters , and the maximum integrity level taken over all possible exported own variables . Functions are taken to be equivalent to a procedure with a single exported parameter . There are two clear choices for implementing this strategy : 1 . whole-program analysis , determining subprogram integrity level once all invocations are known ; or 2 . partial-program analysis , annotating each critical subprogram declaration with its integrity level and checking at declaration and each invocation that the maximum integrity level is not violated . The first choice is minimal-impact , but does not admit analysis until the whole program is written which is likely to prove troublesome ; the integrity level of many subprograms will not be known until very late in the development process , by which time testing should already be ramped up . The second choice is higher impact ; it requires an extra annotation to be added to the SPARK language and checked by the Examiner , and requires developers to add the annotation to each potentially critical subprogram as it is specified . However , the benefits of the partial program analysis are likely to outweigh these drawbacks . 4.3 Subverting the analysis Declassification is occasionally necessary in security programs ; this is when the assigned security level of information is deliberately reduced . An example would be a security filter which took SECRET information , stripped out sensitive fields and output information which was no more than CONFIDENTIAL level . This can be done in SPARK by hiding the body of a declassifying subprogram from the Examiner , but this is clearly not an optimal solution . A better solution should be found . 4.4 Considerations for certification In very high security applications it may be necessary to certify the object code as well as the source code . It remains to be seen whether and how the information known from the SPARK source code analysis can be carried over to inform an object code analysis . There are other ways by which secure information can be observed , such as covert or timing channels . A full implementation of multi-level security information analysis should be followed by an analysis of how much information could be leaked this way . In this paper we have described recent work on applying information flow analysis techniques to enforcing multi-level security in a single software application . We have shown how the requirements listed by Sabelfeld and Myers -LSB- 11 -RSB- are partially satisfied by the information flow analysis possible with SPARK Ada and the SPARK Examiner . We have further shown that the existing SPARK language and analysis may be extended to enforce the Bell-LaPadula security model with relatively little change . SPARK Ada has already proven itself in high-security and safety-critical application development . It now appears to be an effective choice of language to partition data of differing criticality , and provide a low-cost but robust argument of safety or security for an application . SPARK already provides the semantics-based security required by Sabelfeld and Myers ; the extensions to own variable annotations now provide the complementary security type system . For end-to-end confidentiality in a secure system , we believe that SPARK Ada 's extended information flow analysis provides a hard-to-refute justification of data security . 5.1 Acknowledgements The authors are grateful to Peter Amey , Neil White and Will Ward from Praxis Critical Systems for their feedback on this paper and the prototype integrity checking facilities of the Examiner . 44 -LSB- 1 -RSB- R. J. Anderson . Security engineering : a guide to building dependable distributed systems . Wiley Computer Publishing , 2001 . ISBN 0-471-38922-6 . -LSB- 2 -RSB- J. Barnes . High Integrity Software : The SPARK Approach to Safety And Security . Addison Wesley , April 2003 . -LSB- 3 -RSB- D. E. Bell and L. LaPadula . Secure computer systems . Technical Report ESR-TR-73-278 , Mitre Corporation , November 1973 . v. I and II . -LSB- 4 -RSB- J.-F . Bergeretti and B. A. Carr e. Information-flow and data-flow analysis of while-programs . ACM Transactions on Programming Languages and Systems , 7 -LRB- 1 -RRB- : 37 61 , January 1985 . -LSB- 5 -RSB- Common Criteria . Common Criteria for Information Technology Security Evaluation , August 1999 . -LSB- 6 -RSB- D. Dolev and A. Yao . On the security of public-key protocols . IEEE Transactions on Information Theory , 2 -LRB- 29 -RRB- : 198 208 , August 1983 . -LSB- 7 -RSB- A. Hall and R. Chapman . Correctness by construction : Developing a commercial secure system . IEEE Software , pages 18 25 , Jan/Feb 2002 . -LSB- 8 -RSB- International Electrotechnical Commission . IEC Standard 61508 , Functional Safety of Electrical / Electronic / Programmable Electronic Safety-Related Systems , March 2000 . -LSB- 9 -RSB- S. King , J. Hammond , R. Chapman , and A. Pryor . Is proof more cost effective than testing ? IEEE Transactions on Software Engineering , 26 -LRB- 8 -RRB- : 675 686 , August 2000 . -LSB- 10 -RSB- RTCA / EUROCAE . RTCA DO-178B / EUROCAE ED-12 : Software Considerations in Airborne Systems and Equipment Certification , December 1992 . -LSB- 11 -RSB- A. Sabelfeld and A. C. Myers . Language-based information-flow security . IEEE Journal on Selected Areas in Communications , 21 -LRB- 1 -RRB- , January 2003 . 45

O B O B O O O B I O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B I O O O O B O O O O O O B O B O O O O O O O O B O O O O O O O O B O B O O O O O O O O O O O B O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O B I O B O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O B I O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O B B O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O B O O O O O O O O O O O B O O O O O O B I O O O O O O O B I O O O O O B O O O O O B O O O O O O B I O O B O O O B I O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O B O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O B O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O B O O B O O O O O O O O O O O O O O O O O B O O O O O O B O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O B B O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O B O O O O O O O O O B O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O

information flow
integrity
subprogram
safety
security
spark ada
spark


Efficient Phrase Querying with an Auxiliary Index Search engines need to evaluate queries extremely fast , a challenging task given the vast quantities of data being indexed . A significant proportion of the queries posed to search engines involve phrases . In this paper we consider how phrase queries can be efficiently supported with low disk overheads . Previous research has shown that phrase queries can be rapidly evaluated using nextword indexes , but these indexes are twice as large as conventional inverted files . We propose a combination of nextword indexes with inverted files as a solution to this problem . Our experiments show that combined use of an auxiliary nextword index and a conventional inverted file allow evaluation of phrase queries in half the time required to evaluate such queries with an inverted file alone , and the space overhead is only 10 % of the size of the inverted file . Further time savings are available with only slight increases in disk requirements . H. 3.4 -LSB- Information Storage and Retrieval -RSB- : Systems and Software evaluation -- -LRB- efficiency -RRB- ; E. 2 -LSB- Data -RSB- : Storage Representations Algorithms , performance Search engines are used to find data in response to ad hoc queries . On the Web , most queries consist of simple lists of words . However , a significant fraction of the queries include phrases , where the user has indicated that some of the Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page . To copy otherwise , to republish , to post on servers or to redistribute to lists , requires prior specific permission and/or a fee . SIGIR ' 02 , August 11-15 , 2002 , Tampere , Finland . Copyright 2002 ACM 1-58113-561-0 / 02/0008 ... $ 5.00 . query terms must be adjacent , typically by enclosing them in quotation marks . Phrases have the advantage of being unambiguous concept markers and are therefore viewed as a valuable retrieval technique -LSB- 5 , 6 , 7 , 10 -RSB- In this paper , we explore new techniques for efficient evaluation of phrase queries . A standard way to evaluate phrase queries is to use an inverted index , in which for each index term there is a list of postings , and each posting includes a document identifier , an in-document frequency , and a list of offsets . These offsets are the ordinal word positions at which the term occurs in the document . Given such a word-level inverted index and a phrase query , it is straightforward to combine the postings lists for the query terms to identify matching documents . This does not mean , however , that the process is fast . Even with an efficient representation of postings -LSB- 16 -RSB- , the list for a common term can require several megabytes for each gigabyte of indexed text . Worse , heuristics such as frequency-ordering -LSB- 13 -RSB- or impact-ordering -LSB- 1 -RSB- are not of value , as the frequency of a word in a document does not determine its frequency of participation in a particular phrase . A crude solution is to use stopping , as is done by some widely-used web search engines -LRB- the Google search engine , for example , neglects common words in queries -RRB- , but this approach means that a small number of queries can not be evaluated , while many more evaluate incorrectly -LSB- 12 -RSB- . Another solution is to index phrases directly , but the set of word pairs in a text collection is large and an index on such phrases difficult to manage . In recent work , we proposed nextword indexes as a way of supporting phrase queries and phrase browsing -LSB- 2 , 3 , 15 -RSB- . In a nextword index , for each index term or firstword there is a list of the words or nextwords that follow that term , together with the documents and word positions at which the firstword and nextword occur as a pair . The disadvantage of a nextword index is its size , typically around half that of the indexed collection . Also , as described originally , nextword index processing is not particularly efficient , as the nextwords must be processed linearly and -LRB- compared to an standard inverted index -RRB- for rare firstwords the overhead of the additional layer of structure may outweigh the benefits . In this paper we propose that phrase queries be evaluated through a combination of an inverted index on rare words and a form of nextword index on common words . We explore the properties of phrase queries and show experimentally that query evaluation time can be halved if just the three most common firstwords are supported through a nextword index . While phrase browsing is not possible 215 with such an arrangement , the disk overheads of the partial nextword index are small and the benefits are substantial . We have observed that many ordinary queries -- those without quotation marks -- nonetheless resolve successfully if processed as a phrase query , a phenomenon that search engine users are familiar with , as the most popular engines highly rank matches in which the query terms are adjacent . This suggests that phrase querying is a potential method for a fast `` first cut '' evaluation method , as it allows more rapid identification of documents in which the terms occur as a phrase . With large web search engines being used daily by millions of users , it has become straightforward to gather large numbers of queries and see how users are choosing to express their information needs . Some search engine companies have made extracts of their query logs freely available . In our research , we have made extensive use of query logs provided by Excite dating to 1997 and 1999 , as well as more recent logs from other sources . These logs have similar properties -LRB- with regard to our purposes -RRB- , and we report primarily on the Excite logs in this work . In the Excite log , after sanitizing to remove obscenity there are 1,583,922 queries -LRB- including duplicates -RRB- . Of these , 132,276 or 8.3 % are explicit phrase queries , that is , they include a sequence of two or more words enclosed in quotes . Amongst the rest of the queries -- those without a phrase -- about 5 % contain a word that does not occur at all in the 21.9 gigabytes -LRB- Gb -RRB- of data we use . However , almost exactly 41 % of the remaining non-phrase queries actually match a phrase in the 21.9 Gb dataset we use in our experiments . A surprising proportion of the phrases include a common term . Amongst the explicit phrase queries , 11,103 or 8.4 % include one of the three words that are commonest in our dataset , `` the '' , `` to '' , and `` of '' . 14.4 % of the phrase queries include one of the 20 commonest terms . In some of these queries the common word has a structural function only , as in tower of london , and can arguably be safely neglected during query evaluation . In other queries , however , common words play an important role , as in the movie title end of days or the band name the who , and evaluation of these queries is difficult with the common words removed , especially when both `` the '' and `` who '' happen to be common terms -LSB- 12 -RSB- . Taken together , these observations suggest that stopping of common words will have an unpredictable effect . Stopping may yield efficiency gains , but means that a significant number of queries can not be correctly evaluated . We experimented with a set of 122,438 phrase queries that between them match 309 10 6 documents . Stopping of common words means that a query such as tower of london must be evaluated as tower -- london : the query evaluation engine knows that the two remaining query terms must appear with a single term between them . If the commonest three words are stopped , there are 390 10 6 total matches for all queries extracted from the log . However , these are distributed extremely unevenly amongst the queries : for some queries the great majority of matches are incorrect . The figure rises to 490 10 6 for the commonest 20 words , and 1693 10 6 for the commonest 254 words , while a significant number of queries , containing only stopped words , can not be evaluated at all . It can be argued that stopwords are often insignificant , and that even a document that is technically a mismatch-due to the wrong stopword being present -- may be just as likely to be relevant as a document where the match is correct . However , it is worth emphasising that there are many queries in which stopwords do play an important role . The words `` to '' and `` from '' are often stopped , for example , but mismatches to the query flights to london are likely to be incorrect . Another instance is that the word `` the '' often forms part of a description , thus the moon should not match websites about a moon of Jupiter , Keith Moon , or a book publisher . Amongst the phrase queries , the median number of words in a phrase is 2 , and the average is almost 2.5 . About 34 % of the queries have three words or more , and 1.3 % have six words or more . A few queries are much longer , such as titles : the architect of desire beauty and danger in the stanford white family by suzannah lessard . Another point of interest is where in a phrase the common words occur . In English , the common words rarely terminate a phrase query . Only 0.4 % of phrase queries with `` the '' , `` to '' , or `` of '' have these words at the end . Almost all of these queries are short : virtually no queries of four words or more terminate with one of the commonest terms . In the short queries ending in a common term , the other query terms are themselves usually common . We take advantage of these trends in the methods for phrase query evaluation proposed in this paper . Inverted indexes are the standard method for supporting queries on large text databases ; there are no practical alternatives to inverted indexes that provide sufficiently fast ranked query evaluation . An inverted index is a two-level structure . The upper level is all the index terms for the collection . For text databases , the index terms are usually the words occurring in the text , and all words are included . The lower level is a set of postings lists , one per index term . Following the notation of Zobel and Moffat -LSB- 17 -RSB- , each posting is a triple of the form : d , f d , t , -LSB- o 1 , ... , o f d , t -RSB- where d is the identifier of a document containing term t , the frequency of t in d is f d , t , and the o values are the positions in d at which t is observed . An example inverted file is shown in Figure 1 . In this example , there is a vocabulary of five words , each of which has a postings list . It is straightforward to use an inverted index to evaluate a phrase query . Consider the query magdalene sue prentiss . Of these terms , `` magdalene '' is the rarest , and its inverted list is fetched first . The postings are decoded and a temporary structure is created , recording which documents contain this word and the ordinal word positions in each document at which it occurs . The term `` prentiss '' is the next rarest , and is processed next . For each document identifier and word offset in the temporary structure created earlier , a posting is sought to see whether `` prentiss '' is in the document two words later . If the search fails , that word position is discarded from the temporary structure , as is the document identifier if no word positions for that document remain . As both the structure and the postings are sorted , this process is a linear merge . Then the postings list 216 On Disk Vectors 1 , -LRB- < 9,2 , -LSB- 4,1001 -RSB- > -RRB- 53 , -LRB- < 9,3 , -LSB- 3,8,90 -RSB- ... -RRB- 23 , -LRB- < 4,2 , -LSB- 5,34 -RSB- > , ... -RRB- 243 , -LRB- < 5,1 , -LSB- 45 -RSB- > , < 9,1 , -LSB- 7 -RSB- > ... -RRB- In Memory Vocabulary new in historic hampshire railroads 15 , -LRB- < 1,1 , -LSB- 100 -RSB- > , < 9,1 , -LSB- 6 -RSB- > ... -RRB- Figure 1 : An inverted file for a collection with a vocabulary of five words . for `` sue '' is fetched and decoded , and used to further delete entries from the temporary structure . The remaining entries are documents and word positions at which the phrase occurs . Summarizing , phrase queries are evaluated as follows . 1 . Sort the query terms from rarest to commonest , keeping note of their original position in the phrase . 2 . Fetch the postings list for the first -LRB- rarest -RRB- query term . Decode this list into a temporary structure of document identifiers and word offset positions . 3 . For each remaining query term , decode its postings list , merging it with the temporary data ; this merge process discards from the temporary structure all document identifiers and word offsets that do not match any entry in the postings list . In this query evaluation model , processing of the first query term establishes a superset of the possible locations of the complete phrase , which are maintained in a temporary structure ; as the subsequent query terms are evaluated , this structure is pruned , never added to . It is thus essential to begin processing with the rarest query term , to avoid creation of an excessively large temporary structure -LRB- or of having to process the inverted lists in stages to stay within a memory limit -RRB- . A simple heuristic to address this problem is to directly merge the inverted lists rather than decode them in turn . On the one hand , merging has the disadvantage that techniques such as skipping -LSB- 11 -RSB- can not be as easily used to reduce processing costs -LRB- although as we discuss later skipping does not necessarily yield significant benefits -RRB- . On the other hand , merging of at least some of the inverted lists is probably the only viable option when all the query terms are moderately common . Whether the lists are merged or processed in turn , the whole of each list needs to be fetched -LRB- unless query processing terminates early due to lack of matches -RRB- . For ranked query processing it is possible to predict which postings in each inverted list are most likely to be of value , and move these to the front of the inverted list ; techniques for such list modification include frequency-ordering -LSB- 13 -RSB- and impact-ordering -LSB- 1 -RSB- . With these techniques , only the first of the inverted lists need be fetched during evaluation of most queries , greatly reducing costs . In contrast , for phrase querying it is not simple to predict which occurrences of the term will be in a query phrase , and thus such reordering is unlikely to be effective . Offsets only Table 1 : Size of inverted index -LRB- Mb -RRB- after stopping of common words . Number of Index size words stopped -LRB- Mb -RRB- 0 2350 3 2259 6 2195 10 2135 20 2089 254 1708 have to be decoded when there is a document match , but they still have to be retrieved . Other techniques do have the potential to reduce query evaluation time , in particular skipping -LSB- 11 -RSB- , in which additional information is placed in inverted lists to reduce the decoding required in regions in the list that can not contain postings that will match documents that have been identified as potential matches . On older machines , on which CPU cycles were relatively scarce , skipping could yield substantial gains . On current machines , however , disk access costs are the more important factor , and in other experiments we have observed that the increase in length of lists required by skipping outweighs the reduction in decoding time . We therefore do not use skipping in our experiments . We have implemented a phrase query evaluator based on inverted lists , using compression techniques similar to those employed in MG -LSB- 16 -RSB- to reduce costs , and have used it to test the efficiency of phrase query evaluation . Our test data is 21.9 Gb of HTML containing about 8.3 Gb of text -LRB- drawn from the TREC large web track -LSB- 9 -RSB- -RRB- . Table 1 shows the size of the index with a range of levels of stopping . As can be seen , the three commonest words account for around 4 % of the index size , and only small space savings are yielded by stopping . However , as Table 2 shows , the impact of stopping on query evaluation time is dramatic . Just removing the three commonest words reduces average time by about 60 % , and by a factor of 3 for longer queries . For these longer queries , the savings continue to increase as more common words are stopped . It is the scale of these savings that make stopping attractive , despite the fact that they are at the cost of inaccurate query results . 217 Table 2 : Times for phrase query evaluation -LRB- seconds -RRB- on an inverted index after stopping of common words . Results are shown for all queries ; 2-word queries only ; and 5-word queries only . Number of Overall 2-word 5-word words stopped time -LRB- sec -RRB- queries queries 0 1.56 0.49 6.41 3 0.66 0.30 1.94 6 0.45 0.29 1.07 10 0.40 0.28 0.81 20 0.37 0.28 0.70 254 0.18 0.16 0.26 Inverted indexes allow evaluation of phrase queries , but faster evaluation is possible with phrase-oriented indexes . One possibility is to use a conventional inverted index in which the terms are word pairs . Another way to support phrase based query modes is to index and store phrases directly -LSB- 8 -RSB- or simply by using an inverted index and approximating phrases through a ranked query technique -LSB- 5 , 10 -RSB- . Greater efficiency , with no additional in-memory space overheads , is possible with a special-purpose structure , the nextword index -LSB- 15 -RSB- , where search structures are used to accelerate processing of word pairs . The nextword index takes the middle ground by indexing pairs of words and , therefore , is particularly good at resolving phrase queries containing two or more words . As noted above and observed elsewhere , the commonest number of words in a phrase is two -LSB- 14 -RSB- . A nextword index is a three-level structure . The highest level is of the distinct index terms in the collection , which we call firstwords . At the middle level , for each firstword there is a data structure -LRB- such as a front-coded list , or for fast access a structure such as a tree -RRB- of nextwords , which are the words observed to follow that firstword in the indexed text . For example , for the firstword `` artificial '' , nextwords include `` intelligence '' , `` insemination '' , and `` hip '' . At the lowest level , for each nextword there is a postings list of the positions at which that firstword-nextword pair occur . An example nextword index is shown in Figure 2 . In this example , there are two firstwords , `` in '' and `` new '' . Some of the nextwords for `` in '' are `` all '' , `` new '' , and `` the '' . For each firstword-nextword pair , there is a postings list . -LRB- A nextword index is of course a form of inverted index , but for consistency with other work we use `` inverted index '' solely to refer to a standard word-level inverted file . -RRB- In nextword indexes , the postings lists are typically short , because most pairs only occur infrequently . For example , the postings list for the firstword-nextword pair `` the '' `` who '' is orders of magnitude smaller than the postings lists for these words in an inverted file . It follows that phrase query evaluation can be extremely fast . Nextword indexes also have the benefit of allowing phrase browsing or phrase querying -LSB- 4 , 15 -RSB- ; given a sequence of words , the index can be used to identify which words follow the sequence , thus providing an alternative mechanism for searching text collections . However , we do not consider phrase browsing further in this paper . For phrase queries of more than two words , multiple postings lists must be fetched from the nextword index to resolve the query . Selection of which listings to fetch requires a little care . For example , with the query boulder municipal employees credit union the query can be resolved by fetching the postings lists for the firstword-nextword pairs `` boulder '' `` municipal '' , `` employees '' `` credit '' , and `` credit '' `` union '' . Alternatively , it would be possible to get the lists for `` boulder '' `` municipal '' , `` municipal '' `` employees '' , and `` credit '' `` union '' . Which is most efficient depends on which is shorter : the list for `` employees '' `` credit '' or the list for for `` municipal '' `` employees '' . Unfortunately , establishing which is shorter requires two disk accesses , to retrieve the nextwords for `` employees '' and `` municipal '' . However , we have observed that the frequency of a firstword closely correlates to the lengths of its nextword lists . Thus in the query historic railroads in new hampshire we can with confidence choose `` railroads '' `` in '' in preference to `` in '' `` new '' , because `` railroads '' is much less common than `` in '' . We have considered algorithms for choosing order of evaluation elsewhere -LSB- 3 -RSB- . An efficient algorithm for evaluating phrase queries with a nextword index is as follows . 1 . If the number of query terms n is even , the query can consist of n/2 disjoint firstword-nextword pairs . If the number of query terms n is odd , n/2 firstword-nextword pairs must be chosen . However , in both cases it is more efficient to choose more than the minimum number of pairs , if doing so avoids choice of a common word as a firstword . 2 . The method we use is to choose all n - 1 firstword-nextword pairs ; then sort them by increasing firstword frequency ; then discard from the list the pairs that are completely covered by preceding selections . This approach can lead to processing of more than n/2 pairs , but experimentally was shown to reduces costs overall . 3 . The selected word pairs are sorted by increasing frequency of the firstwords , then their postings lists are processed as for inverted file phrase query processing . The nextword index for our Web collection is 4487 Mb in size , almost exactly twice that of an inverted file . For phrase queries , the savings in query evaluation time are dramatic . Average query evaluation time is reduced to 0.06 seconds , faster than inverted files by a factor of 25 . For two-word queries , the time falls to 0.01 seconds , which is faster by a factor of 50 . The time for 5-word queries is 0.32 . An interesting possibility suggested by these results is that -- given space for a nextword index -- all queries be evaluated as if they were phrases . We observed above that a significant fraction of all queries successfully evaluate , and indeed on browsing the query logs it is obvious that many of the queries without quotation marks are nonetheless intended to be phrases . Spink et al. -LSB- 14 -RSB- suggest that most two-word queries should be treated as a phrase query even if they were entered as a ranked query . Given that search engines return as highest matches the pages in which the 218 In Memory Vocabulary On Disk Nextword Lists in new ... age hampshire house ... the all new On Disk Inverted Vectors 15 , -LRB- < 1,15 , -LSB- 100 -RSB- > , < 65,1 , -LSB- 1 -RSB- > , < 74,7 , -LSB- 23,43,54,62,68,114,181,203 -RSB- > , ... -RRB- 1 , -LRB- < 9,1 , -LSB- 6 -RSB- > -RRB- 3 , -LRB- < 1,1 , -LSB- 12 -RSB- > , < 34,3 , -LSB- 23,34,111 -RSB- > , < 77,1 , -LSB- 29 -RSB- > -RRB- 305 , -LRB- < 9,2 , -LSB- 7,199 -RSB- > , < 532,1 , -LSB- 256 -RSB- > , ... -RRB- 2 , -LRB- < 9,1 , -LSB- 423 -RSB- > , < 19,1 , -LSB- 4 -RSB- > -RRB- 2 , -LRB- < 31,3 , -LSB- 21,41,91 -RSB- > , < 44,1 , -LSB- 34 -RRB- -RSB- > -RRB- Figure 2 : A nextword index with two firstwords . query words appear in sequence , use of a nextword index provides a rapid mechanism for finding these pages . Much of the speed improvement for phrase queries yielded by nextword indexes is for queries involving a non-rare word . Indeed , for queries of rare words there may be little gain , as query processing with nextword indexes involves more complex structures than does processing with inverted indexes . As the two approaches to phrase query processing appear , then , to have complementary advantages , it is attractive to try to combine their strengths . We have observed above that inverted indexes are the least efficient for phrases involving common words , the case where nextword indexes yield the greatest speed advantage . We therefore propose that common words only be used as firstwords in a stripped-down nextword index , and that this new index be used where possible in evaluation of phrase queries . We call this a top frequency based scheme , since only the most frequent words are indexed in the nextword index . We have explored other schemes based on the frequency of words in the indexed collection , or based on the frequency of words in the query log . None of the investi-gated schemes offered a better space and time trade-off , so we report only results from the top frequency scheme . An example of a top frequency combined index is shown in Figure 3 . At the left there is a vocabulary of five words . Each word has an inverted list , together constituting a complete inverted file for these words . In addition , the common words `` in '' and `` new '' have a nextword index . With a combined index , processing involves postings lists from both the inverted index and the nextword index . Consider again the query : historic railroads in new hampshire Neither `` historic '' nor `` railroads '' is a common word , so establishing that these terms occur in the phrase involves fetching their postings lists from the inverted index and processing in the usual way . However , `` in '' and `` new '' are both common . The posting list for the firstword-nextword pair `` in '' `` new '' from the nextword index must be fetched and processed . Then there is a choice . On the one hand , the nextword index postings list for `` new '' `` hampshire '' can not be longer than the inverted index postings list for `` hampshire '' and in all likelihood is a great deal shorter . On the other hand , compared to the inverted index , an extra disk access is required to fetch a postings list from the nextword index . In our implementation , we process using the nextword index if possible , and resort to the inverted index only for terms that are not in an indexed firstword-nextword pair . In summary , we use the following process : 1 . Identify all pairs in the list in which the first term is an indexed firstword . Sort these terms , and prune the list as for standard evaluation of phrase queries via a nextword index . 2 . For all terms not in a firstword-nextword pair , sort . 3 . Process the postings lists in increasing order of firstword frequency , so that processing of nextword index lists and of inverted file lists is interleaved . In this model , a common word need only be evaluated via its postings list in the inverted file if it occurs as the last word in a query , which in the Excite query log is a rare event . We have tested other query resolution methods that involved term sorting based on nextword frequency -LRB- or NWF , the number of nextwords for a firstword -RRB- , inverted document frequency -LRB- or IDF , the number of documents in which a word occurs -RRB- , or both . We also experimented with resolving nextword entries of a given query always first , or always last . We found overall that these different resolution methods did not significantly vary in query speed and behaved almost identically to sorting by IDF only . We therefore sort inverted index terms and nextword terms based on IDF since we do not need to keepanother statistical value per index term and sorting is straightforward . All experiments were run on an Intel 700 MHz Pentium III-based server with 2 Gb of memory , running the Linux operating system under light load . In Table 3 we show sizes of nextword indexes in which only the commonest terms are allowed as firstwords . The table shows that a nextword index that contains only the three commonest terms consumes 254 Mb , that is , just over 10 % of the space of the inverted index or around 1 % of the size of the original HTML collection . Query evaluation time with a combined index is shown in Table 4 . -LRB- The `` 0 '' line is repeated from Table 2 . -RRB- As can be seen , use of a nextword index allows evaluation of all phrase queries , and much more rapidly than was previously possible . Use of a partial nextword index of 1 % of the HTML collection halves query evaluation time ; a partial nextword 219 in hampshire historic new railroads In Memory Vocabulary On Disk Nextword Lists ... the all new 15 , -LRB- < 15,1 , -LSB- 100 -RSB- > , < 65,1 , -LSB- 1 -RSB- > , < 74,7 , -LSB- 23,43,54,62,68,114,181 -RSB- > ... -RRB- 251 , -LRB- < 5,1 , -LSB- 45 -RSB- > , < 9,1 , -LSB- 6 -RSB- > ... -RRB- 1 , -LRB- < 9,1 , -LSB- 7 -RSB- > -RRB- 23 , -LRB- < 9,3 , -LSB- 4,8,245 -RSB- > ... -RRB- 2 , -LRB- < 1,1 , -LSB- 53 -RSB- > , < 9,2 , -LSB- 4,1001 > -RSB- -RRB- 23 , -LRB- < 1,2 , -LSB- 65,98 -RSB- > , < 9,4 , -LSB- 7,54,64,69 -RSB- > ... -RRB- age hampshire house ... 15 , -LRB- < 2,1 , -LSB- 100 -RSB- > , < 6,1 , -LSB- 1 -RSB- > , < 9,8 , -LSB- 1,5,54,62,68,114,181,203 -RSB- > ... -RRB- On Disk Inverted Vectors 3 , -LRB- < 1,1 , -LSB- 12 -RSB- > , < 34,3 , -LSB- 23,34,111 -RSB- > , < 77,1 , -LSB- 29 -RSB- > -RRB- 2 , -LRB- < 31,3 , -LSB- 21,41,91 -RSB- > , < 44,1 , -LSB- 34 -RSB- > -RRB- 305 , -LRB- < 9,2 , -LSB- 7,54 -RSB- > , < 532,1 , -LSB- 256 -RSB- > ... -RRB- 2 , -LRB- < 9,1 , -LSB- 423 -RSB- > , < 19,1 , -LSB- 4 -RSB- > -RRB- Figure 3 : A combined inverted file and nextword index . Table 3 : Size of nextword index -LRB- Mb -RRB- containing only common firstwords . Number of Index size common words -LRB- Mb -RRB- 3 254 6 427 10 520 20 657 254 1366 index of less than 3 % of the size of the collection cuts time to a third . These are substantial savings at low cost . Phrase query processing time with a nextword index is only slightly greater than with a stopped inverted file , and no answers are lost . Such combined processing can be integrated with other heuristics for phrase query evaluation . For example , a strategy that is likely to be successful in the context of a web search engine is to maintain indexes -LRB- perhaps for a limited time only -RRB- on phrases , or word pairs from phrases , that are commonly posed as queries . Amongst our 132,276 queries , 72,184 are distinct . The commonest phrase query -LRB- thumbnail post -RRB- occurs 205 times and involves no common words . The queries themselves contain 92,846 distinct word pairs ; the commonest pair occurs 683 times . Indexing of common query pairs has the potential to yield significant further savings . We have proposed that phrase queries on large text collections be supported by use of a small auxiliary index . In this approach , all words in the text are indexed via an inverted file ; in addition , the commonest words are indexed via an auxiliary nextword index , which stores postings lists for firstword-nextword pairs . We have shown that the cost of evaluating phrase indexes can be cut by a factor of three , with an auxiliary index that is only 3 % of the size of the Table 4 : Times for phrase query evaluation -LRB- seconds -RRB- on a combined index , with different numbers of common words used in the nextword index . Results are shown for all queries ; 2-word queries only ; and 5-word queries only . Number of Overall 2-word 5-word common words time -LRB- sec -RRB- queries queries 0 1.56 0.49 6.41 3 0.76 0.31 2.99 6 0.57 0.31 2.28 10 0.53 0.30 2.10 20 0.50 0.30 1.98 254 0.46 0.27 1.83 indexed data . These results show that there is no need to use stopping in phrases . Indeed , the statistics on the number of matches indicate that such stopping leads to significant error rates . While it can be argued that mistakes in matching due to stopping of common words are in many cases unimportant , there are numerous queries where the stopwords are significant ; moreover , we have demonstrated that there is no reason to make such mistakes at all . Our schemes have scope for improvement . In particular , choosing of pairs during query evaluation requires further exploration , and we are further investigating structures for representing nextword lists . However , our results show that evaluation of phrase queries can be dramatically accelerated with only a small additional index , and that stopping of phrases leads to errors and is not necessary for efficiency . 220 This research was supported by the Australian Research Council . We thank Amanda Spink , Doug Cutting , Jack Xu , and Excite Inc. for providing the query log . -LSB- 1 -RSB- V. N. Anh , O. Kretser , and A. Moffat . Vector-Space ranking with effective early termination . In W. B. Croft , D. J. Harp er , D. H. Kraft , and J. Zobel , editors , Proc . ACM-SIGIRInternational Conference on Research and Development in Information Retrieval , pages 35 42 , New York , Sept. 2001 . -LSB- 2 -RSB- D. Bahle , H.E. Williams , and J. Zobel . Compaction techniques for nextword indexes . In Proc . 8th International Symposium on String Processing and Information Retrieval -LRB- SPIRE2001 -RRB- , pages 33 45 , San Rafael , Chile , 2001 . -LSB- 3 -RSB- D. Bahle , H. E. Williams , and J. Zobel . Optimised phrase querying and browsing in text databases . In M. Oudshoorn , editor , Proc . Australasian Computer Science Conference , pages 11 19 , Gold Coast , Australia , Jan. 2001 . -LSB- 4 -RSB- P. Bruza , R. McArthur , and S. Dennis . Interactive internet search : keyword , directory and query reformulation mechanisms compared . In N. J. Belkin , P. Ingwersen , and M.-K . Leong , editors , Proc . ACM-SIGIRInternational Conference on Research and Development in Information Retrieval , pages 280 287 , Athens , 2000 . -LSB- 5 -RSB- C. L. Clarke , G. V. Cormack , and E. A. Tudhope . Relevance ranking for one - to three-term queries . In Proc . of RIAO-97 , 5th International Conference `` Recherche d'Information Assistee par Ordinateur '' , pages 388 400 , Montreal , CA , 1997 . -LSB- 6 -RSB- W. B. Croft , H. R. Turtle , and D. D. Lewis . The use of phrases and structured queries in information retrieval . In A. Bookstein , Y. Chiaramella , G. Salton , and V. V. Raghavan , editors , Proc . ACM-SIGIR International Conference on Research and Development in Information Retrieval , pages 32 45 , Chicago , 1991 . -LSB- 7 -RSB- E. F. de Lima and J. O. Pedersen . Phrase recognition and expansion for short , precision-biased queries based on a query log . In Proc . ACM-SIGIRInternational Conference on Research and Development in Information Retrieval , pages 145 152 , Berkeley , 1999 . -LSB- 8 -RSB- C. Gutwin , G. Paynter , I. Witten , C. NevillManning , and E. Frank . Improving browsing in digital libraries with keyphrase indexes . Decision Support Systems , 27 -LRB- 1/2 -RRB- : 81 104 , 1998 . -LSB- 9 -RSB- D. Hawking , N. Craswell , P. Thistlewaite , and D. Harman . Results and challenges in web search evaluation . In Proc . of the Eighth International World-Wide Web Conference , volume 31 , pages 1321 1330 , May 1999 . -LSB- 10 -RSB- D. D. Lewis and W. B. Croft . Term clustering of syntactic phrases . In J.-L . Vidick , editor , Proc . ACM-SIGIRInternational Conference on Research and Development in Information Retrieval , pages 385 404 , 1990 . -LSB- 11 -RSB- A. Moffat and J. Zobel . Self-indexing inverted files for fast text retrieval . ACM Transactions on Information Systems , 14 -LRB- 4 -RRB- : 349 379 , 1996 . -LSB- 12 -RSB- G. W. Paynter , I. H. Witten , S. J. Cunningham , and G. Buchanan . Scalable browsing for large collections : A case study . In Proc . of the 5th ACM International Conference on Digital Libraries , pages 215 223 , San Antonio , 2000 . -LSB- 13 -RSB- M. Persin , J. Zobel , and R. Sacks-Davis . Filtered document retrieval with frequency-sorted indexes . Journal of the American Society for Information Science , 47 -LRB- 10 -RRB- : 749 764 , 1996 . -LSB- 14 -RSB- A. Spink , D. Wolfram , B. J. Jansen , and T. Saracevic . Searching the web : The public and their queries . Journal of the American Society for Information Science , 52 -LRB- 3 -RRB- : 226 234 , 2001 . -LSB- 15 -RSB- H. Williams , J. Zobel , and P. Anderson . What 's next ? index structures for efficient phrase querying . In M. Orlowska , editor , Proc . Australasian Database Conference , pages 141 152 , Auckland , New Zealand , 1999 . -LSB- 16 -RSB- I. H. Witten , A. Moffat , and T. C. Bell . Managing Gigabytes : Compressing and Indexing Documents and Images . Morgan Kaufmann , San Francisco , California , second edition , 1999 . -LSB- 17 -RSB- J. Zobel and A. Moffat . Exploring the similarity space . SIGIRForum , 32 -LRB- 1 -RRB- : 18 34 , 1998 . 221

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

query evaluation
inverted index
nextword index
stopping
common words
phrase query
indexing

phrase query evaluation
index representation
evaluation efficiency