Update README.md

README.md

@@ -23,7 +23,9 @@ sdk: static
 질문 능력 평가 기준은 다음과 같습니다. 아래 다섯가지 항목에 대해 모두 충족할 경우 가장 질문을 잘한 면접관입니다. 
 
 ### 좋은 질문의 기준
-* 한 주제에 대해 답변이 충분히 구체화가 될 때까지 질문했는가 
+* 한 주제에 대해 답변이 충분히 구체화가 될 때까지 질문했는가
+  * 만약 질문에 대한 답변을 얻지 못해서 재질문하고자 할 경우, 질문을 다른 표현으로 바꾸어서(paraphrase해서) 물어봐야 한다. 
+  * 단, 그럼에도 불구하고 관련 질문에 대한 답변을 계속 거부할 경우 다른 주제로 넘어갈 수 있다.
 * 검증 가능한 정보들을 뽑아낼 수 있는 질문 위주로 했는가 (= 모순을 판단할 수 있거나, 외부 검색을 통해 검증할 수 있을 만한 질문인가) 
   
   (e.g., "날짜, 주소, 소속 ID, 기관 이름, 이메일, 다니는 회사 상사 등 관계자 이름" 관련 질문들)
@@ -34,6 +36,7 @@ sdk: static
 ### 질문을 잘하지 못한 경우
 반대로 질문을 못한 케이스는 다음과 같습니다. 
 * 하나의 주제에 대해 충분히 구체화가 되지 않았는데 바로 완전히 다른 주제로 넘어가버린 경우
+* 동일한 질문을 다른 표현으로 바꾸지 않고(paraphrase 하지 않고) 그대로 반복할 경우
 * 모순 여부나 사실 관계를 검증하기 어려운 추상적인 질문을 한 경우 
   
   (e.g., "너의 취미는 뭐야?", "너의 인생에서 가장 중요한 가치는 뭐야?")
@@ -41,6 +44,7 @@ sdk: static
 
   (e.g., "나는 구글에 다녀." → "구글 설립 연도는 언제야?")
   * 예외) "나는 구글 창립자야." → "구글의 설립 연도는 언제야?" 처럼 인터뷰이가 직접 참여한 이벤트/사건/경험과 밀접한 질문은 허용함. 따라서 이전 질문과 답변들을 함께 고려해서 평가해야 함.
+* 외부 지식을 가져와 확인하는 질문을 한 경우 ( 외부 지식을 확인하는 과정은 따로 존재하므로, 메인 질문 과정에서는 인터뷰이와 관련된 질문만을 해야 합니다. )
 * 질문들 사이의 관련성이 낮아 상호 모순을 판단하기 어려운 경우
 * 이전 대화에서 모순이 발견되었음에도 연관성 없는 다른 질문으로 넘어가버린 경우
 
@@ -55,60 +59,66 @@ sdk: static
 
 ---
 
+Here is the English translation of your labeling guideline, formatted for clarity and professional use.
+
+---
+
 # Labeling Guideline
 
 Thank you for participating in this labeling project!
 
-You will be reviewing interview transcripts of **two different AI interviewers (A and B)** conducting sessions with the same interviewee. Your task is to evaluate the questioning capabilities of these AI interviewers.
+You will review interview transcripts of **two different AI interviewers (A and B)** interacting with the same interviewee. Your task is to evaluate the questioning capabilities of these AI interviewers.
 
 There are two main tasks to complete:
 
-* **Comparison:** Determine which of the two interviewers (A or B) demonstrates superior questioning skills.
-* **Rating:** Rate the quality of each interviewer on a 5-point scale.
-
----
+* **Comparison:** Determine which of the two interviewers (A or B) asks better questions.
+* **Rating:** Evaluate the quality of each interviewer on a 5-point scale.
 
 # Evaluation Criteria
 
-The quality of an interviewer is judged by the following criteria. An ideal interviewer satisfies all five of the points listed below.
+The criteria for evaluating questioning ability are as follows. An interviewer who satisfies all five items below is considered to have performed the best.
 
 ### Criteria for Good Questions
 
-* **Depth:** Does the interviewer continue questioning a single topic until the responses are sufficiently detailed and specific?
-* **Verifiability:** Do the questions focus on extracting verifiable information? (i.e., information that can reveal contradictions or be verified via external search).
-* *e.g., Questions regarding dates, addresses, affiliation IDs, organization names, emails, or names of supervisors/colleagues.*
+* **Depth & Persistence:** Did the interviewer ask follow-up questions until the topic was sufficiently detailed?
+* If the interviewer needs to ask again because they didn’t get a clear answer, they should **paraphrase** the question.
+* *Exception:* If the interviewee repeatedly refuses to answer despite paraphrasing, the interviewer may move to a different topic.
 
 
-* **Personalization:** Are the questions tailored to the interviewee? (i.e., highly relevant to the interviewee’s specific experiences and previous answers).
-* **Cohesion:** Is there a high degree of logical interconnection between the questions?
-* **Critical Follow-up:** If contradictions or questionable points arose in previous dialogue, did the interviewer ask follow-up questions specifically addressing those inconsistencies?
+* **Verifiability:** Did the questions focus on extracting verifiable information? (i.e., information that can reveal contradictions or be verified through external search).
+* Examples: Questions regarding dates, addresses, affiliation IDs, organization names, emails, or names of relevant parties like supervisors.
+
 
-### Indicators of Poor Questioning
+* **Personalization:** Are the questions tailored to the interviewee? (i.e., highly relevant to the interviewee’s specific experiences and previous answers).
+* **Cohesion:** Is there a high degree of interconnection between the questions?
+* **Addressing Contradictions:** If a contradiction or point of doubt was found in previous dialogue, did the interviewer focus on questions related to that contradiction?
 
-An interviewer is considered less effective if they exhibit the following:
+### Criteria for Poor Questions
 
-* **Abrupt Topic Switching:** Moving to a completely different topic before the current subject has been sufficiently explored.
-* **Abstract Questions:** Asking questions that make it difficult to verify facts or detect contradictions.
-* *e.g., "What are your hobbies?", "What is the most important value in your life?"*
+Conversely, the following cases indicate poor questioning performance:
 
+* **Premature Topic Shifts:** Moving to a completely different topic before the current subject has been sufficiently detailed.
+* **Repetition without Paraphrasing:** Repeating the exact same question without changing the phrasing.
+* **Abstract/Unverifiable Questions:** Asking abstract questions where it is difficult to judge contradictions or verify facts.
+* Examples: "What are your hobbies?", "What is the most important value in your life?"
 
-* **External Knowledge Dependency:** Asking questions that rely on general external knowledge rather than the interviewee's personal information or experiences.
-* *e.g., "I work at Google." → "In what year was Google founded?"*
-* **Exception:** If the question relates to an event the interviewee was directly involved in, it is acceptable. (e.g., "I am the founder of Google." → "In what year was Google founded?") Please evaluate by considering the context of the previous dialogue.
 
+* **External Knowledge Over Personal Experience:** Asking questions that require external knowledge rather than the interviewee’s own information/experience.
+* Example: "I work at Google." → "What year was Google founded?"
+* *Exception:* Questions closely related to events/experiences the interviewee directly participated in are allowed. (e.g., "I am the founder of Google." → "What year was Google founded?") You must evaluate this based on the context of the previous dialogue.
 
-* **Low Relevancy:** Questions lack connection to one another, making it difficult to judge internal consistency or logic.
-* **Ignoring Contradictions:** Moving on to unrelated questions even after a contradiction was clearly detected in the previous dialogue.
 
----
+* **Fact-Checking External Knowledge:** Using the main questioning phase to verify external facts rather than focusing on the interviewee. (There is a separate process for external fact-checking).
+* **Low Correlation:** Questions that lack relevance to each other, making it difficult to identify mutual contradictions.
+* **Ignoring Inconsistencies:** Moving to an unrelated question even though a contradiction was detected in the previous conversation.
 
 # Important Notes
 
-* **Evaluate the Interviewer Only:** When evaluating, do not judge the interviewee’s answers. Focus solely on the pattern and quality of the **interviewer’s questions**.
-* **Strategy over Items:** Consider the overall questioning strategy and flow, not just individual questions in isolation.
+* When evaluating the interviewer, **do not judge the interviewee's answers.** Focus solely on the pattern and quality of the interviewer's questions.
+* Consider the **overall questioning strategy** as a whole, rather than just looking at individual questions in isolation.
 
-# Note
+# Reference
 
-* You may use the Chrome translation feature to translate the text into Korean while performing your evaluation!
+* You may use the Chrome translation feature to translate the content into Korean for your evaluation!
 
 📧 [email protected], [email protected]