re-baseline sim residuals: use-each-word-once on Aldo's, 2c-f2 accept variants, direct yes/no rubric on 2a-f3

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
2026-08-22 08:51:16 +08:00
parent 688495b7cb
commit f3a7e85203
2 changed files with 5 additions and 5 deletions

View File

@@ -458,7 +458,7 @@ $PRODUCE The Shop That Serviced My Car
ID: l5-s8 ID: l5-s8
REFERENCE: l5-s8-g REFERENCE: l5-s8-g
INTRO: A whole neighborhood of shops, clerks, and mechanics. Complete each sentence with two words from the box — the connecting word (who or that for people, which or that for things) and a verb in its correct form: "service", "refund", "guarantee", "charge", "exchange", "leak", "credit", "tune up", "give back". INTRO: A whole neighborhood of shops, clerks, and mechanics. Complete each sentence with two words from the box — the connecting word (who or that for people, which or that for things) and a verb in its correct form: "service", "refund", "guarantee", "charge", "exchange", "leak", "credit", "tune up", "give back".
INSTRUCTION: Type the connecting word and the verb in its correct form. INSTRUCTION: Type the connecting word and the verb in its correct form. Use each word from the box only once.
INPUT: type INPUT: type
CHECK: exact CHECK: exact

View File

@@ -221,7 +221,7 @@ INTRO: Listen to each question about Mark and Sam's conversation. Type your answ
INSTRUCTION: Type your answer in a complete sentence. INSTRUCTION: Type your answer in a complete sentence.
INPUT: type INPUT: type
CHECK: llm CHECK: llm
RUBRIC: Accept any sentence that correctly conveys the factual meaning of the model answer based on the dialogue. RUBRIC: Accept any sentence that correctly conveys the factual meaning of the model answer based on the dialogue. For yes/no questions, a direct correct yes/no answer ("No, he isn't.") is acceptable without the model's extra detail.
REPEAT REPEAT
PROMPT: Is Mark happy? {bk12-l2a-f3-01-q.mp3} PROMPT: Is Mark happy? {bk12-l2a-f3-01-q.mp3}
@@ -973,7 +973,7 @@ ACCEPT: No, I'ven't finished it yet.
TEMPLATE: yes, already TEMPLATE: yes, already
PROMPT: Has Janet graduated from college yet? {bk12-l2c-f2-01-q.mp3} PROMPT: Has Janet graduated from college yet? {bk12-l2c-f2-01-q.mp3}
RESPONSE: Yes, she has already graduated. {bk12-l2c-f2-01-a1.mp3} RESPONSE: Yes, she has already graduated. {bk12-l2c-f2-01-a1.mp3}
ACCEPT: Yes, she's already graduated. ACCEPT: Yes, she's already graduated. | Yes, she's already graduated from college. | Yes, she has already graduated from college.
TEMPLATE: no, yet TEMPLATE: no, yet
PROMPT: Has Janet graduated from college yet? {bk12-l2c-f2-01-q.mp3} PROMPT: Has Janet graduated from college yet? {bk12-l2c-f2-01-q.mp3}
@@ -995,7 +995,7 @@ ACCEPT: No, I'ven't seen her yet.
TEMPLATE: yes, already TEMPLATE: yes, already
PROMPT: Has your son learned Spanish yet? {bk12-l2c-f2-03-q.mp3} PROMPT: Has your son learned Spanish yet? {bk12-l2c-f2-03-q.mp3}
RESPONSE: Yes, he's already learned Spanish. {bk12-l2c-f2-03-a1.mp3} RESPONSE: Yes, he's already learned Spanish. {bk12-l2c-f2-03-a1.mp3}
ACCEPT: Yes, he has already learned Spanish. ACCEPT: Yes, he has already learned Spanish. | Yes, he's already learned it. | Yes, he has already learned it.
TEMPLATE: no, yet TEMPLATE: no, yet
PROMPT: Has your son learned Spanish yet? {bk12-l2c-f2-03-q.mp3} PROMPT: Has your son learned Spanish yet? {bk12-l2c-f2-03-q.mp3}
@@ -1006,7 +1006,7 @@ ACCEPT: No, he'sn't learned Spanish yet.
TEMPLATE: yes, already TEMPLATE: yes, already
PROMPT: Have the Johnsons realized that they are rich yet? {bk12-l2c-f2-04-q.mp3} PROMPT: Have the Johnsons realized that they are rich yet? {bk12-l2c-f2-04-q.mp3}
RESPONSE: Yes, they have already realized it. {bk12-l2c-f2-04-a1.mp3} RESPONSE: Yes, they have already realized it. {bk12-l2c-f2-04-a1.mp3}
ACCEPT: Yes, they've already realized it. ACCEPT: Yes, they've already realized it. | Yes, they've already realized that they are rich. | Yes, they have already realized that they are rich.
TEMPLATE: no, yet TEMPLATE: no, yet
PROMPT: Have the Johnsons realized that they are rich yet? {bk12-l2c-f2-04-q.mp3} PROMPT: Have the Johnsons realized that they are rich yet? {bk12-l2c-f2-04-q.mp3}