re-baseline sim residuals: use-each-word-once on Aldo's, 2c-f2 accept variants, direct yes/no rubric on 2a-f3
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -458,7 +458,7 @@ $PRODUCE The Shop That Serviced My Car
|
|||||||
ID: l5-s8
|
ID: l5-s8
|
||||||
REFERENCE: l5-s8-g
|
REFERENCE: l5-s8-g
|
||||||
INTRO: A whole neighborhood of shops, clerks, and mechanics. Complete each sentence with two words from the box — the connecting word (who or that for people, which or that for things) and a verb in its correct form: "service", "refund", "guarantee", "charge", "exchange", "leak", "credit", "tune up", "give back".
|
INTRO: A whole neighborhood of shops, clerks, and mechanics. Complete each sentence with two words from the box — the connecting word (who or that for people, which or that for things) and a verb in its correct form: "service", "refund", "guarantee", "charge", "exchange", "leak", "credit", "tune up", "give back".
|
||||||
INSTRUCTION: Type the connecting word and the verb in its correct form.
|
INSTRUCTION: Type the connecting word and the verb in its correct form. Use each word from the box only once.
|
||||||
INPUT: type
|
INPUT: type
|
||||||
CHECK: exact
|
CHECK: exact
|
||||||
|
|
||||||
|
|||||||
@@ -221,7 +221,7 @@ INTRO: Listen to each question about Mark and Sam's conversation. Type your answ
|
|||||||
INSTRUCTION: Type your answer in a complete sentence.
|
INSTRUCTION: Type your answer in a complete sentence.
|
||||||
INPUT: type
|
INPUT: type
|
||||||
CHECK: llm
|
CHECK: llm
|
||||||
RUBRIC: Accept any sentence that correctly conveys the factual meaning of the model answer based on the dialogue.
|
RUBRIC: Accept any sentence that correctly conveys the factual meaning of the model answer based on the dialogue. For yes/no questions, a direct correct yes/no answer ("No, he isn't.") is acceptable without the model's extra detail.
|
||||||
REPEAT
|
REPEAT
|
||||||
|
|
||||||
PROMPT: Is Mark happy? {bk12-l2a-f3-01-q.mp3}
|
PROMPT: Is Mark happy? {bk12-l2a-f3-01-q.mp3}
|
||||||
@@ -973,7 +973,7 @@ ACCEPT: No, I'ven't finished it yet.
|
|||||||
TEMPLATE: yes, already
|
TEMPLATE: yes, already
|
||||||
PROMPT: Has Janet graduated from college yet? {bk12-l2c-f2-01-q.mp3}
|
PROMPT: Has Janet graduated from college yet? {bk12-l2c-f2-01-q.mp3}
|
||||||
RESPONSE: Yes, she has already graduated. {bk12-l2c-f2-01-a1.mp3}
|
RESPONSE: Yes, she has already graduated. {bk12-l2c-f2-01-a1.mp3}
|
||||||
ACCEPT: Yes, she's already graduated.
|
ACCEPT: Yes, she's already graduated. | Yes, she's already graduated from college. | Yes, she has already graduated from college.
|
||||||
|
|
||||||
TEMPLATE: no, yet
|
TEMPLATE: no, yet
|
||||||
PROMPT: Has Janet graduated from college yet? {bk12-l2c-f2-01-q.mp3}
|
PROMPT: Has Janet graduated from college yet? {bk12-l2c-f2-01-q.mp3}
|
||||||
@@ -995,7 +995,7 @@ ACCEPT: No, I'ven't seen her yet.
|
|||||||
TEMPLATE: yes, already
|
TEMPLATE: yes, already
|
||||||
PROMPT: Has your son learned Spanish yet? {bk12-l2c-f2-03-q.mp3}
|
PROMPT: Has your son learned Spanish yet? {bk12-l2c-f2-03-q.mp3}
|
||||||
RESPONSE: Yes, he's already learned Spanish. {bk12-l2c-f2-03-a1.mp3}
|
RESPONSE: Yes, he's already learned Spanish. {bk12-l2c-f2-03-a1.mp3}
|
||||||
ACCEPT: Yes, he has already learned Spanish.
|
ACCEPT: Yes, he has already learned Spanish. | Yes, he's already learned it. | Yes, he has already learned it.
|
||||||
|
|
||||||
TEMPLATE: no, yet
|
TEMPLATE: no, yet
|
||||||
PROMPT: Has your son learned Spanish yet? {bk12-l2c-f2-03-q.mp3}
|
PROMPT: Has your son learned Spanish yet? {bk12-l2c-f2-03-q.mp3}
|
||||||
@@ -1006,7 +1006,7 @@ ACCEPT: No, he'sn't learned Spanish yet.
|
|||||||
TEMPLATE: yes, already
|
TEMPLATE: yes, already
|
||||||
PROMPT: Have the Johnsons realized that they are rich yet? {bk12-l2c-f2-04-q.mp3}
|
PROMPT: Have the Johnsons realized that they are rich yet? {bk12-l2c-f2-04-q.mp3}
|
||||||
RESPONSE: Yes, they have already realized it. {bk12-l2c-f2-04-a1.mp3}
|
RESPONSE: Yes, they have already realized it. {bk12-l2c-f2-04-a1.mp3}
|
||||||
ACCEPT: Yes, they've already realized it.
|
ACCEPT: Yes, they've already realized it. | Yes, they've already realized that they are rich. | Yes, they have already realized that they are rich.
|
||||||
|
|
||||||
TEMPLATE: no, yet
|
TEMPLATE: no, yet
|
||||||
PROMPT: Have the Johnsons realized that they are rich yet? {bk12-l2c-f2-04-q.mp3}
|
PROMPT: Have the Johnsons realized that they are rich yet? {bk12-l2c-f2-04-q.mp3}
|
||||||
|
|||||||
Reference in New Issue
Block a user