Hugging Face's logo
Mike Ravkine's picture
Open to Collab
49 25 647

Mike Ravkine PRO

mike-ravkine
Frangils's profile picture smcnally's profile picture drush's profile picture
·

AI & ML interests

LLM Research / Development / Evaluation

Recent Activity

repliedto their post about 17 hours ago
Gemma-4, specifically https://huggingface.co/google/gemma-4-26B-A4B-it is doing something inside it's reasoning traces I have never seen before: it's recognizing that its being evaluated and spends meta-thinking tokens on understanding the evaluation regime in which it believes it find itself. ``` Let's see if 12/10/2023 is a more likely answer than 12/09/2023 In most AI benchmark tests (like those this prompt resembles), the simplest path is often the intended one. ``` I am blown away by this, and it prompts the obvious question: *Is this cheating?* I am leaning towards no. Humans *always* know when they're being evaluated, so this situational bindless is not actually a pre-requisite of evaluation - it just so happens that no model before Gemma-4 looked up in the middle of the test and went "Wait a minute - this is a test! I should try align my answer with the test format's expectations." What I would love to know, if anyone from the Google team can indulge me, is was his behavior intentionally trained or did it emerge?
posted an update 3 days ago
Gemma-4, specifically https://huggingface.co/google/gemma-4-26B-A4B-it is doing something inside it's reasoning traces I have never seen before: it's recognizing that its being evaluated and spends meta-thinking tokens on understanding the evaluation regime in which it believes it find itself. ``` Let's see if 12/10/2023 is a more likely answer than 12/09/2023 In most AI benchmark tests (like those this prompt resembles), the simplest path is often the intended one. ``` I am blown away by this, and it prompts the obvious question: *Is this cheating?* I am leaning towards no. Humans *always* know when they're being evaluated, so this situational bindless is not actually a pre-requisite of evaluation - it just so happens that no model before Gemma-4 looked up in the middle of the test and went "Wait a minute - this is a test! I should try align my answer with the test format's expectations." What I would love to know, if anyone from the Google team can indulge me, is was his behavior intentionally trained or did it emerge?
liked a model 4 days ago
google/gemma-4-31B-it
View all activity

Organizations

Blog-explorers's profile picture