The gap between standard scoring and complete answer scoring for Gemini Flash is striking. What do you think explains that model losing so many points on completeness specifically?
in part, it is explained by the fact that these tasks are subjective and perhaps no two instructors or experts may agree on the rubric. I suspect that even really good MBA students are Mackenzie endless will score similarly although I do not have clear evidence to support this claim.
MR.KARTIK HOSNAGAR, John C. Hower Professor, Renowned Author, Professor of Operations, Information and Decisions,
Co-Director, Wharton Human-AI Research,Professor of Marketing,in Creative Intelligence post—AI versus the MBA—navigates the upcoming AI economy and shares deep research based insights on—What skills will entry-level knowledge work require and how should MBAs and undergrads be trained?
The post is worthy of a complete book and contains latest Research and insights by MR.KARTIK on the intersection of AI and business and draws on:
# MR.KARTIK HOSNAGAR’s new research,with his coauthors that raises a question: what happens when AI becomes remarkably good at the entry-level tasks that most college and MBA graduates perform…
## MR.KARTIK HOSNAGAR’s BusinessCaseBench, a benchmark based on 238 business school cases spanning 18 disciplines,...
### MR.KARTIK HOSNAGAR’s recent HBR article—How do you market to an AI customer…
####MR.KARTIK HOSNAGAR’s recent keynote at the just-concluded CommerceNext conference...
An aspect that is perhaps beyond the scope of your project, however is the role of AI - in-house or internal consultants.
If AI provides the analytical framework & the model is fine tuned internally, high end expensive firms may have to move further up in the value chain too. Research in management theory ? And then what about academics in this space.
In fact AI labs are already recruiting retired/ self employed executives to enhance their models. So current knowledge and its distilled use is not an entry level analyst job, imho.
The article highlights an important shift: AI is getting very good at producing strong first drafts of complex business analysis—but it’s not yet consistently delivering decision-ready answers.
That’s where human judgment becomes even more critical.
The real skill is no longer just building the analysis, but evaluating it, stress-testing assumptions, identifying gaps, and shaping it into something actionable. That capability doesn’t come from prompts alone; it’s built through experience.
I strongly agree with the point that education, and on-the-job training, must evolve toward developing critical thinking and verification skills. Understanding how to challenge an answer may become more valuable than generating one.
AI will accelerate the work, but experienced judgment is what ultimately drives better decisions.
The gap between standard scoring and complete answer scoring for Gemini Flash is striking. What do you think explains that model losing so many points on completeness specifically?
in part, it is explained by the fact that these tasks are subjective and perhaps no two instructors or experts may agree on the rubric. I suspect that even really good MBA students are Mackenzie endless will score similarly although I do not have clear evidence to support this claim.
MR.KARTIK HOSNAGAR, John C. Hower Professor, Renowned Author, Professor of Operations, Information and Decisions,
Co-Director, Wharton Human-AI Research,Professor of Marketing,in Creative Intelligence post—AI versus the MBA—navigates the upcoming AI economy and shares deep research based insights on—What skills will entry-level knowledge work require and how should MBAs and undergrads be trained?
The post is worthy of a complete book and contains latest Research and insights by MR.KARTIK on the intersection of AI and business and draws on:
# MR.KARTIK HOSNAGAR’s new research,with his coauthors that raises a question: what happens when AI becomes remarkably good at the entry-level tasks that most college and MBA graduates perform…
## MR.KARTIK HOSNAGAR’s BusinessCaseBench, a benchmark based on 238 business school cases spanning 18 disciplines,...
### MR.KARTIK HOSNAGAR’s recent HBR article—How do you market to an AI customer…
####MR.KARTIK HOSNAGAR’s recent keynote at the just-concluded CommerceNext conference...
Great perspective on knowledge work.
An aspect that is perhaps beyond the scope of your project, however is the role of AI - in-house or internal consultants.
If AI provides the analytical framework & the model is fine tuned internally, high end expensive firms may have to move further up in the value chain too. Research in management theory ? And then what about academics in this space.
In fact AI labs are already recruiting retired/ self employed executives to enhance their models. So current knowledge and its distilled use is not an entry level analyst job, imho.
The article highlights an important shift: AI is getting very good at producing strong first drafts of complex business analysis—but it’s not yet consistently delivering decision-ready answers.
That’s where human judgment becomes even more critical.
The real skill is no longer just building the analysis, but evaluating it, stress-testing assumptions, identifying gaps, and shaping it into something actionable. That capability doesn’t come from prompts alone; it’s built through experience.
I strongly agree with the point that education, and on-the-job training, must evolve toward developing critical thinking and verification skills. Understanding how to challenge an answer may become more valuable than generating one.
AI will accelerate the work, but experienced judgment is what ultimately drives better decisions.