Or maybe it tells us the limits of what we all think of as reasoning. That's how I approach education. If it can be done by a computing machine, it never was a uniquely human ability. Radical sociologist have suggesting for more than a century that capitalist forces are shaping what we thing of as 'critical thinking' or 'creativity', but these aspects of human life are no more uniquely human than long division. Many academics stressed these aspects as what made their instruction distinctive and human. Now we are seeing this for the Emperor's clothing it always was.
The modeling used to design the LSAT takes random advance into account, or at least such kinds of models exist. The problem with modeling the score is that it assumes all students would use an LLM aid + their own ability. It would make it so you'd have to use one, and organizations like ETS and the ABA would have to allow them into test rooms. I'm not saying this is wrong or undoable. It would mean reconceptualizing the test.
I don't mean students would use an LLM to take tests, I mean tests should be designed so that LLMs do no better than randomness on them. Currently it's possible for LLMs--which can't reason--to do well on some tests of reasoning. That tells us the tests aren't good tests of reasoning ability.
I wish you could come and elaborate on this at my panel on ChatGPT in education. The Luddite response you see among teachers is something along the lines of creative thinking and critical skills. The understanding of these terms is not well elaborated and often contradictory. Much of it is trying to argue back from 'reasoned' conclusions of moral problems. This doesn't go very far for classroom teachers in contemporary society. I think a better understanding of what exactly it is that these machines are doing would help address this confusion.
Maybe my path to riches is as an LLM consultant for people in a moral panic over them. The paper on "reasoning" by neural nets that I've linked here [https://arxiv.org/pdf/1907.07355.pdf] is really interesting, and may be of particular interest to you as an expert on testing, as it makes clear how far you can get with just analysis of statistical regularities in language as cues to the correct answer, without any understanding of the subject matter. Tests like the LSAT should be designed so that LLMs do no better than chance on them, at least if those tests are intended to measure actual knowledge and understanding. Making sure LLMs can't game them will ensure there are no hidden cues that humans can use to guide their answers without actually understanding the material.
Or maybe it tells us the limits of what we all think of as reasoning. That's how I approach education. If it can be done by a computing machine, it never was a uniquely human ability. Radical sociologist have suggesting for more than a century that capitalist forces are shaping what we thing of as 'critical thinking' or 'creativity', but these aspects of human life are no more uniquely human than long division. Many academics stressed these aspects as what made their instruction distinctive and human. Now we are seeing this for the Emperor's clothing it always was.
The modeling used to design the LSAT takes random advance into account, or at least such kinds of models exist. The problem with modeling the score is that it assumes all students would use an LLM aid + their own ability. It would make it so you'd have to use one, and organizations like ETS and the ABA would have to allow them into test rooms. I'm not saying this is wrong or undoable. It would mean reconceptualizing the test.
I don't mean students would use an LLM to take tests, I mean tests should be designed so that LLMs do no better than randomness on them. Currently it's possible for LLMs--which can't reason--to do well on some tests of reasoning. That tells us the tests aren't good tests of reasoning ability.
I wish you could come and elaborate on this at my panel on ChatGPT in education. The Luddite response you see among teachers is something along the lines of creative thinking and critical skills. The understanding of these terms is not well elaborated and often contradictory. Much of it is trying to argue back from 'reasoned' conclusions of moral problems. This doesn't go very far for classroom teachers in contemporary society. I think a better understanding of what exactly it is that these machines are doing would help address this confusion.
Maybe my path to riches is as an LLM consultant for people in a moral panic over them. The paper on "reasoning" by neural nets that I've linked here [https://arxiv.org/pdf/1907.07355.pdf] is really interesting, and may be of particular interest to you as an expert on testing, as it makes clear how far you can get with just analysis of statistical regularities in language as cues to the correct answer, without any understanding of the subject matter. Tests like the LSAT should be designed so that LLMs do no better than chance on them, at least if those tests are intended to measure actual knowledge and understanding. Making sure LLMs can't game them will ensure there are no hidden cues that humans can use to guide their answers without actually understanding the material.