The secret of all studies of this type is simple-- make the humans follow the same algorithm used by the computer rather than he kind of scoring that an actual English teacher would use. Right off the bat, Tovia Smith offers three advantages of robo-grading: cheap, quick and without human bias.
Use some big words.
Highlight the most relevant and irrelevant sentences of the essay. An approach to automatic grading, followed by AEA, is to automatically grade essays by comparing them to learning materials and human-graded essays.
Repeat words from the prompt. The unhappy lesson there is that the robo-graders merely exacerbate the problems created by standardized writing tests. The full dismantling is herebut the basic problem, beyond methodology itself, was that the testing industry has its own definition of what the task of writing should be, which more about a performance task than an actual expression of thought and meaning.
The king of robo-score debunking is Les Perelman.
The point is that their inability to distinguish between good writing and baloney makes them easy to game.