There are a few issues if it’s AI-written/edited/drafted:
1. AI generated text works off of whatever model it came from, whatever prompt was used, and probabilities. It’s been described as a “next token predictor” and that, for the purposes of this answer, is how I’m describing it as well. First problem: it’s only as good as the above.
2. Because of the above, its accuracy relies on probabilities. It can confidently give me wrong information, and as someone who would read this book to learn a topic I don't already know, I have no frame of reference to know whether I’m being fed accurate information.
3. Since it greatly speeds up the time it takes to physically write material, there is a possibility the author didn’t closely vet the information provided by the LLM as a next token predictor. That means it could be wrong in subtle ways, and the only way for me to know is to rely on knowledge I don’t have.
4. Had AI not been used, I could conceivably go on the past work of the author, rely on the person who has the expertise who wrote the forward, or the fact that physically writing a book takes prohibitively long enough that the dopamine from having written a book doesn’t outweigh the amount of time it takes, and if you are going to take months to years to write a book, you’re going to not want that effort to be in vain. You’re going to do your diligence and not just “wing it”.
AI makes it relatively trivial to appear knowledgeable and to get that dopamine rush from saying “I’m an author” and appearing to be an expert in a field without being an expert in a field. For some, that temptation is too great and there’s no way to know without the passage of time. Time for folks to review the book who know the subject matter. Time for reviews. Time for folks to put signs up that say “don’t trust this material.” In the meantime we still have that content out there, further infecting new AI models and people into believing it’s accurate.
That’s why the use of AI is dangerous, and is especially dangeorus without disclosure.
As a professional mathematician it seems that AI is already at the point where, when well guided, it both does better research and writes better exposition than run if the mill mathematicians.
Well guided is a qualifier that matters.
In my opinion, doing good mathematics is very different from writing good exposition.
I have observed that models are not able to effectively come down to a human level. They are mediocre at approximating what a first-time reader would want to know. If you will forgive the anthropomorphization, they are not very empathetic. In this book for example, the author pauses the flow of thought with countless little anecdotes, anticipating where the reader might have a question or want to know more.
Of course, a well-guided AI is capable of anything a human is capable of. But is that not tautological?
Sure, but you can build an agentic system which verifies outputs and fact corrects, with probably better accuracy than a human would.
As well, if people were a bit less anti-AI then maybe we can get disclaimers that things are written by AI that don't immediately torpedo any value the writing has.