Where generated PR descriptions fall short
A model can summarise a diff. It cannot tell a reviewer whether the change is right. What the tool does, and the three things it will get wrong.
Generated PR descriptions are genuinely useful and genuinely limited, and it is worth knowing which is which before you rely on one.
What it does well
Grouping a scattered diff into coherent changes, and producing a first draft of the what-changed section. This is real work that used to be done by hand, and the draft is usually better than an empty textarea.
Three things it gets wrong
- Intent. It can describe that a check was added. It cannot say that a customer hit this in production on Tuesday.
- Risk. It has no way to know that a three-line change touches the payment path. It judges by size, which is close to the opposite of the useful signal.
- Omission. It summarises what is in the diff. If the important thing about this change is that you deliberately did not do something, the draft will not mention it.
The rule we built to
PrGen shows which parts of the description it generated from the diff and which are waiting for you to fill in. A description that looks complete when it is not is worse than an obviously unfinished one, because an unfinished one gets finished before it is sent to a reviewer.