People have complained that the authority list of X growes over time since stale
credentials don't get removed. Apart from crashing login managers there used to
be two sources that left entrires behind even under normal circumstances: startx
and ssh. startx has been fixed a while back already and should remove all
credentials now. With #98627 this should even work when the hostname has changed
due to dynamic host name setting.
ssh still remains a problem. I've looked into it and it doesn't seem to be too
easy to fix: 
The ssh server forks, the child sets up the environment for the user session
calls  either sshrc or xauth to set up the credentials and then execve()s the
user shell or command. From then on it has no more control over the process,
thus the child that has set up the credentials can no longer remove them.
This would have to be done in the father process which waits until the child
exits. The changes necessary there however go beyond what I feel comfortable to
do with my limited knowledge of ssh server internals from my casual review.
All this takes place in session.c.
A possible (kludgy!) workaround would be to use some bash logout script to do
this task. It would have to detect if an ssh session was running and remove the
local display of this was the case.