Setting the security level to set the permissions specified in /etc/permissions.secure takes the SUID bit off /usr/X11R6/bin/Xorg. This means that unpreveleged users can't run the startx command to get a graphical interface on a text console, say a machine in runlevel 3. The files and permissions in permissions.secure seem to be lagging the files and links in /usr/X11R6/bin as: 
lrwxrwxrwx  1 root root       4 2005-09-27 03:30 XFree86 -> Xorg*
-rwx--x--x  1 root root 1847788 2005-09-16 22:58 Xorg*
lrwxrwxrwx  1 root root       7 2005-04-28 18:28 Xwrapper -> XFree86*

/usr/X11R6/bin/Xorg                                     root:root         0711
/usr/X11R6/bin/Xwrapper                                 root:root         4755

Xorg gets its SUID bit stripped, Xwrapper is just a link so doesn't benefit from a SUID bit, and Xfree is left out entirely.

I'd suggest that if leaving a generally acessible SUID bit on Xorg is dangerous, that it's group be changed to video and only members of that group be allowed  allowed to run startx.

/usr/X11R6/bin/Xorg                                     root:video        4710
/usr/X11R6/bin/Xwrapper ( -> Xorg directly )            root:root          755
/usr/X11R6/bin/Xfree                                    root:root          755

Or else Xwrapper carry the SUID bit but be made a real wrapper that protects against misuse of a preveleged Xorg.

I can't see the sense of setting permissions.secure so tight that people are forced to back down to permissions.easy to use the system normally.