Been using Gentoo for yonks now so I might be able to help you out a bit. Reverse dependancies are still something of a weakness of portage. Anyways, removing X and anything that depends on it will remove a LOT of stuff but if you REALLY wanna do it
First make sure you have the ultra useful gentoolkit package installed (emerge gentoolkit). Then type:
# equery depends xorg-x11
You'll might have to be a bit more specific than that though so something like "x11-base/xorg-x11" may be needed (can't test it 'cause I'm away from my Gentoo boxes).
This will generate a list of all the packages that depend on xorg. Get that to output to a file and then check that there's nothing silly in there, you could be surprised what depends on X.
# equery depends xorg-x11 > package_list
Will output to file. You might need use the nocolour option for equery or something to avoid weird character strings (again can't test it ATM).
Once you've checked the file listing you can then hand it off to emerge but you'll have to stick an = at the beginning of each package name (e.g. =media-sound/xmms-1.2.10-r9) or take out the version numbers. You can probably just edit the file and type
# emerge `cat package_list`
Hope this helps.