So, I finally dropped lighttpd/fcgi and switched back to Apache 2.2. Trac is running with mod_python now. I think I've tracked down the main cause of our load/memory issues - googlebot. It was indexing all 4000 some revisions of our svn repository via the trac browser... bad stuff, but it should be blocked from that by our robots.txt now.
Also, I've added WebDav for Subversion now. Paludis is at
http://svn.pioto.org/paludis, and the same is true for most of the other repos I host as well. Now those who have evil company/school firewalls and proxies can get through.
Update: SVN is sooooo 2007. Paludis is using git now: http://git.exherbo.org/paludis/paludis.git/