I enjoy programming and fabricating complicated redstone contraptions to automate processes, so personally, I can't stand vanilla Minecraft. When we were modded, I played it a bit, but after having things stolen I quit. The Direwolf modpack is nicely balanced and has a lot of content for people to explore before quitting after reaching end game. I believe there is a server version of the mod pack that removes some of the objects that would constantly cause lag by reloading or processing information at obnoxious rates.
As for griefing problems, I know there is a plug-in that records the moment a block placed by one player is destroyed by another and saves the chunk's state before the block was destroyed. An admin can then access the plug-in, seeing who destroyed the block and reverting the building to its previous state.
There was also another plug-in that gave each player an amount of plots they could purchase (with in game currency) and control/build in. Other players could not access this land and no one could access land that had not been bought. I believe the amount of plots you could have was based on rank.
And for the money problema, perhaps there could be a Minecraft specific subscriber, I might actually pay for that.
I have also been on servers that take multiple game modes and spread them out on multiple dimensions within one server, dunno how. They were also able to maintain a high population with little lag, dat elasticity though.