Gaining XP for recovering loot is the cornerstone of level progression in D&D, or at least the early editions. You can remove it, but if you do you’ll have to make other changes to keep the whole thing from collapsing. People deride it as “unrealistic” in the sense that suddenly being given a wad of cash in the real world doesn’t make you any more skilled, or argue that “wealth is its own reward”, but it serves a vital function in the dynamic of the game. In essence XP for loot makes D&D a game about recovering treasure, not fighting monsters, and that’s a huge difference in how things play out and what strategies make sense.
In early editions, the XP for killing monsters is pitiful compared to the XP for the monster’s treasure. Typically treasure is worth ten times as much as a monster. This means that players are able to advance quite nicely even if they manage to avoid fighting monsters completely, and that they advance relatively quickly if they’re successful. If you take away XP for treasure then, at around 100 XP per monster hit die a party will have to defeat 20 encounters of equal numbers and level for a fighter to advance to second level, instead of two or three. All of a sudden the fragility of low level D&D characters becomes a much bigger issue. If a single blow can kill you, but you only have to survive a handful of well-chosen battles to advance it’s a completely different proposition from needing to survive twenty such fights in a row.
Changing the game from needing a couple of big treasure scores in order to advance to having to grind through twenty dangerous combat encounters completely changes the complexion of the game. Many of the things that are regarded as problems with old-style D&D are really problems with old-style D&D once you remove XP for loot. If expeditions in the dungeon are concentrating on finding loot and getting out with it, with as little fighting as practical, you have to press ever deeper to get a good pay-off. In addition, since combat is a relatively small part of the reward, and hirelings are relatively cheap, it’s viable to try to have enough force to overwhelm what opposition you can’t avoid. You’ll get almost nothing from the combat itself, but the treasure more than makes up for it; still, to pay off, you have to get in, get the loot, and get out…
Take away XP for loot, and you introduce the “15-minute workday” and “scumming” (bumming around the upper levels of the dungeon and retreating or holing up as soon as you’ve expended any resources, be they spells or hit points). Since the only way to advance is fight stuff that’s on the whole as tough as you, it doesn’t really matter what you fight or where… if the parts of the dungeon that you clear out near the entrance get restocked by wandering monsters, so much the better. The only reason to go deeper is to find more monsters, or monsters of your level if you’ve leveled up. Taking extra hands with you to help fighting just means you’ll have to have that many more fights. And since it’s pretty much pointless to explore or try to find treasure if you’re not looking for a fight, as soon as a fight would get dicey (which for a low-level D&D party is as soon as anybody has expended any spells or taken any hits), the only sensible thing is to “turtle”–hunker down until you’re in fighting trim again. A party looking for treasure… preferably treasure that’s lightly guarded or protected by traps or obstacles they can plan their way around can profitably press on even when their resources are depleted. A party looking for a fight, not so much.
Shifting the focus of D&D to combat has a number of other unfortunate effects. For one thing, combat in D&D is pretty abstract, and not all that interesting. Combat is also quite hard for low level D&D characters to survive, and for Wizards and Thieves to participate very meaningfully in. (It’s not so much that a dart or dagger are that bad as attacks at low levels, as the poor armor and low HP are basically a death sentence if you have to survive twenty combats to level up–twenty-five if you’re a wizard). Removing the XP for loot is usually the start of a death spiral of “improvements”: increasing the HP so they’ll survive to level up, but then combats are even less interesting since there’s less on the line, so criticals are added to spice it up, but then characters die too much, or spend almost all their game time in turtle mode so healing and even resurrection are made cheap and plentiful so they can get back to the business of combat. Wizards are given spell points so they never have to choose between combat and non-combat spells, then more spell points or “at will” spells so they always have something to do during combat, which is now the complete focus of the game, and so on.
Some GMs give out XP for “defeating” monsters without necessarily killing them, or as “story awards”, but it’s generally pretty ad hoc. In an XP for loot game of D&D, the players know exactly what they can do to get that XP… and possibly even how much it is if they’ve gotten some information about what the treasure contains through scouting, rumors and treasure maps, talking to the denizens of the dungeon, and such. How liberally the GM will interpret defeating the monster by bypassing it all together (if they find a way down to the level with the treasure vault skipping over a level in between, do they get XP for all the monsters on that level? Doubtful). When and how much they’ll get from “story awards” is much murkier and hard to plan around…for instance knowing whether it’s worth hiring some specialist or extra men at arms to push for accomplishing an objective. And when all is said and done, such awards usually don’t keep pace with the awards from loot at one XP = one GP. This is probably just a psychological hurdle, but somehow once XP for GP is out the window, GM’s seem to think it’s “cheap” to award XP in multiples of what the party got for killing things, even though if you don’t you’re probably not reproducing the advancement curve of the original game.
Obviously, despite all this, many people play D&D without XP for loot awards… in fact, giving out XP for GP is probably terribly old-fashioned. Still, I think it’s worth considering how D&D was designed around certain assumptions about how fast you could advance for what sorts of activities, particularly while avoiding or limiting combat, so if you want to make changes you can compensate. I feel like I see too many people who’ve never actually played it the way it was intended, and then been disappointed and blamed the game for faults that they themselves introduced. It’s fine, more than fine, to prefer more recent editions or other games entirely (I certainly do, that’s why I designed the SFX! games), but I think it’s better when you do that fully appreciating what the older editions do if you play them as designed.
Tales of the Rambling Bumblers 
You speak so true, if you think of the pc’s as adventurers. What do you think of the following idea. XP equals accumulated wealth originating from or used to:
Warrior: spend on training and weaponry; henchmen
Cleric/Paladin: spend on your church/faith
Thief: spend on your guild and shadow affaires (bribery); or
Wizard:spend on magical goods, ingredients and tome;
And earned according to your alignment.
No idea what to do with a druid though and other folk without the concept of wealth.
That’s a reasonably common kind of house-rule (I think Dave Arneson did something like that). I think it has its pluses and minuses. On the one hand, it does provide some kind of RP justification for why the loot is translating into some kind of class-relevant experience, which is nice. On the other, applied rigidly it can penalize RP (if you’re Robin Hood, giving the money to the poor better count or you get short-changed on advancement), and it might be unbalancing if some classes get to spend it on stuff that improves their adventuring (weapons for the fighter) while others have to squander it (give to the church for the cleric). It can also slow down the campaign by enforcing waits while you spend the money if the GM has limits on things like just how many henchmen or magical goods you can find in an area. On the whole, I like it as long as the GM is generous in interpreting what you can do to justify turning the loot into XP.
Personally, I prefer an ad-hoc system where the GM just hands out some amount of XP at the end of a session or an adventure. It removes the metagaming element where adventurers are calculating their XP-per-hour or loot-per-hour, or whatever. I realize that’s somewhat tangential to the point, and I only bring it up because I know that’s the system the original Star Frontiers used, which was contemporary with red-box D&D. And that makes me curious what the other early TSR games used, such as Gamma World or Top Secret.
The problem with XP for loot is that is says “We are going to play a game with people who go into a dungeon to steal stuff.” It excludes billions of game concepts that aren’t centered around getting loot. “We’re the Kingdom’s private forces that they send in whenever there’s a problem.” “Our nation has withered under the thumb of the tyrant and we are fighting to free ourselves.” “Our clan has been granted land on the wild frontier. If we can tame and civilize it, we’ll be nobility.” “We’re the city guard of Southport. The criminals run this place but we intend to take it back.”
Yes, you can shoehorn treasure into those concepts, but it changes them.
My solution is to award xp for dealing with challenges. This is more than just killing them. It can be sneaking past them, fast-talking them, trapping them, disarming them, finding a route around them, etc. If you sneak into a dragon’s lair and rescue the princess, you’ve done something heroic. Under the killing rule or the loot rule you get nothing. I give them the full XP because they’ve dealt with the problem. Of course they might end up getting more xp when the dragon comes looking for them, but that’s a new challenge.
That’s not a problem with XP for loot, that’s a problem with using the game for things it wasn’t designed for. D&D was designed as a game about going into a dungeon and stealing stuff. Obviously you can use it for other things, but you have to adjust other parts of the design in order to compensate….and to do that adequately it helps to understand the design. XP for “dealing with challenges” is almost as old as the game itself (maybe older, based on some descriptions of how Dave Arneson ran things), but to do it without frustrating players requires at the very least making sure that the individual rewards the PCs get are on par with the rewards for loot, and that if you start shorting the players on the rewards for doing what was smart behavior in the original design (e.g. they bring along a bunch of hirelings to help with the fighting, but you reduce or divide the “challenge” XP in a way it wouldn’t have been to when the bulk of the XP was for loot) then you compensate for all the added danger their fragile characters will have to endure before they can level up.
To use the metaphor from the post, you can remove the keystone, but you’ve got make sure you put up a scaffolding to compensate. If you don’t, it’s no fair blaming all the resulting problems on the inadequacies of the original design. For many players, changing D&D from a game about recovering treasure from ruins into a game about rescuing princesses and doing other heroic deeds required a whole lot of extra rules and design work… or even a new game entirely.
Speaking of “inadequacies of the original design”, the old games weren’t very explicit about telling the players their goal wasn’t to kill monsters. I remember way back in the day when I played those as a preteen, we felt that we weren’t successful unles we killed all of the monsters in a dungeon. The goal of avoiding combat was something that could be inferred by the rules in the DM’s section. It should have been very explicitly said to players. Heck, a lot of people outside of the OSR still don’t get this. Learning this fact transformed the OSR from “What are those crazy people thinking?” to “Oh, they just like a different play style than me.”
I agree with you that if you change the basis for XP, you have to balance it out and make sure that you give a reasonable amount of XP for the pace of advancement you want.
Not only does challenge XP expand the scope of the game, it stops you from having to torture logic to justify treasure as xp in world. I haven’t heard of a in-world justification that doesn’t break down with a couple of common cases.
I’ve always liked this idea, and I think you do an excellent job of spelling out why it’s not only a viable way to play, but what the consequences are for moving to the “XP for homicide” game-style.
For a fun twist on the XP for GP model, I once mentioned in a blog post how Fiery Dragon’s Mastering Iron Heroes has a similar idea, where PC’s only gain XP for GP that they spend on goods and services with no game-mechanical reward. For example, spending 15 gp on a new sword gains you nothing, since the sword has game stats. Spending 15 gp on ale and whores earns you 15 XP, since there’s otherwise no mechical gain for spending that money.
This system keeps the PC’s from perpetually on the make, and therefore continually in search of adventure. It also ties XP and material rewards much closer together (e.g. “do I spend this gold on a new magic sword, or on getting closer to levelling?”), which I think is a good thing.