I've been spending a lot of time on tabletop RPG design in recent years. My current major project along these lines is the latest iteration of my D&D-inspired system, Vanquish Vanguard.
Vanquish Vanguard was inspired by a variety of factors. D&D 5th edition was the main system my roleplaying groups played for years. 5e is a great system and fun to play and DM, but I've been inspired to make my own for two major reasons:
These factors, combined with my inherent love of game design, led to the birth of VanVan. Its design has evolved a lot over the few years of its development. One thing I've wanted to try to do with VanVan from the start is keep it fairly simple. To me, spending long stretches of time combing through rules and doing lots of fiddly bookkeeping is not what makes tabletop RP fun, so I want a game that plays fast and is easy to understand.
However, the earliest iterations of VanVan took this ethos a little too far. Although simplicity is good, in a D&D-like RPG with a heavy emphasis on combat and tactics, you also want a robust character creation system with lots of interesting options and a combat system that provides interesting tactical play. As VanVan has evolved through multiple iterations, I've basically built on the first overly-simplistic core and added options and mechanics to restore the richness of tactical combat and character building while (hopefully) sticking to that ethos of simplicity.
In some ways, VanVan can be thought of as the ultimate set of D&D homebrew rules, because the D&D inspiration at its core is very evident. I've essentially pulled my favorite aspects from different editions of D&D, thrown in inspirations from a few other fantasy franchises (like some Final Fantasy-inspired class options and monsters), added a healthy dose of my own original material, massaged it into a seamless brew, and the potion that flows out is VanVan. The current version, now undergoing rigorous playtesting and likely the final major iteration because playtests are going very well, is essentially a mix of 5e and 4e (4e had a lot of great mechanics that got foolishly discarded because of the strong fan backlash to that edition), with smatterings of 3e and B/X thrown in, plus a bunch of original material.
So far, I find that the current iteration of VanVan is working out very well. It plays smoothly, and playtesters are having a lot of fun with it. Within the "fantasy tabletop RPG with a heavy emphasis on tactical combat" space, I think I can say it's the most enjoyable system I've played, even surpassing D&D. But then, as the creator, my opinion is obviously very biased.
Shifting gears slightly, the other system I've been using in my tabletop RPG adventures lately is the sci-fi RPG system Traveller, the current version published by Mongoose. This is a VERY different beast from D&D. In some ways, it is extremely rules-light compared to D&D. Characters are basically just a race, core stats, and set of skill ranks. The core mechanic consists entirely of rolling skill checks and trying to beat a target number, and there's not too much to know beyond that. In that respect, it's a fairly standard skills-based system. There's a little bit more to combat, but the combat likewise is simpler, and since characters don't get a bunch of spells and combat maneuvers to use, combat options are comparatively much more limited.
At first blush it seems TOO simple, but in practice it's actually a lot of fun to play. For one thing, there's a lot less emphasis on combat in Traveller than in D&D. That isn't to say there's NO combat (the party has gotten in plenty of fights in the campaign I'm running), but in Traveller, going an entire session with no fights is not unusual, whereas in D&D that's a relative rarity. I've found the variety of gameplay types in the campaign I'm running to be pretty impressive, from social interactions to investigating mysteries to engaging in speculative interstellar trade and to, yes, sometimes getting in firefights with space monsters, haywire robots, mercenaries, and so on. In some ways, the rules being less focused on combat and most actions devolving to simple skill checks throws more initiative back into the imaginations of the players and is kind of freeing.
There's also the simplicity of stat blocks and the like. Even in a relatively simple system like VanVan (simple compared to, say, Pathfinder at least), a monster or NPC stat block can be a somewhat meaty chunk of text. By comparison, here's a typical stat block for a Traveller space monster:
TENSHER'S WOLF Hits 36 Speed 12m (8 spaces) Skills Athletics 1, Melee 3, Recon 2, Stealth 2, Survival 3 Attacks Bite (3D) Traits Armor 2, Camouflaged
There is something beautifully elegant about encapsulating all of a creature's relevant characteristics in such a small chunk of data. And NPC blocks aren't much more complicated:
YIPPER Male Vargr
STR 6 INT 7
DEX 9 EDU 7
END 7 SOC 7
Skills Astrogation 1, Deception 1, Gun Combat (energy) 1, Melee (blade) 1,
Persuade 0, Pilot 1, Streetwise 0
Weapons Laser Pistol (+2, 3D, range 20m), Cutlass (+2, 3D)
Armor Flak Jacket (+5)
That's not to say that EVERY aspect of Traveller is simple. It has some pretty robust rules for interstellar trade (which my players have made considerable use of, lining their pockets significantly in the process), detailed star system and planetary classification systems, rules for starship construction, specialized combat rules for fighting in vehicles and starships, and so on. But the core of the game is beautifully simple and works well, with more of an emphasis on freeform roleplaying adjudicated with occasional skill checks, and less emphasis on a robust tactical combat system. I'm not saying it's inherently better, but it's an interesting contrast in style, and it plays well.
That's not to say that Traveller is perfect. From running a campaign so far, I've found 2 major things I'd be tempted to house-rule if I run more games with this system in the future:
Anyway, that's a lot of words to say there's something very appealing to me about the elegant simplicity of Traveller's system, and it's something I'm going to keep in mind as I design more of my own RPG systems going forward.