When first released in the late 1970s, improve on D&D rules and setting details they find constraining.
The introduction to the D&D 5th edition Player's Handbook describes three pillars of the classic D&D experience: Combat, Exploration, and Social Interaction. Combat and exploration, of course, are key parts of the classic "dungeon crawl," where players and their fantasy characters scout out each nook and cranny of an underground labyrinth, gathering loot, evading traps, and slaying monsters. Most of the hard rules in various editions of Dungeons & Dragons, particularly on the player side of things, are focused around the combat and exploration pillars, while social interactions are generally resolved on a more free-form narrative basis.
The RPGs listed below, like Dungeons & Dragons, are dedicated to the creation of fantasy stories where heroes explore dungeons, fight monsters, and interact with interesting side characters. Pathfinder and Shadow Of The Demon Lord, among other things, focus on creating crunchier rules for combat and spell-casting, letting players engage in more tactically intricate battles. 13th Age and Dungeon World have more narrative rules for diplomacy, investigation, and persuasion, putting less of a burden on the Game Master to design NPCS and social encounters from scratch. Quest, among its other qualities, is designed to be as accessible as possible, with simple base mechanics and modular rules new tabletop RPG players can quickly pick up.
D&D RPG Alternative: Pathfinder
The 1st edition of Pathfinder, released by Paizo Publishing, was designed to be a spiritual successor to Dungeons & Dragons 3.5 Edition, retaining the same D20-based skills checks and race/class abilities, but with revised rules and a new heroic fantasy campaign setting called Golarion. For a time, Pathfinder sourcebooks outsold the sourcebooks of D&D 4th edition by a notable margin.
The 2nd edition of Pathfinder, released in 2019 after extensive play-testing, has moved beyond its D&D 3.5e roots to become its own system, with new sub-classes, "Ancestries" in place of Races, and a more tactically versatile action economy for combat.
D&D RPG Alternative: Shadow Of The Demon Lord
Shadow of The Demon Lord is a dark fantasy blend of D&D and Warhammer-style fantasy roleplay, set in a grim world of adventurers fighting desperately to keep the Demon Lord from manifesting and bringing the end. The basic dice mechanics of Shadow of The Demon Lord revolve around players rolling a twenty-sided die, then rolling a handful of six-sided dice to represent the advantage of skills.
Instead of classes, tabletop RPG player characters in Shadow of The Demon Lord have Paths, character archetypes that start out simple (Warrior, Rogue, Priest, Magician) then branch out into more colorful and powerful specialties (Sorcerer, Druid, Duelist, Gunslinger, etc.). Rules for corruption and insanity let Game Masters represent the physical and mental distortion afflicting those who face off against the Demon Lord and its minions.
D&D RPG Alternative: Dungeon World
Dungeon World is essentially a translation of D&D-style dungeon-crawling to the more narrative gameplay of Apocalypse World and other "Powered By The Apocalypse" games. The character Playbooks correspond to traditional D&D classes, while their starting equipment and special abilities are built around narrative themes. The "Fighter" Playbook, for instance, is focused around a character's bond with their signature weapon and their un-nerving familiarity with violence.
When a Move such as "Hack'n'Slash" or "Defy Danger" is triggered during a session of Dungeon World, players roll two six-sided dice and add the relevant attribute bonus. Rather than a binary /fail, the rules of each Move are designed to create interesting story developments, even in the case of partial success or outright failure.
D&D RPG Alternative: Quest
The premiere publication of The Adventure Guild, Quest is a colorful fantasy RPG designed primarily to be intuitive and beginner friendly. The language of the core rulebook is very clear, straightforward and easy to digest, while Character Creation is as simple as writing out a description of a hero, then selecting six abilities from one of the game's "Roles," which are familiar classes such as Fighter, Wizard, and Ranger, along with more original classes such as Doctor, Invoker or Spy.
Quest's rules for action scenes are also designed to be easy grasped, with a simple action economy of "move and do one thing," along with a D20 dice mechanic that doesn't require the player to add numerical bonuses.
D&D RPG Alternative: 13th Age
The 13th Age RPG combines the D20 tactical rules of Dungeons & Dragons 4th Edition with the storytelling mechanics of narrative systems like FATE or "Powered By The Apocalypse." Player of 13th Age create and customize their characters by asg points to attributes, choosing backgrounds that denote their skill sets, assembling a list of class/race features, talents, and feats, then adding "One Unique Thing," a special character feature they create from scratch.
The core setting of 13th Age is a detailed world of fantasy and adventure dominated by 13 "Icons," powerful NPCs and antagonists embodying archetypes such as "The Archmage," "The Crusader," "The Elf Queen," "The Great Gold Wyrm," etc. The personalities, names, and histories of these Icons are blank spaces both players and Game Masters will define over the course of their campaign.