D&D cities should be like icebergs

The majority of what happens in a fantasy metropolis should be "beneath the surface": whether literally (sewers, catacombs and ruins below the city streets); or metaphorically (secret plots and political machinations).

I've mentioned the city of Marchion (in Splintered Peace by David Chart) before. The megadungeon beneath Mage Hill is mostly implied - it is up to the individual DM to populate that location with specific dungeon levels and denizens.

The site of Marchion was settled in the distant past, but no one knows by whom. Sometimes strange artifacts are unearthed by people digging foundations for houses, and many think that Mage Hill isn't a natural feature. However, these inhabitants, whoever they were, were long gone before the first recorded attempts to settle the site. Local folk tales tell of the buried complexes they left behind, but no one knows whether these stories have any substance.

Aboveground, most of the mages' towers still stand, and some are still empty - guarded by the traps and wards that their erstwhile owners established. Over the centuries, parties of adventurers have cleared out many of the towers, and thus the residences of mighty wizards have been reduced to cheap housing for the lower classes. Some of the towers are still closed, either because no one has tried to enter them, or because the groups that did go in didn't come out. They provide opportunities for a classic dungeon adventure, with the ability to go home at night and sell the dungeon afterwards; Marchion law gives ownership to whoever clears them. One of the closed towers is Pharran's; it has claimed dozens of adventuring parties over the years, and thus has an ominous reputation.

Another good example is Ptolus (by Monte Cook) with its banewarrens and spire. However, that setting perhaps has too much detail. If I was going to run a campaign set in Ptolus, I think I would want to spend at least a year familiarizing myself with the 672 pages (6.8lbs hardcover) of material. Also, I'm not about to pay $45 for a PDF version of a book that I already own, and these days I DM almost exclusively using searchable ebooks on my laptop.

On the opposite end of the spectrum (in terms of volume, at least) is Vornheim, by Zak S. This is not a pre-packaged city sourcebook, but rather a collection of random tables for running an "urbancrawl" (as opposed to dungeoncrawl) adventure on the fly. There are lots of weird and colourful details here - sprinkle liberally through your own city, as needed.

In the real-world archaeological record, there are plenty of examples of cities built atop old Roman ruins and suchlike. In the Middle East, Jericho has consecutive layers of habitation built up over time, all the way back to 9,000 BC. In the archaeological site of Troy, the layers are numbered from Troy I (the oldest) up to Troy IX, just like the levels of a dungeon.

In D&D, with monsters (undead, constructs, dragons, etc.) that can live for thousands of years, there is no reason why these subterranean cities wouldn't still be populated.

If you want a city that will capture your players' imaginations and fuel adventures, then you need to go beyond simple demographics, innkeepers and merchants. Inject some mystery into your urban setting.