Great OP, and very interesting thread so far!
My guess is that any and all of the root causes set forward by various posters apply to many individuals. For me, the one that really rings true is that the more striking the watch is (in unconventional visible ways), the faster I tire of it. I first found the PuristS in 2004 as I searched for a PP 5015, which I found devastatingly attractive but wore less and less. More recently, i bought a PSM Thalassa sight unseen, and within months determined that the dial was just "too blue." Same thing, by the way, for the UN Maxi Marine chrono with the blue dial, and even for the UN Executive Dual Time -- just too harsh on the eyes after a bit, despite the initial "wow" factor.
A potential exception (I hope) is the VH Antiqua, which certainly has "wow" and is unusual looking, but is also both horologically significant and just gorgeously made. It also has a visual feature that plays well to my eye, which is the asymmetric layout. As an amateur/hack watch photographer, even I know that there are certain visual layouts that appeal to the eye, and some that do not (rule of thirds, triangles, etc.). Some of these rules seem fairly universal, and others I would suggest are more personal. For me, the idea of asymmetric layout really plays -- so when I peek at my wrist, it makes me happy. The 5015 fit this bill (but was ultimately doomed by its more flamboyant visual cues); I have owned several watches based on the JLC dual time movement with its distinctive PR, date, and second time zone displays, the Antiqua, even the slightly differently sized sub-dials on the VC OS chrono. I'll hazard a guess that if you look at all of your watches, you will find that several of the keepers have similar visual cues of some type.
Best,
Gary G