In retrospect, Pop was the beginning of the still-ongoing U2 career-curve, which goes like this: Spend as long as possible recording an album, with as many producers as possible, spending as much time and money as can be justified, only to re-record the same songs ten thousand times until their vitality has been sapped, and then embark on an endless, time and energy-consuming tour of stadiums (with corporate sponsorship -- also a new thing that the Pop-era brought). How well or poorly this career-curve works varies with the commercial success of the relevant album.
But more to the question by the OP -- and besides the endless time spend on sessions, as mentioned -- what I think went 'wrong' with Pop is that its period simply marks the end of U2's main and most vital creative burst. This is nothing to be ashamed of, as it happens to every stable combo after a certain period (usually 6-7 years, or 4-5 albums, whichever comes first -- for example, The Beatles collectively and solo after about 1972, or Prince after 1987). U2's peak as a creative combo (and as recording artists, I would argue) was roughly 1983 to 1993, obviously an incredible period of success for them. I think, with the 1996 sessions, they hit a wall for the first time, where it became difficult to generate ideas naturally and organically (the more so given the incredibly high standards they set for themselves -- this in itself necessitates staring down brick walls far more than an average group in their sales bracket would bother with). In other words, I think, in 1996-ish, U2 started generating ideas intellectually. While this inevitably led to some good ideas and some great songs, it also decisively ended the era of U2 being vital artists.
Bob once said, regarding the year 1974 (when he wrote the Blood On The Tracks songs), that, after going to art classes in New York City, he had to "learn to do consciously what I used to do unconsciously". He certainly succeeded with that particular set of songs, but only sporadically thereafter. I think this is broadly comparable with U2 from the mid-1990s onwards.
(Saying this, I don't mean to discount other fairly obvious factors in Pop's lukewarm reception, such as the lack of a big radio song, people's tastes moving on to post-grunge and Britpop, the poorly received video, the overblown tour, etc. But I think focus on those -- or the band's own absurd 'excuses' for why the record "failed" -- is to overlook the larger matter.)
Eventually, you reach a point as an artist when you can't keep going in the same line of artistic pursuit anymore. That one direction may have involved all sorts of diverse artistic results, but it all occurred in one's original, most 'natural' method of creation. Reaching this point of not being able to continue creating in such an organic way, the artist has to decide (itself an 'unnatural' self-conscious process) whether to continue repeating what's been done, or whether to keep developing artistically, but in newer, less natural, highly self-conscious ways.
U2 chose the latter, and they're the better for it, even if they'll never be as artistically vital as they were. That's not their fault. Pop simply marks the point when U2 reached this inevitable stage in an artist's development.
