Lots of misinformation in this thread regarding charts and sales.
In terms of sales and hits, R&H was #1 in the U.K. and the U.S. It sold 5M copies in the U.S. - U2's second biggest selling album at the time. R&H produced two big hits in the U.S., "Desire", which went Gold (U2's first Gold single in the U.S.) and "Angel of Harlem". These songs hit #3 and #13 on the U.S. Billboard charts, respectively. However, both "When Love Comes to Town" and "All I Want Is You" charted in the Billboard Top 100, which makes four top 100 hits in the U.S. - far better than U2's last four albums ("Zooropa" through HTDAAB).
As for the "failure" aspect of it... it's true, critics thought U2 was placing themselves up there with the "greats" by recording with Dylan and B.B. King and others - when, as we know, U2 was paying homage to them. Still the movie, well, it's not really that good. It's fun to see from a fan perspective, but that's about it.
However, I don't think AB came out of any "failure". The album sold very well and produced several hits and the movie was a Top 10 hit as well, making over $8M in the U.S. just at theaters (a good sum for that type of film, especially in 1988). And I think U2 could have withstood the critical bashing too.
I won't say that this bashing didn't play any role, but I think the real reason for the change in sound was because U2 needed to do something different. All of them complained about the Love Town tour - how they felt they were a jukebox of U2 hits. They had lost their passion, it seems, even if it didn't show in concert. Bono talked about destroying the "myth of U2". This was key. It seemed that people placed U2 above humans - almost too holy. AB and the ZOO tour restored U2's humanity.
So even if R&H was a super success, I have a feeling that U2 were going to change anyway. First, it's in their nature - see the change from "War" to UF. Second, it was a new decade - time to start fresh. Third, the crashing of the Berlin wall played a role, as did the war in Iraq. Fourth, U2 were itching for a change from what they had done.
Keep in mind that while recording AB, Adam and Larry weren't all that key on making these big changes - they wanted more music like what U2 had done. So it's not as if the entire group was ready to just abandon their recent sound because of the critics. I think Larry and Adam wanted a more natural evolution, whereas Bono and Edge wanted a more abrupt change. They came together with "One" and the album seemed to flow from there.
Hence, I think the change would have happened regardless. I think the better question is whether we would have had ZOO TV if R&H or the Love Town tour was viewed more positively (by both fans and critics). I think AB - or something close to AB - would have been created. But would U2 have been so willing to make a change in their appearance if they weren't so adamant on destroying the "myth of U2"?