if you follow the directions for the new flooring, the answer would be yes, to scrape it all up. here's the thing, when doing floating flooring, the floor underneath needs to be smooth, or you could feel imperfections through the floor as you walk. now that being said, as long as there are no big ridges in the glue and its all smooth, you should be able to get away without removing it. make sure that you remove all of the old tack strips and nails, getting the subfloor/concrete as smooth as you can before you install the new flooring.
I put in laminate flooring with attached underlayment back in april. There were a couple of spots on my subfloor that were not 100% even, joints in that were off by 1/4" and I was able to leave them, and now you cant even tell they are there.