Yes, Rigid foam (XPS and ISO) boards are frequently used in exterior applications, and can be a great option for improving efficiency. There are a couple of issues with your proposed approach, though:
- Vapor barrier location. The vapor barrier should be on the warm side (probably inside, unless you live in a very hot & humid climate). Ideally, this is immediately behind your interior drywall. You also do not usually need an extra vapor barrier layer adjacent to rigid foam sheets, they are typically highly resistant to moisture.
- Attachment. You cannot rely on glue to support foam and siding panels. These need to be mechanically fastened.
On a wood frame house, you'd want to use this order:
- interior drywall
- possibly a vapor barrier (e.g. poly sheeting), though the wisdom of this is debatable (see the comment from iLikeDirt below)
- wall studs (optionally with batt or cellulose insulation in between)
- exterior sheathing
- rigid foam
- wooden strapping for attaching siding, screwed to studs through foam. This provides support for the siding and also allows air circulation behind the siding, so any moisture that gets behind the siding can dry out.
- siding panels, nailed to strapping
Here's a visual from Fine Homebuilding, which has a couple of good articles on this:
Note that adding foam on the exterior of your house complicates some things. You will likely need to redo the trim on your windows and doors, which will become more in-set. These inset areas will need to be flashed properly or you will have leaks, which have more potential to do damage when you've got multi-layered, thick walls that could trap moisture. If done well, you can get very good insulation. But make sure to work with someone who knows what they're doing!