I have a question: Can I use Rigid Foam Insulation instead of fiber insulation on outside walls from the inside of the house?
Yes. There are several types of rigid foam board. Polyisocyanurate (polyiso) has the best R-value at 6.5 (after aging). A 2 inch thick board will be at least R-12.9.
There are two main methods you can use from inside the room. With the first method, sometimes called "Cut-and-Cobble", you cut the panels to fit inside the joists and then use spray foam (e.g. Great Stuff) around each panel to create a tight air seal. If you have a 2x4 exterior wall, you could even get 3 inches of foam in there if you can find 3" panels or use a 2" and 1" panel together. I recently did exactly this myself in a bedroom that I gutted down to the studs.
Another method is to add the foam panels to the inside of the wall on top of the studs, tape the panels together with foil tape, and then put the drywall over the panels. This second method is preferred since it reduces thermal bridging from the cold studs into the wall and also provides a continuous air seal across the entire wall. If you do put the panels over the studs, you must fasten them to the studs and then use longer drywall fasteners to connect to the studs since the foam is not considered structural. The downsides to this methods are that all of your trim and receptacles must account for the additional wall thickness and you lose a bit of space in the room, depending on foam thickness.
I recently gutted out the exterior walls of my house and actually used BOTH fiberglass and foamboard.
First, I replaced the fiberglass insulation in the walls with R-13 (previously had R-11, which I repurposed for soundproofing between interior walls).
Then (as dslake mentioned in his option 2) I used 1.5" polyiso(cyanurate) OVER THE TOP of the interior studs (which built the wall out). Both sides are foil-faced, so I used the foil tape to seal the seams and cover up the cap nails used to hold the foam in place. Also, because I went scorched earth, I used acoustic sealant ("black death") on the tops, bottoms, and wall edges in order to fully seal the edges of the foam board to the wall.
Rather than using long drywall screws, I ripped down 3/4" plywood into furring strips and screwed them through the foam board into the studs with 3" screws (make sure you mark your stud locations and an impact drill is your friend here). This makes it much easier to mount your sheetrock. It also has the added benefit of providing a 3/4" "air gap" which increases the insulative quality of the wall (I believe the 1.5" has a rating of about R9.5, but with the added 3/4" of air it ups it to about R12.5).
Here's a link to the stuff I used - http://www.rmax.com/wall-thermasheath3.asp. Make sure you don't use the junky stuff from Home Depot, it's NOT polyiso and does NOT have the same R ratings.
Also, as dslake mentioned, the other painful part of this is having to "bump out" all your receptacles and switches to meet the new finished wall. I just used 2x6s and ripped them down to the proper depth, screwed them to the stud and then mounted new boxes.
I will also add the caveat that some people have disagreements on what this process will do to the "dew point" in your wall (i.e. where moisture will accumulate). You may want to do some research into this related to your particular climate. I looked at this as basically replacing the crappy 4-mil vapor barrier with something that actually helped insulate and make my home more comfortable.
Hope that helps, I had a friend who used this method when he built his house 20 years ago and he provided me with a lot of helpful hints. Let me know if you have any other questions!