The floor joists of the first floor are sitting directly in breaks in the concrete/cinderblock walls, and there's large gaps all around the edges. I've done some looking around and it appears that these are a large source of air loss, so I need to insulate them. Can I just spray some expanding foam in that area, or is it better to get some rigid foam, cut it, and then just use foam to seal the edges where it meets the joists/wall? What about a vapor barrier?
Not sure if this is exactly what you're looking to do, but the idea should work in your situation I think.
Summary of steps:
Cut rigid foam to fit between joists.
Insert foam into gaps (against rim joist).
Caulk around the edge of the foam.
Fill any gaps in the rigid foam where pipes or vents pass through, with spray foam.