That's the expensive way to do it. The cheaper and just-as-effective method is to put rigid foam boards (EPS or XPS foam) up against the walls, and seal their edges with little bits of spray foam, caulk, and tape (as appropriate). Seal them to the floor too. You can use the same method to insulate the rim joists. Just make sure everything's sealed up tight so no inside air can reach the walls and rim joists. This is much cheaper and more DIY-friendly, and the results are just as good. These foam boards are easier to finish too since they're flat and straight, so you can lay drywall right over them, fastened to the block walls underneath with Tapcon screws. Or you can build a stud wall and drywall over that.
For bonus points, use cementboard instead of drywall for the first two feet off the ground for flood safety. Drywall will dissolve and grow mold when soaked; cementboard will just happily sit there. If you build a stud wall, use pressure-treated lumber for the same reason.
If you choose to finish the basement, use the same approach and avoid moisture-sensitive materials in the bottom few feet just in case the basement floods. This means tile or plastic baseboards, trim, and flooring. No carpet, no hardwood, no MDF, no paper, gypsum, or wood products in general.