The flue for my boiler has a large hole around it, presumably from when it was put in. I'd like to fill it in because it's unsightly, but in not sure of the best way to do it, as it's a large hole and near a boiler.
Would expanding foam and a layer of plaster/filler work, or should I be using mortar?
The house was built very early 1900 so there is little chance of it having a cavity wall.
If a cover would suffice, the term for what to look for is a "stove pipe escutcheon" plate or flange. If you want to block air flow as well, use something non-flammable like rock wool (the newer versions that don't have asbestos). Do NOT use expanding foam...
There are several spray foams in the marketplace designed specifically for this situation. I have used Great Stuff Fireblock and Pur Fill Fireblock in many commercial applications and have passed all Fire Marshal inspections. They should work well in a residential application as well. Both of these products are rated at a 240 degree flashpoint and can also be trimmed flush after drying.
Here's a neat video that shows how well these products work.
It might not have to be fire resistant. Depending on the type of venting and the approval of the boiler. It looks like a coaxial venting system, meaning that it exhaust is sealed in the middle of the intake which is evident because of the visible screws. If the pipe is a soft plastic it's almost certainly zero clearance to combustibles but it's still safest to check the installation manual. If it is zero clearance you can fill the gap with whatever your heart desires, provided your heart doesn't desire something that dissolves pvc.
Here's an example of a coaxial venting system I have sitting in my garage for some reason
I'd stuff it with some batt insulation pink or green and cut some wire mesh to give morter a binding surface. I find non-shrink grout tools well and obviously doesn't really shrink or crack but it's hard as rock if you ever need to remove it. Dryish mixed morter works well.