As is typical of your past questions, you are being hyperbolic and absurd.
ASRHEASHRHE suggests a ventilation rate of 40 to 70 CFM - if we took 70 CFM and 4 air changes per hour, your house would be a 10 foot cube, roughly. As far as I recall past mention of your purported house, it will be much, much, much larger than that.
As such, the ACH (which would still be at most 70 per that standard) will be far below 4. And the surface area of poorly insulated or uninsulated building to enclose that area will be large, and losing a metric-butt-ton® of heat as a result.
I've run the numbers in past answers. Are BTU demands only calculated using square feet?
Dropping them into my shop's spreadsheet, 70 CFM is 0.1736 ACH and amounts to 6381 BTU/hr at design temperature (-20°F). It will be a shade less if yours is only -10°F The walls (R33 SIPs) account for 6709 BTU/hr at design temperature. If you made the walls crappy R11 they'd burn 20128 BTU/Hr instead. I don't even need to get into the ceiling, floor, doors, or windows (all additional heat flows) to make fairly clear that even without a HRV, yes, insulation matters. Of course, if you added an HRV you could shave off 5104 btu/hr at my design temperature, and that's something I'm certainly considering doing.