No, you're not crazy. The key is to consider how many of the blade's teeth are in contact with the workpiece at a given moment. The more teeth, the more resistance because you're essentially forcing each tooth into the material with your own muscle. Pulling will usually result in less resistance for the blade because it's cutting an effectively thinner depth at any point.
That said, it's not always best to do a pull or push cut. Some reasons to do plunge cuts:
- You're trying to precisely orient the blade to a mark or other point in the center of the workpiece
- You're attempting to minimize tearout on one face or another
- You're making an incomplete cut
Also, there's possibly more risk of "blade walking" with a pull cut. Your arm is less prepared to resist motion if the blade does grab when you're already pulling toward yourself. I've witnessed some pretty hairy situations with very large radial arm saws walking into heavy boards. A 16" industrial blade doesn't like to stop suddenly with all its rotational inertia at play.
Push cuts are often inappropriate for trim work because they result in the blade tearing out upward, though what's usually the finished face of the workpiece.