How are you setting the circular saw when cutting? If you are clamping your straightedge so only the narrow part of the saw baseplate (1-1/2" or so away form the cut line) is on the wood, the saw is prone to wobble much moreso than when the wide part of the saw base is referenced (4-5" away from the cut.)
The other likely cause is that the surface profile of the face of the door (the raised panel section) is affecting the position of the blade as the baseplate rides along it - solve THAT by using a sawing straightedge that supports the saw above/on top of the door surface - something like a 1/2" piece of plywood 5-6 inches wide, with a straight board (or metal angle) screwed to the edge, using the saw to trim the edge of that plywood when the baseplate is run along the straightedge, and then you clamp the whole thing to the door with the cut edge aligned on your desired cut.
Something like this link, which I did not steal the idea from (it's common idea) but which illustrates it tolerably well (they use an absurdly wide straightedge, I think) and I don't feel like taking the time to make my own illustrations, nor do I wish to steal theirs. http://www.woodworkersjournal.com/making-straight-cuts-with-a-circular-saw/
Then again, an even wider base might be easier to clamp down without getting in the way of the saw - several of their pictures seem like they have it clamped in such a way that they could not have started or won't be able to end the cut without running into the clamps.
