I'm replacing a cat door with a smaller one, the old one was glued to the exterior paneling, so when I pulled it off, it pulled a bit of the panel as well.
You can certainly panel over the unattractive bits. Be sure to prime paint all surfaces (old and new), with special attention given to the edges and grooves. Use a high quality paintable exterior caulking to make sure the grooves don't channel water behind anything.
Having said that (and maybe I'm misinterpreting your order of operations), I'd start with an oversized piece of siding (bonus points if you line up the grooves; that looks like T1-11 siding). Cut the new door into the big piece. Attach the new door in the old opening in whatever way you need. Attach the big piece of plywood to the wall. Spray foam1 (low expand) or use batt insulation to insulate the gaps. Trim out the interior as needed. (If you're drywalling the interior you might need to add backing around the new cat door for the drywall to anchor to.)
1 When using spray foam, wear gloves and eye protection and have acetone handy for cleanup if needed. (A good source of acetone if you haven't bought a can deliberately is nail polish remover. However, some nail polish removers have acetone; others don't, so check beforehand.)