Installing a ceiling fixture. With the other fixtures I installed the bracket just screwed into the box. This one the bracket width is too small to screw in, the box is too big. What kind of adapter do I need, or should I just replace the whole box with a smaller one?
It looks like it's clamped in one corner so hopefully the wire is loose.
Since it looks like it's been screwed in, I would buy a metal fixture box (should fit inside the existing hole), unscrew the existing clamp, knock out a hole and reattach the clamp to the new box. If you're lucky, it should all fit in the same hole and the fixture will cover the remaining parts.
Don't forget to ground your new metal box!
Try a 4" square to round device ring
The thing you want is a 4", square to round device (mud) ring. A flat one (say a Garvin 52C3-F, or equivalent) would be preferable for this situation, but one with a small depth (say 1/2") would be manageable still if you can't find a flat one, even though you should be able to find the flat version at your local electrical supply house.
Assuming the fixture canopy is large enough to cover the existing box, you are better off leaving it in.
There are numerous types of mounting crossbars that cover a range of box sizes. Your box looks like a 4" one, so the corner to corner distance is about 4.75"
If the canopy does not cover, you may need a ceiling rose (a decorative larger cover that goes above the canopy), or you may need to change the box. You do not want gaps at the corners that leave parts of the interior of an electrical box exposed.
Images and links are illustrative only, not an endorsement of goods or sources.
You can install a 4/0 or 3/0 mud/plaster ring, but the 4 sq box has to be recessed back into the wall at around 1/2 inch since the mud ring has a raise anywhere from 1/2 and up to 2" in 1/4 inch increments. I'm not sure if they make a flush raise, but it definitely would be a special order. If you can't make a drywall patch, I wouldn't try to just plaster since it will not adhere to metal. If you need to cover it you might try and use a medaillen.
They make them in various decorated styles and sizes.