I had my fireplace checked and was told that the flue is too small for the fireplace (historic home, botched renovation by prior owner). As a result, a normal fire sends smoke billowing into the room. I'd still like to use the fireplace for appearance rather than heat. Is there anything I can burn that will minimize the smoke? I know I could get an electric insert but I'm interested in other options first.
You can purchase fireplace inserts or individual fake logs used in those inserts that accept gel fuel cartridges. The gel fuel burns with very little smoke and may be a good alternative for your situation. Most major hardware stores will carry them or be able to order them for you.
We have the same problem here and decided to use lots of colorful candles instead as @BrownRedHawk suggested in their answer.
Short answer - not really. Electric insert is the only way to go if you don't want to mess with your chimney.
A wood burning insert (or gas, or pellet) will certainly keep everything contained, but requires a functional chimney with proper venting.
Most inserts require the chimney to be lined with a 6" diameter steel liner, although some smaller inserts will support 5" liner if you have a tall chimney (tall chimney = better draft).
If you have a small chimney opening with a clay liner, there may not be room for a steel liner inside it (our 1924 colonial didn't). The existing clay liner may need to be broken out.
Gas inserts (natural or propane) produce flue gases that need to go up the chimney too. These flue gases are corrosive, so require PVC vents to the outside (4" should suffice as noted by Steven).
You could install one of those fire simulators what has colored lights shining on a piece of translucent fabric that is fluttered around via a small electric fan. No smoke, no heat and low energy consumption. I've seen some of these that are amazingly convincing even fairly close up.
You can find these also referred to as a "flameless fireplace".
A natural gas/propane insert does not produce any smoke. It does produce CO2 and CO, which are vented up the chimney. You don't specify how small your chimney is, but they don't require a huge one. If I recall, my unit has a 4" duct.
If the chimney is just a no-go then you could still install a direct-vent unit and vent out the exterior wall instead of the chimney.