We discovered a non-functional smoke detector that's hard-wired into the electrical system of the house we recently bought. I removed the smoke detector and am replacing it with a battery operated one. But there is one set of 4 small wires hanging out of the ceiling. They all run through one single wrap. The colors are black, red, green and yellow. The hole in the ceiling is small. I won't be able to cap each wire and push it back through. Not sure if they can all be capped together into one wire nut or what....
Those wire colors (black, red, green and yellow) are commonly found in the wiring for security systems; smoke detectors that hardwire into security systems typically run on 12VDC or 24VDC, unlike the 120VAC detectors that you generally find at home improvement stores. I'm guessing the wires are stranded instead of solid core (like mains wiring) and of a smaller gauge wire. Another clue is that there's a small hole in the ceiling; if it was a 120V detector, there should be a ceiling box, and the wires would be colored black and white, possibly with a third, red, insulated wire, and probably a bare copper wire for ground.
Since this smoke detector isn't working, the security system may have been disconnected or removed. Do you have other smoke detectors in the house, and do they work? What about other security system components like motion detectors, glassbreaks, the control panel and/or keyboard?
Use a multimeter to check that there's no voltage between any of the wires; the supply for the detector was probably on the red and black, but check all the combinations to be sure. I would seal the hole with some expanding foam to prevent air leaks into the ceiling, then wrap each exposed wire individually with electrical tape and just leave them behind the base plate for the new detector.
As someone who has had smoke detectors save his house twice please rethink your plan. The big advantage to the electric detectors are they are wired together, one goes off,they all go off. The cost of having the electrician come back is minimal to the added benifit. There is also the chance that modifications to the wiring may make them all inoperative.
If the old security system is gone, you should be able to just push the wires, uncapped, into the wall. If you want to be really smart about it, wrap them in electrical tape before you do and leave a tiny bit protruding out the hold and taped flat to the wall so you can retrieve them at a later date if you choose.