I was afraid of doing this the first time; but after a plumber let me assist him I saw it was pretty basic. When you want to clear a drain pipe you want to consider the diameter of the pipe and the material of the pipe. The size of the pipe will determine what size bits you want to use. Bits come in a variety of sizes and shapes to handle different size pipes and types of blockage. Since you are overflowing at the toilet seal, I'm going to assume you want to clear your main line which is going to be pretty big.
The size of the bit will also determine the thickness of the cable you need to use. If you use a cable that's too thin for the bit, it will bind up and could become a twisted mess when it hits the blockage. For most jobs, the power drain cleaner you can rent at Home Depot is sufficient.
An important point is to go slow and use a low speed until you get to the blockage. The thing you want to avoid is to have the bit banging around with so much force that you damage the pipe, especially if you have a brittle material like clay. Start with a small bit to see how far the blockage is and how tough it is. Work the small bit through the blockage if you can, and then replace the bit with a larger one and repeat until you have the pipe cleared.
The 'tricky' part is working through the blockage without twisting the cable. You don't want to push the cable, just let it pull itself through the pipe and blockage. You also want to make sure to wear a pair of thick leather gloves. If the cable does twist up it will protect your hands and fingers from getting seriously pinched.
If you think there is some kind of damage or other serious issue, you can rent a camera to inspect the inside of the drain pipe. Most of them will also give you a distance measurement and record the video. This would be useful if you have to bring in a professional plumber.