Are there any options for connecting into main stack for venting branch below waterline? Could I go up for a few feet with a with a wye join and then come back down? The reason for trying to come up with a solution to this is that I dont have access to the pipe above the waterline and the toilet connection and I'm trying to avoid running another pipe on the side of the house just to get venting for the basement bathroom.
The short answer is: No.
However, there are some other alternative solutions to avoid running another pipe outside the house you should consider:
As N R stated, you can can use what are called 'air admittance valves' or 'Studor Vents' which are essentially just an air 'check valve' that allows the air to go into a pipe and is suppose to stop it from going back out of the pipe (so you don't smell swear gas). The problem with these is that they don't always close unless there is a significant back flow of air, which is why they aren't always legal to use.
Alternatively, it seems unlikely to me that you would have no other fixtures on the same floor that are already vented. Consider running horizontally. If your horizontal access to a vent would require you to go below run the vent piping below the wet line to you can use what is called a "Chicago Vent"