I interpret the question to be not about tread depth or riser height, but the remaining, load-bearing portion of the stringer (dimension Z), and the height and depth of the upper stringer heel (dimensions X and Y). Here are some general thoughts...
- The remaining thickness (perpendicular to the bottom edge) generally shouldn't be much less than it is below one of the standard notches.
- As the end portion doesn't carry as much bending force as the central area of the stringer, it could be somewhat thinner. See engineered floor and commercial steel roof trusses as an example of this--some hang solely on a top flange.
- If you're using a steel hanger of some sort, it all becomes fairly moot. A flexible-angle stringer hanger could solve all your woes.
- Another strategy is to line the stringers with a secondary member to the inside. In fact, my family did that for every stair set we built, for added stiffness.
- Dimension Y doesn't really matter as long as there's sufficient material to fasten to from the end, even if only down low. It'll become prone to splitting on the wood grain, though, so consider piloting for screws.