I think what you’re missing is the matter of acceleration. At some point, a higher flap deflection, despite adding lift in principle, lengthens your takeoff roll again because the plane has to work so much harder to get to a speed where it’s capable of flying. You’d be surprised how much of a negative effect a little extra frontal area can have. That’s why short-field takeoffs call for neutral elevator on the runway all the way to vr (as opposed to soft-field takeoffs, where you want to unload the front gear - see below).
I remember when I trained on the C172s. They have a third flap setting at 40 degrees. My instructor only called them “drag flaps”. They were only useful for coming down, not going up. In a go-around, going back to the middle flap setting was the immediate second step after applying power. Otherwise, the plane might simply not climb.
Also, short field and soft field are two different animals in terms of t/o configuration. On a soft field, you want to lighten the load on the gear as quickly as possible and may be content with accelerating in ground effect - or even “hanging it on the prop” if the power-to-weight ratio is favorable enough. That’s very different from a short field, where you might not have that luxury and need to be at vx asap to climb over an obstacle.