The problem I had in being in academia is you work your tail off you usually don't make more money. Add to it, if you're in a research institution that only promotes based on publications, hard and effective work in the inpatient unit, clinic, teaching doesn't add anything to your income.
In this world of people (including you and your children) graduating with crippling debt, the thought of being a physician and not being able to pay for my kids' tuitions was IMHO ridiculous.
Add to it the institutions I worked in IMHO were wasting money hand-over-fist. Made the aspect of working hard and not making more money that much more frustrating when you saw their inefficiency, you knew how to fix it, they wouldn't fix it, nor let you have a handle in fixing it.
As an example, I know of an institution that has several TMS machines that are just sitting there collecting dust. The psychiatrists in the department have no incentive to start a TMS clinic because it will not help their careers at all. It's a waste to the institution, and to the community. If it got in operation it could generate tremendous revenue, but the rules in the institution prevent anyone from wanting to get this thing off the ground into a functioning, profitable, and effective clinic for the community.
People see income in a black and white sense. "But you're not supposed to want money." Ahem, paying off your debt? Making enough money for the institution to be viable? These are things that you must do in the real world not only to survive, but also if an institution is profitable because it's doing what it's supposed to be doing then it is a sign it is a viable institution in and of itself. (Again if they're doing what they're supposed to be doing which is provide good care). Of course money isn't the only barometer, but this "money doesn't matter" attitude in academia and other institutions such as the government is limited and ultimately ignorant virtue-signaling.
As I've mentioned, I do recommend new graduates spend some time in academia, but I left ultimately because the last institution I was in was dysfunctional on several levels. Had I stayed in the prior academic institution I was in I might've never left.