I would encourage to focus on the bigger implications to you beyond the 3 year training period. Even if your interest is primarily ADULT medicine, FM residency prepares you well for an out-patient practice. The residency is more focused on out-pt experiences than in most IM programs. If you are interested in an out-patient (clinic) existence after residency, FM may make sense for you. If in-patient is your cup of tea.. IM is well designed to prepare you for this. Others may disagree... but here's why:
Let me give you an example... The RRC requirements for FM have FM residents in clinic in their second and third years from 2 to 4 times per week. By the time you've graduated, you will have done about 350 clinics at a minimum though many programs build in much more than this with "clinic resident" rotations. By comparison, the RRC requirements for IM only require 1 clinic per week all 3 years... a minimum of 150 clinics or so with vacations.
This is one example of many that describe a much heavier in-patient slant for IM training. This makes sense in some ways... many IM residents choose in-pt based fellowships, and IM programs are generally depended on to "run" a hospital's in-patients service. However, when you look at the survey literature, ironically IM residents overwhelmingly enter into out-patient primary care practices. IM has struggled with this and tried to answer this with "primary care tracks" in training which do help but even those residents are way below the out-patient clinical volume of FM trainees.
Now add psych. The vast majority of psychiatrists are out-patient. Even "in-patient" psychiatrists are mostly on PSYCH in-patient units, not running traditional hospital consult services (the other way to be "in-patient" as a psychaitrist). To marry a medical specialty with psychiatry, again.. you should think about the mode of practice you want... in or out-patient. It makes the most sense to couple your psych and medical training in way that makes them complimentary whichever way you choose.
When it comes to training.. FM programs are generally family friendlier than IM programs. The schedule is more flexible, there are more out-patient rotations, and they are "family" medicine for a reason. That said, I have seen very capable IM and FM residents alike very capably protect their real lives in the face of a medical education very well. At the end of the day this will depend more on you than on your mode of training.
The issue of assessing the value and quality of individual IM/Psy vs FM/Psy programs is also a significant issue but beyond the scope of your post. Hope this helps... However you decide to approach this I think you're on to something already... You are only as good as a physician as your ability to preserve who you are inside the white coat. The fact that you're paying attention to striking this balance early is great.