So by self observing, comprehension in the moment comes naturally?
It can, for superficial things. But to really comprehend, one must meditate.
How do we finally make progress beyond this point to make much more profound change?
Meditation.
Your physical perceptions only see the most superficial and illusory appearances. To see the reality, one must take the consciousness out of the body, affections, and the mind. This is only possible by mastering meditation.
And before you feel like that is impossible, it is not.