Yoga was apparently not as much about physical exercise until it came to the West. It was a way of helping the body endure longer meditation sittings.
An oversimplification I'm sure since there are many traditions, but it's at least true of some.
If I understand correctly, it used to be considered a path to enlightenment / nirvana / liberation etc by itself, not just as a mere supplement. Way different from today's new age / feel good / get fit commercial yoga.