You can buy horses in stables outside all major cities.
To steal a horse, just mount any horse you find. Stormcloak camps often have horses. Just start galloping away to escape!
Riften stables—you can borrow a horse by winning a brawl with Hofgrir, or by erasing Shadr’s debt with Sapphire. Katla’s Farm in Solitude. If you help Katla harvest her crops, she’ll allow you to borrow a horse.
You can also make your horse rear up on its back legs.
