Fights won't be long exactly, but they'll be long enough that you'll definitely be taking damage. You won't be taking out many enemies in just 1-2 swings, so you'll need as much armor as possible to protect yourself. The first is that you need all of the protection you can get. There are two reasons you need to use heavy armor for this build.
It just means you will always do less damage, so you need to account for that with other things I talk about in this article.
Having the Battlecry racial power is nice if you get into a jam. Personally, my favorite race is Nord - which surprisingly can still work. In fact, these races actually do more damage unarmed than they do with most starting weapons. Makes sense, right? They have freaking claws! Humans (i.e. Khajit have a starting unarmed damage rating of 22, while Argonians are at 20. I just recommend them for one reason - they have racial bonuses to unarmed damage.
I'll start by saying you don't have to use these two races. Skyrim Unarmed Build Race: Khajit or Argonian Unless you turn the difficulty way down, there are a few things you pretty much have to do to make this build viable without mods. The game isn't really catered towards unarmed builds, though.