"There is no conflict without hate.
Do you have hatred for those monsters that seek to harm you?
Do you have hatred for those worthless subordinates who do not live up to your expectations?
Do you have hatred for the cowardly commoner who is not willing to fight to the death?
Do you have hatred for the King who stands above you, handing down selfish commands?
Do you have hatred for that arrogant dragon knight who always looks down on you?
Do you have hatred for the one who is better than you and is beloved by the masses?
Or...
Do you have hatred for that monster who drove your father to his death?"

"I hate myself.
I hate myself for the inexperience that makes me a failure in battle!
I hate myself for the selfishness that will not allow me to accept others!
I hate myself for the arrogance that makes me look down on those weaker than I!
I hate myself for my suspicion-filled heart that distrusts the King!
I hate myself for the rebellious nature that prevents me from cooperating with others.
I hate myself for the jealous nature that makes me envious of those superior to me.
And..
I hate myself for the death of my father, brought by my own impotence!"

"This, then, is your answer?"

"Yes! I hate nothing more than myself!"

"You may take me in hand, warrior of strong heart. I shall bend to your will. Together we shall destroy all beings cleaved by my blade. And all that we slay shall be cast out from this world for all eternity."

These are the quotes from the scene where Albel takes the trial of the Crimson Scourge when he was 15 years old (year SD 763 (2849 A.D.), random fact). It's a pretty important scene if you ask me, it explains a lot! Judging by what he answers, there's more below the surface of the arrogant, selfish swordsman. He appears arrogant and over-confident. Deep down, he hates himself for more then just one reason.

One of the obvious reasons would be the loss of his father, Glou Nox. Glou was the captain of the Dragon Brigade. His son of all beings failed the ceremony to take place among the dragon knights (which has a low percent of success in the first place), thus not making up to everyone's expectations (he is the son of a dragon brigade captain after all!). Albel was supposed to pay his failure with his life, but his father died instead. His father died because he failed. Albel probably held high respect for his father. He wanted his father to be proud of him. However, probably being the arrogant and bratty 15-year-old kid, he failed, losing his father instead. I think Albel grew to be afraid to attach himself to others since then, afraid to go all through the pain and misery of losing someone again. The way he acts suggests that he doesn't want to attach to others, or others to approach him. He more or less has build a wall around himself and he doesn't want to let anyone in. I don't think he just despises others, he just despises weakness. He despises others for being weak, but most of all; he despises himself for being weak. The death of his father made him grow in the person he is during the events of the game. Of course, we don't know how he was as a child, but probably not as he is now.
Many people view him as arrogant and wicked. People fear him in fact. Though, he threats and insults people a lot, and he is merciless in battle, I don't exactly see him killing innocent people. He doesn't kill for pleasure. Sure, he loves to fight, but he doesn't kill innocent women and children. Some people say he's evil, he's not...he's just misunderstood to put it in clishés. Most people don't see behind his arrogant and wicked nature. But Woltar did, and Fayt and Arzei probably did too.

Through the game you can see him open-up to Fayt a little. It begins when Fayt has defeated Albel. And later, the Peterny scene. Albel wouldn't ask someone wether he hated him or not just for fun. For some reason, he wanted to know if Fayt hated him. If you reply with "yes", you can notice a little change in Albel's voice. He's hurt. This again proves that he has build a wall around him, but he has a softer side. Through the game, he opens up little by little. He starts to respect and trust Fayt.