I don't deny that he may have been a spoiled, rich, brat and can be extremely selfish, but there are moments where he doesn't seem like a chaotic evil or just plain evil character, he remains neutral. There is no affect on him. Based on my character's interaction with him, he just seems extremely selfish and all he cares about is his freedom. If he needs to help someone to further his freedom, he'll do it. The feeling I get from him is, the more loyal you are with him, the more loyal he is with you. Especially if you side with him a lot, he'll change the way he talks to you. Of course this goes with any other companion.
...so chaotic evil? Evil in 5e is basically defined as selfishness to the point of willingness to hurt others if it benefits you. And chaotic is essentially caring about personal freedom over societal rules/responsibilities. CE characters can certainly be loyal to and even care about individuals - they'll just add those individuals to the list of people they'll hurt others/break laws for. An evil character will certainly perform acts that help others...as long as it benefits them more.