Denied Permanently banned members rep not to count.

The reason why he negged/repped you stays, the fact he is banned doesn't mean his cautious or compliments should be taken away from your reputation page. It's also an informative tool for people to check.