Could hacking a server get you banned on steam?

Probably. Its highly likely that hacking in to a server which in its self illegal will probably get you banned from steam.