Could hacking a server get you banned on steam?

No. not if you interfered with steam or manually emailed them saying " I hacked someones server on your network" << that would be very very dumb.