When should I start paying more attention to artifacts?

It would start mattering at AR 45, because that's when you can unlock the highest level of artifact domains which would give you guaranteed drops of 5* artifacts.