I have a question about ants. When should I worry about them? They live my passion flower vine, caladiums and now my herb garden. I’ve been using diatomaceous earth around the edges of my pots and then tonight I find them going into the drainage holes of my herb planter. Help.