1 Chronicles 12:22 KJV
For at that time day by day there came to David to help him, until it was a great host, like the host of God.
Though King David did nothing to injure Saul, toward the end of Saul's life and then shortly after, the people of Israel began coming one after the other to follow him.
It was a long journey and a painful one no doubt. But David's faith in God and refusal to take the kingdom for himself, but to wait upon the Lord to give it to him proved to be the right move.
David did not refuse to take these men when they came to him, but I notice that he refused to allow them to do any damage to Saul. Though the accounts happened a little earlier in the history of David and Saul, twice David was in a position to kill Saul, his men wanted him to do it, but he refused to let it be done.
We never need to attack another person, even if they have gotten off course. We only need to follow the Lord for ourselves. He will bring to us those He chooses.