How to associate with devotees and how to disassociate from non-devotees is explained in the following quote by Srila Prabhupada.
A devotee shows compassion to those poor souls who are wanting in spiritual knowledge by enlightening them in order to elevate them to Krsna consciousness. That is one of the duties of a devotee. He should also make friendship with persons who are on an equal level with himself or who have the same understanding that he does. For a devotee, there is no point in making friendships with ordinary persons; he should make friendship with other devotees so that by discussing among themselves, they may elevate one another on the path of spiritual understanding. This is called ista-gosthi.
In Bhagavad-gita there is reference to bodhayantah parasparam, "discussing among themselves." Generally pure devotees utilize their valuable time in chanting and discussing various activities of Lord Krsna or Lord Caitanya amongst themselves. There are innumerable books, such as the puranas, Mahabharata, Bhagavatam, Bhagavad-gita and Upanisads, which contain countless subjects for discussion among two devotees or more. Friendship should be cemented between persons with mutual interests and understanding. Such persons are said to be sva jati, "of the same caste." The devotee should avoid a person whose character is not fixed in the standard understanding; even though he may be a Vaisnava, or a devotee of Krsna, if his character is not correctly representative, then he should be avoided. One should steadily control the senses and the mind and strictly follow the rules and regulations, and he should make friendship with persons of the same standard.