I didn't say they mention it, I said they hint at it. For example, when Sherem says to Jacob that he sought much opportunity to speak with him. A small group fifty people living together doesn't require much opportunity when they live and work together. Also, it says "there came a man among the people of Nephi, whose name was Sherem." If he came among them, he was an outsider. If he was an outsider, he had a people he belonged to who were outsiders. It also mentions he had a perfect knowledge of their language. Why wouldn't he if it was his own language.
Many such hints.