The unbelievers came across the companions in the Café 18:21-22
18:21 And so We made them (the unbelievers) stumble upon them, so that they might know that the promise of Allah is true and that there is no doubt about the Hour. They argued among themselves over their affair, then (the unbelievers) said, 'Build a building over them (their remains). Their Lord knows best who they were. 'But those who prevailed over the matter said, 'We will build around them a Mosque.'
18:22 Some will say, 'They were three; their dog was the fourth.' Others, guessing at the Unseen, will say, 'They were five and their dog was the sixth.' And yet others, 'Seven; their dog was the eighth.' Say, 'My Lord knows best their number. Except for a few none know their number.' Therefore, do not dispute with them except in outward disputation, and do not ask any of them concerning them.