Muwatta Malik, Book 56, Hadith 22 (USC-MSA)

Malik related to me that he had heard that Umm Salama, the wife of the Prophet, may Allah bless him and grant him peace, said, "Messenger of Allah! Shall we be destroyed while there are people who are salih among us?" The Messenger of Allah, may Allah bless him and grant him peace, said, "Yes, if there is much wickedness."

حَدَّثَنِي مَالِكٌ، أَنَّهُ بَلَغَهُ أَنَّ أُمَّ سَلَمَةَ، زَوْجَ النَّبِيِّ صلى الله عليه وسلم قَالَتْ يَا رَسُولَ اللَّهِ أَنَهْلِكُ وَفِينَا الصَّالِحُونَ فَقَالَ رَسُولُ اللَّهِ صلى الله عليه وسلم ‏ "‏ نَعَمْ إِذَا كَثُرَ الْخَبَثُ ‏" ‏ ‏.‏

sunnah.com

also referenced under: Book 56, Hadith 1835 (In-Book)