Only white asbestos (also known as Chrysotile) --- and the great number of people who have been exposed to large amounts of it and lived long healthy lives certainly makes one wonder; AFAIK it's only the other types, particularly blue asbestos, which are unanimously carcinogenic and the industry stopped using them a long time ago, but the effects from that (and contamination of white asbestos with the other types) have continued and led the majority of fears.
In the case of Mesothelioma I believe there's a fair amount of evidence that you need both : 1) Asbestos exposure and 2) to be a smoker. This might explain why some people exposed to asbestos remain healthy.
This article is worth reading: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3581056/