Towards Fine-Grained and Multi-Granular Contrastive Language-Speech Pre-training