Accurate, real-time source localization of gravitational wave (GW) events is important for electromagnetic follow-up observations of short gamma ray bursts, which follow binary neutron star mergers. Current parameter estimation methods like Markov Chain Monte Carlo (MCMC) and Nested sampling used by the LIGO-Virgo-Kagra collaboration, however, can take anywhere between a few hours to several days to infer the GW source parameter posteriors. Faster, approximately Bayesian methods like Bayestar (Singer and Price, 2016) can localize GWs in around 1 sec, but is less accurate. In this work, we introduce a deep learning model using the Normalizing Flows technique to estimate the sky direction posteriors of all compact binary coalescence (CBC) sources - binary black hole, binary neutron star, and neutron star - black hole mergers at orders of magnitude faster speed of a few milliseconds, at comparable accuracy to existing observed and published results. This is the only deep learning-based model, at the time of writing, that can achieve accurate source localization of all kinds of CBC sources at milli-second latency. We demonstrate the performance of this model on simulated two and three detector CBC signals injected in stationary, Gaussian noise and coloured by advanced LIGO power spectral density. We also discuss the application of this model for rapid estimation of other source parameter posteriors which are important for electromagnetic follow-up, like the masses and distance of the binary system.