Quantification of heterogeneous multiscale permeability in geologic porous media is key for understanding and predicting flow and transport processes in the subsurface. Recent utilization of in situ imaging, specifically positron emission tomography (PET), enables the measurement of three-dimensional (3-D) time-lapse radiotracer solute transport in geologic media. However, accurate and computationally efficient characterization of the permeability distribution that controls the solute transport process remains challenging. Leveraging the relationship between local permeability variation and solute advection rates, an encoder-decoder based convolutional neural network (CNN) is implemented as a permeability inversion scheme using a single PET scan of a radiotracer pulse injection experiment as input. The CNN consists of Densely Connected Neural Networks that can accurately capture the 3-D spatial correlation between the permeability and the radiotracer solute arrival time difference maps in geologic cores. We first test the inversion accuracy using 500 synthetic test datasets. We then use a suite of experimental PET imaging datasets acquired on four different geologic cores. The network-inverted permeability maps from the geologic cores are used to parameterize forward numerical models that are directly compared with the experimental PET imaging datasets. The results indicate that a single trained network can generate robust, denoised 3-D permeability inversion maps in seconds. Numerical models parameterized with these permeability maps closely capture the experimental solute arrival time behavior. This approach presents an unprecedented improvement for efficiently characterizing multiscale permeability heterogeneity in complex geologic materials.