The Drs. Eide have a short post about mental math ability. Young children memorize numbers and the simplest arithmetic facts in the prefrontal region of the brain. But more complex math is done as the brain matures and grows in the parietal cortex.
The ability to have 'number sense' is related to the amount of white matter in the left corona radiata, which can be affected by premature birth, birth stress, dyslexia, and some other conditions at birth; these kids tend to have difficulty with math, and this new research helps explain why.