In my experience, the only time the efficiency penalty comes into play is woodcutting, as that is the only time that your distance increases significantly with time. A long walk mining stone or gold is 2 tiles. That is a short walk for wood. The solution is to increase the number of SP/TC. My experience is after jihad, you don't notice the penalty if you place the new building anytime the forest line increases to 5. After cutting a long time into the forest, you will see
ttt
ttv
ttvSSS.SSS.SSS
ttvSSS.SSS.SSS
ttvSSS.SSS.SSS
ttv
ttt
where v is villager, t is trees, S is your SP/TC, and . is just a placeholder to denote the space.
And don't worry. If you have 10 choppers, I *think* you make up the wood in under 15 sec, and after that, its just bonus wood.
Keep your stick on the ice.