Had the same issue in my app. Bottleneck is cfdirectory which is awful slow. Switched to linux system command (was possible in my case) and performance was much better.
Just my 2ct.
I have similar experience which getting timeout at asset upload page when trying to select / search in folder select list. I believe when there is too many folders it spent time to calculate the permission.
The current script filter out permitted folders then return with given maxrows.