LR is working as designed. When originals are available they are used, when they are not, then any available Smart Previews are used. Smart Previews are smaller versions of photos so are not supposed to be used for final Detail work, just things that are applicable to wider areas like toning, spotting, cropping, etc. But final exports or prints should be done with the detail photos visible.
The issue for you is that the originals are on a very slow drive connection that probably doesn't allow any caching so every byte has to be read every time. Hopefully your camera raw cache is on an internal drive.
I'd suggest you get a 1 or more TB external USB/eSATA-attached drive that you can plug into the computer, directly. I use this for all my LR work and it isn't too bad.