Most likely this is because of your software setup. Please share more details. Perhaps if you replace the Nitrokey with a software key, you get similar results?
A test with a newly generated key of the same length (4096 RSA) and encryption + decryption on the computer (without Nitrokey) resulted in comparable (asymetric) results.
Even a test on a Macbook Pro on El Capitan showed almost the same results.
I cannot imagine that it is explainable by the enc/decryption algorithm but a software issue.
When I have (much) more time I will profile gpg with the perf tool of Linux to find the “bottleneck” during encryption.
BTW: The decryption speed of 45 MB/s using an USB 2.0 crypto stick “smells” like cheating since USB 2.0 can only transfer 480 MBits/s (about 60 MB/s raw speed but overhead reduces this).
If the Nitrokey were doing the complete decryption of a huge file the file would be transfered to the Nitrokey, decrypted and transfered back again completely which would allow only a max. decryption speed of 30 MB/s (ignoring the decryption overhead which is also significant).
The reason for measuing 45 MB/s is simple: Only the session key is decrypted by the Nitrokey, the file itself is decrypted by the local computer (not the Nitrokey).
: 45 MB/s