That is the desired behavior.
The HSM internally associates the private key and the related certificate. If you remove the key, you also remove the certificate.
You can of course have additional certificates (e.g. CA certificates), but those are not related to a private key.
This is somewhat different from the PKCS#15 model, where you have separate objects for the private key, the public key and the certificate. In the HSM the public key object is always retrieved from the certificate, there is no separate object internally.
That brings up the interesting question, how the public key is stored internally before the certificate is issued ? That is where public key attestation enters the scene: The public key of a key pair generated on the HSM is directly internally signed by the Device Authentication Key to form a CVC-Req as defined by TR-03110. That CVC-Req is treated as a certificate in the middleware, so that in the absence of the final certificate, the public key is retrieved from the CVC-Req.
The beauty of this approach is, that one could extract the CVC-Req and pass that to a certification authority to validate key attestation. That is what we do in the PKI-as-a-Service Portal to assure, that only keys generated on a HSM are certified.