We know from this thread
that SE050 is not used in the Trussed-based firmware released as of April 2023.
I think the question is about physical security of the data on the chip itself. Only in the case we could demonstrate that the firmware uses certain features preventing certain physical attack(s), like using some kind of more protected element than the main CPU on the token itself to store sensitive data, we can disregard physical security properties of the chip.
In other words, suppose the main CPU on the token is compromised, does that mean that the data we have entrusted in the device are compromised, too?