I self host my mail server using mailcow and have an instruction to add a TLSA DNS record.
I have done this in the UI, but for some reason it posts an error about the key being returned as noin-hex.
I have ensured there are no spaces or paranthesis when adding the key in the UI.
That appears to be a bug in the verification tool you’re using. The wire format of a TLSA record is raw bytes for the certificate data (full certificate or hash), so “non-hex characters” in that part is a nonsensical error message. I assume it tries to interpret these bytes as a string of hex characters, but since those are raw bytes, you’re bound to have “characters” that aren’t 0-9a-f. The parentheses are also not in the record delivered by deSEC. They’re added in the presentation by the dmarcguard check, so they are not the cause of the error either. Use a different tool for verifying your record.
So the tool is at fault? Interesting. Ill check around to see if I can find a working tool.
The server mailer software is also not able to “see” this record, so maybe they have a bug too? Ill check with them.
Tx