Yes, for inbound email you don't need anything set in the relay field (it's only for mails going outside your mail domain).
If your MX record points to the public IP address of your Astaro, you also don't need transparent mode.
Also, check that the Allow upstream/relay hosts only checkbox is OFF on the Relay tab.
If these don't help, you could perhaps post a few related lines (a few lines before and after the mail is rejected) from the smtp.log.
I'm 99% sure it is not some exotic bug, just a simple wrong setting. It's just a lot harder to find without looking at your config...