Figured this out. I have connected NAV servers C-drive with my domain admin credentials and for some reason, instead of credentials used to log in to desktop, RTC uses those credentials when connecting. Not always, just most of the time.
A bit weird behaviour, but at least the problem got solved by adding both credentials to NAV server.