Communicating between latent spaces with limited semantic correspondence