Unify Latent Spaces to Reuse Neural Components