Modulation
x(t) is the information bearing signal and c(t) is the carrier signal. The modulated signal y(t) is given by
$ x(t)\! $ ----------> x --------> $ y(t)\! $ ^ | | $ c(t) = cos(\omega_c t+\theta_c)\! $
so $ y(t)=x(t)c(t) $. Then by taking the Fourier Transform of both sides of this equations yields $ y(\omega) $. Also, for convenience, we choose $ \theta_c=0 $.
$ F(y(t))=F(x(t)c(t))=F(x(t)cos(\omega_c t)) $
$ y(\omega)=\frac{1}{2\pi}F(x(t))*F(cos(\omega_c t)) $
$ y(\omega)=\frac{1}{2\pi}\chi(\omega)*(\pi(\delta(\omega+\omega_c)+\delta(\omega-\omega_c))) $
$ y(\omega)=\frac{1}{2}(\chi(\omega+\omega_c)+\chi(\omega-\omega_c)) $
If $ \omega_c>\omega_m $, then this gives two copies of our signal centered at $ -\omega_c $ and $ \omega_c $ and one half the magnitude of the original signal.
If $ \omega_c<\omega_m $, Then the copies overlap and there is no way to recover the original signal.
If the signal is not band limited, the copies will always overlap.
How to recover
The first step is to modulate y(t) with the same carrier signal.
$ y(t)\! $ ----------> x --------> $ x(t)cos^2(\omega_c t)\! $ ^ | | $ c(t) = cos(\omega_c t)\! $
This creates 3 copies of the original $ \chi(\omega) $. One centered at $ -2\omega_c $, one centered at $ 2\omega_c $, and one centered at the origin. The ones centered at $ -2\omega_c $ and $ 2\omega_c $ have a magnitude of 1/4 and the one at the origin has one of 1/2.
To recover x(t) from $ x(t)cos^2(\omega_c t) $, we run the signal through a low pass filter with a gain of 2 and cut off frequency $ \omega $. $ \omega $ must be chosen so that $ \omega_m<\omega<2\omega_c-\omega_m $.