Updated Answer
After reading my discussion feedback, I suppose my first instinct on how to approach this problem was incorrect. It looks like Tyler pointed me in the right direction. To do this correctly you have to expand it first. Thanks for the help.
$ e^{2jt} = cos(t) + 2j*sin(t) \rightarrow system \rightarrow t*e^{-2jt} = t*cos(t) - 2jt*sin(t)\! $
$ cos(2t) \rightarrow \frac{1}{2}(te^{2jt} \; + \; te^{-2jt}) \rightarrow system \rightarrow \frac{1}{2}t(e^{2jt} \; + \; e^{-2jt}) \rightarrow t*cost(2t)\! $
Original Answer
First let's take a quick look at the example systems' results to determine what the system does.
What we know:
$ input \rightarrow system \rightarrow output\! $
$ e^{2jt} \rightarrow system \rightarrow te^{-2jt}\! $
$ e^{-2jt} \rightarrow system \rightarrow te^{2jt}\! $
It looks like the system is performing the following operation:
$ x(t) \rightarrow system \rightarrow tx(-t)\! $
When the preceding system is applied to $ cos(2t) $, we the get result:
$ cos(2t) \rightarrow system \rightarrow tcos(-2t)\! $
Therefore, the system's response will be $ t*cos(-2t) $.
