First let's take a quick look at the example systems' results to determine what the system does.
What we know:
$ input \rightarrow system \rightarrow output\! $
$ e^{2jt} \rightarrow system \rightarrow te^{-2jt}\! $
$ e^{-2jt} \rightarrow system \rightarrow te^{2jt}\! $
It looks like the system is performing the following operation:
$ x(t) \rightarrow system \rightarrow tx(-t)\! $
When the preceding system is applied to $ cos(2t) $, we the get result:
$ cos(2t) \rightarrow system \rightarrow tcos(-2t)\! $
Therefore, the system's response will be $ t*cos(-2t) $.
