A quantum operation $\mathcal{E}$ is a map from the set of density operators for the input space $Q_1$ to the set of density operators for the output space $Q_2$, that satisfies these axioms:

Axiom 1	$\text{tr}\{\mathcal{E}(\rho)\}$, with value between 0 and 1 inclusive, is the probability that the process represented by $\mathcal{E}$ occurs, when $\rho$ is the initial state. 💡 Recall Eq. (1) in the discussion of density operator.
Axiom 2	$\mathcal{E}$ is a convex-linear map on the set of density operators, i.e., for probabilities $\{p_i\}$, $\mathcal{E}\left(\sum_ip_i\rho_i\right) = \sum_ip_i\mathcal{E}(\rho_i).$
Axiom 3	$\mathcal{E}$ is a completely positive map, i.e., if $\mathcal{E}$ maps density operators of system $Q_1$ to density operators of system $Q_2$, then $\mathcal{E}(\mathbf{A})$ must be positive for any positive operator $\mathbf{A}$. Furthermore, if we introduce an extra system $R$ of arbitrary dimensionality, then $(\mathbf{I}\otimes\mathcal{E})(\mathbf{A})$, where $\mathbf{I}$ denotes the identity map on system $R$, is positive for any positive operator $\mathbf{A}$ on the combined system $RQ_1$.

This example is meant to show that the transpose operation is not a completely positive map and hence not a quantum operation.

We first observe that $\rho^\top$ is positive if $\rho$ is positive because the quadratic form

$\langle\psi|\rho^\top|\psi\rangle = \text{tr}(\langle\psi|\rho^\top|\psi\rangle) = \text{tr}\{(\langle\psi|\rho^\top|\psi\rangle)^\top\} = \text{tr}(\langle\psi|\rho|\psi\rangle) = \langle\psi|\rho|\psi\rangle.$

However, the transpose operation is not completely positive due to the following.

Suppose system $AB$ contains entangled subsystems $A$ and $B$, and has state $|\Psi\rangle = \sum_i|i\rangle_A\otimes|i\rangle_B$, ignoring the normalisation constant.

The density matrix of $AB$ is

$|\Psi\rangle\langle\Psi|=(\sum_i|i\rangle_A\otimes|i\rangle_B)(\sum_j\langle j|_A\otimes\langle j|_B) = \color{red}{\sum_{i,j}|i\rangle_A\langle j|_A \otimes |i\rangle_B\langle j|_B}.$

Above, we applied the identity $(\mathbf{A}\otimes\mathbf{B})(\mathbf{C}\otimes\mathbf{D}) = \mathbf{AC}\otimes\mathbf{BD}$.

Now consider the composite map $\mathbf{I}\otimes\mathbf{T}$, where $\mathbf{T}$ is the transpose operation, which transforms $|i\rangle\langle j|$ to $|j\rangle\langle i|$.

Applying the composite map $\mathbf{I}\otimes\mathbf{T}$ to $|\Psi\rangle\langle\Psi|$, we get

$\sum_{i,j}|i\rangle_A\langle j|_A \otimes |j\rangle_B\langle i|_B.$

Applying the composite map $\mathbf{I}\otimes\mathbf{T}$ to the above, we get back

$\color{red}{\sum_{i,j}|i\rangle_A\langle j|_A \otimes |i\rangle_B\langle j|_B}.$

Therefore, if we represent the map $\mathbf{I}\otimes\mathbf{T}$ with a square matrix, the matrix is involutory (square = identity), and it is trivial to show that the eigenvalues of an involutory matrix are $\pm1$ (see below), implying the composite map is not positive.

Note: For involutory $\mathbf{M}$, $\mathbf{M}\vec{x} = \lambda\vec{x} \implies \mathbf{I}\vec{x} = \lambda\mathbf{M}\vec{x} \implies \det(\lambda\mathbf{M} - \mathbf{I}) = 0$ $\implies \det(\lambda^2\mathbf{I} - \mathbf{I}) = 0 \implies \lambda^2-1 = 0 \implies \lambda = \pm1$.

In other words, the transpose operation is not a completely positive map, and hence not a quantum operation.

The map $\mathcal{E}$ satisfies Axioms 1-3 if and only if

$\mathcal{E}(\rho) = \sum_i\mathbf{E}_i \rho \mathbf{E}_i^\dagger,$

(1)

for some set of operators $\{\mathbf{E}_i\}$ that map the input Hilbert space to the output Hilbert space, and $\sum_i\mathbf{E}_i^\dagger\mathbf{E}_i\leq\mathbf{I}$.

Consider the Kraus operators or operation elements: $\mathbf{E}_1=\frac{1}{\sqrt{2}}\mathbf{I}$ and $\mathbf{E}_2=\frac{1}{\sqrt{2}}\begin{bmatrix}0&1\\1&0\end{bmatrix}$, where $\mathbf{E}_1^\dagger\mathbf{E}_1+\mathbf{E}_2^\dagger\mathbf{E}_2=\mathbf{I}$.

An equivalent operator-sum representation can be provided by $\mathbf{F}_1=\frac{1}{\sqrt{2}}(\mathbf{E}_1+\mathbf{E}_2)$ and $\mathbf{F}_2=\frac{1}{\sqrt{2}}(\mathbf{E}_1-\mathbf{E}_2)$, because

$\begin{split} \mathbf{F}_1\rho\mathbf{F}_1^\dagger + \mathbf{F}_2\rho\mathbf{F}_2^\dagger &= \frac{1}{2}(\mathbf{E}_1\rho\mathbf{E}_1^\dagger + \mathbf{E}_1\rho\mathbf{E}_2^\dagger + \mathbf{E}_2\rho\mathbf{E}_1^\dagger + \mathbf{E}_2\rho\mathbf{E}_2^\dagger) \\&+ \frac{1}{2}(\mathbf{E}_1\rho\mathbf{E}_1^\dagger - \mathbf{E}_1\rho\mathbf{E}_2^\dagger - \mathbf{E}_2\rho\mathbf{E}_1^\dagger + \mathbf{E}_2\rho\mathbf{E}_2^\dagger) = \mathbf{E}_1\rho\mathbf{E}_1^\dagger + \mathbf{E}_2\rho\mathbf{E}_2^\dagger. \end{split}$

Any quantum operation $\mathcal{E}$ on a system of $d$-dimensional Hilbert space can be generated by an operator-sum representation containing at most $d^2$ elements:

$\mathcal{E}(\rho) = \sum_{i=1}^M\mathbf{E}_i\rho\mathbf{E}_i^\dagger,$

where $1\leq M\leq d^2$.