Regarding question 3:
There are many ways to solve this problem. Some answers are here and here. You can also create a system of equations by writing down the matrix multiplication symbolically.
$$
X \cdot A = B
$$
$X$ is the transformation matrix you are looking for, $A$ your matrix before the transformation, and $B$ the matrix after the transformation. So you can write:
$$
\begin{bmatrix}
x_{11}&x_{12}&x_{13}\\
x_{21}&x_{22}&x_{23}\\
x_{31}&x_{32}&x_{33}
\end{bmatrix}
\cdot
\begin{bmatrix}
a_{11}&a_{12}&a_{13}\\
a_{21}&a_{22}&a_{23}\\
a_{31}&a_{32}&a_{33}
\end{bmatrix}
=\begin{bmatrix}
b_{11}&b_{12}&b_{13}\\
b_{21}&b_{22}&b_{23}\\
b_{31}&b_{32}&b_{33}
\end{bmatrix}
$$
with the individual matrix products being:
$$
\begin{matrix}
b_{11} =&x_{11}\cdot a_{11} + x_{12}\cdot a_{21} + x_{13}\cdot a_{31}\\
b_{12} =&x_{11}\cdot a_{12} + x_{12}\cdot a_{22} + x_{13}\cdot a_{32}\\
b_{13} =&x_{11}\cdot a_{13} + x_{12}\cdot a_{23} + x_{13}\cdot a_{33}\\\\
b_{21} =&x_{21}\cdot a_{11} + x_{22}\cdot a_{21} + x_{23}\cdot a_{31}\\
b_{22} =&x_{21}\cdot a_{12} + x_{22}\cdot a_{22} + x_{23}\cdot a_{32}\\
b_{23} =&x_{21}\cdot a_{13} + x_{22}\cdot a_{23} + x_{23}\cdot a_{33}\\\\
b_{31} =&x_{31}\cdot a_{11} + x_{32}\cdot a_{21} + x_{33}\cdot a_{31}\\
b_{32} =&x_{31}\cdot a_{12} + x_{32}\cdot a_{22} + x_{33}\cdot a_{32}\\
b_{33} =&x_{31}\cdot a_{13} + x_{32}\cdot a_{23} + x_{33}\cdot a_{33}
\end{matrix}
$$
or in matrix form:
$$
\begin{matrix}
\begin{bmatrix}
a_{11}&a_{21}&a_{31}\\
a_{12}&a_{22}&a_{32}\\
a_{13}&a_{23}&a_{33}
\end{bmatrix}
\cdot
\begin{bmatrix}
x_{11}\\
x_{12}\\
x_{13}
\end{bmatrix}
=\begin{bmatrix}
b_{11}\\
b_{12}\\
b_{13}
\end{bmatrix}\\\\
\begin{bmatrix}
a_{11}&a_{21}&a_{31}\\
a_{12}&a_{22}&a_{32}\\
a_{13}&a_{23}&a_{33}
\end{bmatrix}
\cdot
\begin{bmatrix}
x_{21}\\
x_{22}\\
x_{23}
\end{bmatrix}
=\begin{bmatrix}
b_{21}\\
b_{22}\\
b_{23}
\end{bmatrix}\\\\
\begin{bmatrix}
a_{11}&a_{21}&a_{31}\\
a_{12}&a_{22}&a_{32}\\
a_{13}&a_{23}&a_{33}
\end{bmatrix}
\cdot
\begin{bmatrix}
x_{31}\\
x_{32}\\
x_{33}
\end{bmatrix}
=\begin{bmatrix}
b_{31}\\
b_{32}\\
b_{33}
\end{bmatrix}\\
\end{matrix}
$$
Watch out: the matrix here is transposed -> $A^T$
Now you have 9 equations for 9 unknowns that you can solve with Gaussian elimination, LU decomposition, or any other linear solver you are familiar with.
You might say that those are just 3x3 matrices and yours are 3x5. Adding those two columns yourself should be easy, but you actually don't need them to determine the matrix. It would give you an overdetermined system of equations. Just pick 3 columns instead (but the same from both matrices).
Note that if you combine the 3 matrix-vector equations into a single matrix-matrix equation, you basically end up with something similar to this solution. All you need to do is to multiply by the inverse of $A^T$ and then transpose everything. Then you have the same equation.
I wrote a short python script to check if everything is correct that I wrote:
import numpy as np
a = np.array([[2, 5, 1], [1, 3, 1], [1, 1, 1], [3, 1, 1], [3, 3, 1]]).transpose()
b = np.array([[7, 0, 1], [9, 4, 1], [9, 8, 1], [5, 8, 1], [5, 4, 1]]).transpose()
# pick first 3 vectors
a_red = a[:, :3]
b_red = b[:, :3]
# solve for each row vector of x
x_1 = np.linalg.solve(a_red.transpose(), b_red.transpose()[:, 0])
x_2 = np.linalg.solve(a_red.transpose(), b_red.transpose()[:, 1])
x_3 = np.linalg.solve(a_red.transpose(), b_red.transpose()[:, 2])
# combine vectors to the transformation matrix we are looking for
x = np.array([x_1, x_2, x_3])
print(x)
# check if x*a yields b
b_check = np.matmul(x, a)
print(b_check)
assert np.allclose(b, b_check)
The assertion holds and the matrix looks like expected:
$$
\begin{bmatrix}
x_{11}&x_{12}&11\\
x_{21}&x_{22}&10\\
0&0&1
\end{bmatrix}
$$
Seems like you swapped the numbers in dx and dy cause I got dx=11 and dy=10. The first two rows of the last column represent the translation vector as described here.
The last row is (0, 0, 1) as it should be since we are not doing any transformation into the 3. dimension.
I left out the upper left 2x2 matrices for you to find out. As another hint, $x_{11}=x_{22}$ and $x_{12}=x_{21}$. I think you will manage to figure them out. If you can't, just ask for help or run the python script but I recommend trying it yourself.
Regarding question 2
The usual forward order is scale, rotate, and translate for reasons I won't further explain. But this is not a "rule" and you can not rely on that. It is just the approach that makes the most sense for us humans. However, looking at the graphics and the calculated matrix, they followed this order.
Regarding question 1
Now with the information, I gave you for question 2, it should be obvious. Just pick the point Γ and follow it if you first apply the scaling, then the rotation, and finally the translation. If you manage this, you should also be able to just guess the remaining numbers of the matrix with all the information you got, at least if you understood all the involved substeps and the basic transformation matrices (check this link again). You will also see that dx and dy are indeed swapped in your question.
