SuNT's Blog | AI in Practical

Trong quá trình tìm hiểu về mạng NN, mình thấy khá là khó hiểu, đặc biệt với các bạn không mạnh về toán. Bài này, mình sẽ diễn giải cách thức làm việc của NN một cách trực quan, dễ hiểu cho các bạn thông qua một ví dụ cụ thể.

1. Nhắc lại lý thuyết

Giả sử ta có mạng NN như sau:

Quá trình training model bao gồm 2 phases:

1.1 Forward Path

Phase này tính toán (dự đoán) đầu ra $o_{1}, o_{2}$ , tính loss.

Giả sử activation là hàm sigmoid:

Ta sẽ tính lần lượt các đại lượng trung gian:

$i n_{h_{1}}$ : input của $h_{1}$
$i n_{h_{2}}$ : input của $h_{2}$
$o u t_{h_{1}}$ : output của $h_{1}$
$o u t_{h_{2}}$ : output của $h_{2}$
$i n_{o_{1}}$ : input của $o_{1}$
$i n_{o_{2}}$ : input của $o_{2}$
$o u t_{o_{1}}$ : output của $o_{1}$
$o u t_{o_{2}} : o u t p u t c ủ a$ o_2$$

Công thức tính của từng đại lượng như sau:

$i n_{h_{1}} = w_{1} * i_{1} + w_{2} * i_{2} + b_{1} * 1$

$i n_{h_{2}} = w_{3} * i_{1} + w_{4} * i_{2} + b_{1} * 1$

$o u t_{h_{1}} = s i g m o i d (i n_{h_{1}}) =$ $\frac{1}{1 + e^{- i n_{h_{1}}}}$

$o u t_{h_{2}} = s i g m o i d (i n_{h_{2}}) =$ $\frac{1}{1 + e^{- i n_{h_{2}}}}$

$i n_{o_{1}} = w_{5} * o u t_{h_{1}} + w_{6} * o u t_{h_{2}} + b_{2} * 1$

$i n_{o_{2}} = w_{7} * o u t_{h_{1}} + w_{8} * o u t_{h_{2}} + b_{2} * 1$

$o u t_{o_{1}} = s i g m o i d (i n_{o_{1}}) =$ $\frac{1}{1 + e^{- i n_{o_{1}}}}$

$o u t_{o_{2}} = s i g m o i d (i n_{o_{2}}) =$ $\frac{1}{1 + e^{- i n_{o_{2}}}}$

Tiếp theo là tính loss bằng cách so sánh đầu ra của mạng NN với các giá trị thực tế:

$t a r g e t_{o_{1}}$
$t a r g e t_{o_{2}}$ :

Công thức tính loss như sau:

$E_{t o t a l} = \sum_{i = 1}^{2} E_{o_{i}} = \sum_{i = 1}^{2} \frac{1}{2} (t a r g e t_{o_{i}} - o u t_{o_{i}})^{2} = E_{o_{1}} + E_{o_{2}}$

$E_{o_{1}} = \frac{1}{2} (t a r g e t_{o_{1}} - o u t_{o_{1}})^{2}$

$E_{o_{2}} = \frac{1}{2} (t a r g e t_{o_{2}} - o u t_{o_{2}})^{2}$

1.2 Backward Path

Mục đích của phase này là cập nhật trọng số $w$ sao cho tối thiểu hóa loss.

Ta sẽ sử dụng thuật toán tối ưu Stochastic Gradient Descent (SGD) để cập nhật $w$ .

Công thức cập nhật như sau:

θ = θ - η \nabla_{θ} f (θ)

với:

$\nabla_{θ} f (θ)$ là đạo hàm của Loss Function tại $θ$ (đạo hàm từng phần theo $\nabla$ ).
$η$ là một số > 0, gọi là learning rate.
$θ$ là tập hợp các vector các tham số của model cần tối ưu. Trong trường hợp này là các trọng số $w$ .

Đạo hàm từng phần của các $w$ tại output layer được tính theo quy tắc chain rule như sau:

$\frac{\partial E_{t o t a l}}{\partial w_{5}} = \frac{\partial E_{t o t a l}}{\partial o u t_{o_{1}}} * \frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial w_{5}}$

$\frac{\partial E_{t o t a l}}{\partial w_{6}} = \frac{\partial E_{t o t a l}}{\partial o u t_{o_{1}}} * \frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial w_{6}}$

$\frac{\partial E_{t o t a l}}{\partial w_{7}} = \frac{\partial E_{t o t a l}}{\partial o u t_{o_{2}}} * \frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}} * \frac{\partial i n_{o_{2}}}{\partial w_{7}}$ </p>

$\frac{\partial E_{t o t a l}}{\partial w_{8}} = \frac{\partial E_{t o t a l}}{\partial o u t_{o_{2}}} * \frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}} * \frac{\partial i n_{o_{2}}}{\partial w_{8}}$

Đạo hàm từng phần của các $w$ tại hidden layer được tính như sau:

$\frac{\partial E_{t o t a l}}{\partial w_{1}} = \frac{\partial E_{t o t a l}}{\partial o u t_{h_{1}}} * \frac{\partial o u t_{h_{1}}}{\partial i n_{h_{1}}} * \frac{\partial i n_{h_{1}}}{\partial w_{1}}$

$\frac{\partial E_{t o t a l}}{\partial w_{2}} = \frac{\partial E_{t o t a l}}{\partial o u t_{h_{1}}} * \frac{\partial o u t_{h_{1}}}{\partial i n_{h_{1}}} * \frac{\partial i n_{h_{1}}}{\partial w_{2}}$

$\frac{\partial E_{t o t a l}}{\partial w_{3}} = \frac{\partial E_{t o t a l}}{\partial o u t_{h_{2}}} * \frac{\partial o u t_{h_{2}}}{\partial i n_{h_{2}}} * \frac{\partial i n_{h_{2}}}{\partial w_{3}}$

$\frac{\partial E_{t o t a l}}{\partial w_{4}} = \frac{\partial E_{t o t a l}}{\partial o u t_{h_{2}}} * \frac{\partial o u t_{h_{2}}}{\partial i n_{h_{2}}} * \frac{\partial i n_{h_{2}}}{\partial w_{4}}$

Sau khi tính được đạo hàm từng phần của mỗi $w$ , ta áp dụng công thức phía trên để cập nhật $w$ .

2. Ví dụ áp dụng

Vẫn với kiến trúc mạng như trên, ta sẽ gán các giá trị khởi tạo cho các tham số như hình bên dưới:

Ok, bây giờ ta sẽ bắt đầu đi tính toán.

2.1 Fordward Path

Input của $h_{1}$ :

$i n_{h_{1}} = w_{1} * i_{1} + w_{2} * i_{2} + b_{1} * 1$

$i n_{h_{1}} = 0.15 * 0.05 + 0.2 * 0.1 + 0.35 * 1$

$i n_{h_{1}} = 0.3775$

Input của $h_{2}$ :

$i n_{h_{2}} = w_{3} * i_{1} + w_{4} * i_{2} + b_{1} * 1$

$i n_{h_{2}} = 0.25 * 0.05 + 0.3 * 0.1 + 0.35 * 1$

$i n_{h_{2}} = 0.3925$

Ouput của $h_{1}$ :

$o u t_{h_{1}} =$ $\frac{1}{1 + e^{- i n_{h_{1}}}}$

$o u t_{h_{1}} = \frac{1}{1 + e^{- 0.3775}}$

$o u t_{h_{1}} = 0.593269992$

Output của $h_{2}$ :

$o u t_{h_{2}} =$ $\frac{1}{1 + e^{- i n_{h_{2}}}}$

$o u t_{h_{2}} = \frac{1}{1 + e^{- 0.3925}}$

$o u t_{h_{2}} = 0.596884378$

Input của $o_{1}$ :

$i n_{o_{1}} = w_{5} * o u t_{h_{1}} + w_{6} * o u t_{h_{2}} + b_{2} * 1$

$i n_{o_{1}} = 0.4 * 0.593269992 + 0.45 * 0.596884378 + 0.6 * 1$

$i n_{o_{1}} = 1.105905967$

Input của $o_{2}$ :

$i n_{o_{2}} = w_{7} * o u t_{h_{1}} + w_{8} * o u t_{h_{2}} + b_{2} * 1$

$i n_{o_{2}} = 0.5 * 0.593269992 + 0.55 * 0.596884378 + 0.6 * 1$

$i n_{o_{2}} = 1.224921404$

Output của $o_{1}$ :

$o u t_{o_{1}} =$ $\frac{1}{1 + e^{- i n_{o_{1}}}}$

$o u t_{o_{1}} = \frac{1}{1 + e^{- 1.105905967}}$

$o u t_{o_{1}} = 0.75136507$

Output cuat $o_{2}$ :

$o u t_{o_{2}} =$ $\frac{1}{1 + e^{- i n_{o_{2}}}}$

$o u t_{o_{2}} = \frac{1}{1 + e^{- 1.224921404}}$

$o u t_{o_{2}} = 0.772928465$

Tổng lỗi:

$E_{o_{1}} = \frac{1}{2} (t a r g e t_{o_{1}} - o u t_{o_{1}})^{2}$

$E_{o_{2}} = \frac{1}{2} (0.01 - 0.75136507)^{2}$

$E_{o_{1}} = 0.274811083$

$E_{o_{2}} = \frac{1}{2} (t a r g e t_{o_{1}} - o u t_{o_{1}})^{2}$

$E_{o_{2}} = \frac{1}{2} (0.01 - 0.772928465)^{2}$

$E_{o_{2}} = 0.023560026$

$E_{t o t a l} = \sum_{i = 1}^{2} E_{o_{i}}$

$E_{t o t a l} = 0.274811083 + 0.023560026$

$E_{t o t a l} = 0.298371109$

2.2 Backward Path

Tính đạo hàm từng phần của Loss Function theo mỗi $w$ .

Các $w$ của output layer ( $w_{5}, w_{6}, w_{7}, w_{8}$ ) có cách tính giống nhau:

$w_{5}$ :

Ta biết:

$E_{t o t a l} = \sum_{i = 1}^{2} \frac{1}{2} (t a r g e t_{o_{i}} - o u t_{o_{i}})^{2}$

$E_{t o t a l} = \frac{1}{2} (t a r g e t_{o_{1}} - o u t_{o_{1}})^{2} + \frac{1}{2} (t a r g e t_{o_{2}} - o u t_{o_{2}})^{2}$

Nên:

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{1}}}$ $= 2 * \frac{1}{2} (t a r g e t_{o_{1}} - o u t_{o_{1}})^{2 - 1} * (- 1) + 0$

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{1}}}$ $= - (t a r g e t_{o_{1}} - o u t_{o_{1}})$

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{1}}}$ $= - (0.01 - 0.75136507) = 0.74136507$

Tiếp theo, vì:

$o u t_{o_{1}} = s i g m o i d (i n_{o_{1}}) =$ $\frac{1}{1 + e^{- i n_{01}}}$

Nên:

$\frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}}$ $= o u t_{o_{1}} (1 - o u t_{o_{1}})$

$\frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}}$ $= 0.75136507 (1 - 0.75136507)$

$\frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}}$ $= 0.186815602$

Và,

$i n_{o_{1}} = w_{5} * o u t_{h_{1}} + w_{6} * o u t_{h_{2}} + b_{2} * 1$

Nên:

$\frac{\partial i n_{o_{1}}}{\partial w_{5}}$ $= o u t_{h_{1}}$

$\frac{\partial i n_{o_{1}}}{\partial w_{5}}$ $= 0.593269992$

Tổng hợp lại ta được:

$\frac{\partial E_{t o t a l}}{\partial w_{5}}$ $= 0.74136507 * 0.186815602 * 0.593269992$

$\frac{\partial E_{t o t a l}}{\partial w_{5}}$ $= 0.082167041$

$w_{6}$ :

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{1}}} = 2 * \frac{1}{2} (t a r g e t_{o_{1}} - o u t_{o_{1}})^{2 - 1} * (- 1) + 0$ $

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{1}}}$ $= - (t a r g e t_{o_{1}} - o u t_{o_{1}})$

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{1}}}$ $= - (0.01 - 0.75136507) = 0.74136507$

$\frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}}$ $= o u t_{o_{1}} (1 - o u t_{o_{1}})$

$\frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}}$ $= 0.75136507 (1 - 0.75136507)$

$\frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}}$ $= 0.186815602$

$\frac{\partial i n_{o_{1}}}{\partial w_{6}}$ $= o u t_{h_{2}}$

$\frac{\partial i n_{o_{1}}}{\partial w_{6}}$ $= 0.596884378$

Tổng hợp lại ta được:

$\frac{\partial E_{t o t a l}}{\partial w_{6}}$ $= 0.74136507 * 0.186815602 * 0.596884378$

$\frac{\partial E_{t o t a l}}{\partial w_{6}}$ $= 0.082667628$

$w_{7}$ :

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{2}}} = 0 + 2 * \frac{1}{2} (t a r g e t_{o_{2}} - o u t_{o_{2}})^{2 - 1} * (- 1)$

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{2}}}$ $= - (t a r g e t_{o_{2}} - o u t_{o_{2}})$

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{2}}}$ $= - (0.99 - 0.772928465) = - 0.217071535$

$\frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}}$ $= o u t_{o_{2}} (1 - o u t_{o_{2}})$

$\frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}}$ $= 0.772928465 (1 - 0.772928465)$

$\frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}}$ $= 0.175510053$

$\frac{\partial i n_{o_{2}}}{\partial w_{7}}$ $= o u t_{h_{1}}$

$\frac{\partial i n_{o_{2}}}{\partial w_{7}}$ $= 0.593269992$

Tổng hợp lại ta được:

$\frac{\partial E_{t o t a l}}{\partial w_{6}}$ $= - 0.217071535 * 0.175510053 * 0.593269992$

$\frac{\partial E_{t o t a l}}{\partial w_{6}}$ $= - 0.022602541$

$w_{8}$ :

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{2}}} = 0 + 2 * \frac{1}{2} (t a r g e t_{o_{2}} - o u t_{o_{2}})^{2 - 1} * (- 1)$

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{2}}}$ $= - (t a r g e t_{o_{2}} - o u t_{o_{2}})$

$\frac{\partial E_{t o t a l}}{\partial o u t_{o_{2}}}$ $= - (0.99 - 0.772928465) = - 0.217071535$

$\frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}}$ $= o u t_{o_{2}} (1 - o u t_{o_{2}})$

$\frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}}$ $= 0.772928465 (1 - 0.772928465)$

$\frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}}$ $= 0.175510053$

$\frac{\partial i n_{o_{2}}}{\partial w_{8}}$ $= o u t_{h_{2}}$

$\frac{\partial i n_{o_{2}}}{\partial w_{8}}$ $= 0.596884378$

Tổng hợp lại ta được:

$\frac{\partial E_{t o t a l}}{\partial w_{6}}$ $= - 0.217071535 * 0.175510053 * 0.596884378$

$\frac{\partial E_{t o t a l}}{\partial w_{6}}$ $= - 0.022740242$

Các $w$ của hidden layer ( $w_{1}, w_{2}, w_{3}, w_{4}$ ) có cách tính giống nhau:

$w_{1}$ :

--------------------------------------------------------------

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}} + \frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}}$

----------------------------------------------------

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}}$

---------------------------------

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{o_{1}}} * \frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}}$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} = \frac{\partial (\frac{1}{2} (t a r g e t_{o_{1}} - o u t_{o_{1}})^{2})}{\partial o u t_{o_{1}}} * \frac{\partial (\frac{1}{1 + e^{- i n_{o_{1}}}})}{\partial i n_{o_{1}}}$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= 2 * \frac{1}{2} (t a r g e t_{o_{1}} - o u t_{o_{1}}) * (- 1) * o u t_{o_{1}} (1 - o u t_{o_{1}})$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= (0.01 - 0.75136507) * (- 1) * 0.75136507 (1 - 0.75136507)$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= 0.138498562$

-------------------------

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}} = \frac{\partial (w_{5} * o u t_{h_{1}} + w_{6} * o u t_{h_{2}} + b_{2} * 1)}{\partial o u t_{h_{1}}}$

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}}$ $= w_{5}$

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}}$ $= 0.4$

Gộp lại:

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}}$

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}}$ $= 0.138498562 * 0.4$

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}}$ $= 0.055399425$

----------------------------------------------------

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}} * \frac{\partial i n_{o_{2}}}{\partial o u t_{h_{1}}}$

---------------------------------

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}} = \frac{\partial E_{o_{2}}}{\partial o u t_{o_{2}}} * \frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}}$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}} = \frac{\partial (\frac{1}{2} (t a r g e t_{o_{2}} - o u t_{o_{2}})^{2})}{\partial o u t_{o_{2}}} * \frac{\partial (\frac{1}{1 + e^{- i n_{o_{2}}}})}{\partial i n_{o_{2}}}$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{1}}}$ $= 2 * \frac{1}{2} (t a r g e t_{o_{2}} - o u t_{o_{2}}) * (- 1) * o u t_{o_{2}} (1 - o u t_{o_{2}})$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{1}}}$ $= (0.99 - 0.772928465) * (- 1) * 0.772928465 (1 - 0.772928465)$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{1}}}$ $= - 0.038098237$

-------------------------

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{1}}} = \frac{\partial (w_{7} * o u t_{h_{1}} + w_{8} * o u t_{h_{2}} + b_{2} * 1)}{\partial o u t_{h_{1}}}$

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{1}}}$ $= w_{7}$

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{1}}}$ $= 0.5$

Gộp lại:

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{2}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}}$

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}}$ $= - 0.038098237 * 0.5$

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}}$ $= - 0.019049118$

---------------------------------

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}} + \frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}}$

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{1}}}$ $= 0.055399425 + (- 0.019049118) = 0, 036350307$

----------------------

$\frac{\partial o u t_{h_{1}}}{\partial i n_{h_{1}}} = \frac{\partial (\frac{1}{1 + e^{- i n_{h_{1}}}})}{\partial i n_{h_{1}}}$

$\frac{\partial o u t_{h_{1}}}{\partial i n_{h_{1}}}$ $= o u t_{h_{1}} (1 - o u t_{h_{1}})$

$\frac{\partial o u t_{h_{1}}}{\partial i n_{h_{1}}}$ $= 0.59326999 (1 - 0.59326999) = 0.241300709$

----------------------

$\frac{\partial i n_{h_{1}}}{\partial w_{1}} = \frac{\partial (w_{1} * i_{1} + w_{2} * i_{2} + b_{1} * 1)}{\partial w_{1}}$

$\frac{\partial i n_{h_{1}}}{\partial w_{1}}$ $= i_{1}$

$\frac{\partial i n_{h_{1}}}{\partial w_{1}}$ $= 0.05$

--------------------------------------------------------------

$\frac{\partial E_{t o t a l}}{\partial w_{1}}$ $= 0.036350306 * 0.241300709 * 0.05$

$\frac{\partial E_{t o t a l}}{\partial w_{1}}$ $= 0.000438568$

$w_{2}$ :

--------------------------------------------------------------

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}} + \frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}}$

----------------------------------------------------

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}}$

---------------------------------

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{o_{1}}} * \frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}}$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= 2 * \frac{1}{2} (t a r g e t_{o_{1}} - o u t_{o_{1}}) * (- 1) * o u t_{o_{1}} (1 - o u t_{o_{1}})$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= (0.01 - 0.75136507) * (- 1) * 0.75136507 (1 - 0.75136507)$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= 0.138498562$

-------------------------

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}} = \frac{\partial (w_{5} * o u t_{h_{1}} + w_{6} * o u t_{h_{2}} + b_{2} * 1)}{\partial o u t_{h_{1}}}$

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}}$ $= w_{5}$

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}}$ $= 0.4$

Gộp lại:

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}}$

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}}$ $= 0.138498562 * 0.4$

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}}$ $= 0.055399425$

----------------------------------------------------

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}} * \frac{\partial i n_{o_{2}}}{\partial o u t_{h_{1}}}$

---------------------------------

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}} = \frac{\partial E_{o_{2}}}{\partial o u t_{o_{2}}} * \frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}}$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{1}}}$ $= 2 * \frac{1}{2} (t a r g e t_{o_{2}} - o u t_{o_{2}}) * (- 1) * o u t_{o_{2}} (1 - o u t_{o_{2}})$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{1}}}$ $= (0.99 - 0.772928465) * (- 1) * 0.772928465 (1 - 0.772928465)$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{1}}}$ $= - 0.038098237$

-------------------------

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{1}}} = \frac{\partial (w_{7} * o u t_{h_{1}} + w_{8} * o u t_{h_{2}} + b_{2} * 1)}{\partial o u t_{h_{1}}}$

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{1}}}$ $= w_{7}$

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{1}}}$ $= 0.5$

Gộp lại:

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{2}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{1}}}$

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}}$ $= - 0.038098237 * 0.5$

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}}$ $= - 0.019049118$

---------------------------------

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{1}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{h_{1}}} + \frac{\partial E_{o_{2}}}{\partial o u t_{h_{1}}}$

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{1}}}$ $= 0.055399425 + (- 0.019049118) = 0, 036350307$

----------------------

$\frac{\partial o u t_{h_{1}}}{\partial i n_{h_{1}}} = \frac{\partial (\frac{1}{1 + e^{- i n_{h_{1}}}})}{\partial i n_{h_{1}}}$

$\frac{\partial o u t_{h_{1}}}{\partial i n_{h_{1}}}$ $= o u t_{h_{1}} (1 - o u t_{h_{1}})$

$\frac{\partial o u t_{h_{1}}}{\partial i n_{h_{1}}}$ $= 0.59326999 (1 - 0.59326999) = 0.241300709$

----------------------

$\frac{\partial i n_{h_{1}}}{\partial w_{2}} = \frac{\partial (w_{1} * i_{1} + w_{2} * i_{2} + b_{1} * 1)}{\partial w_{2}}$

$\frac{\partial i n_{h_{1}}}{\partial w_{2}}$ $= i_{2}$

$\frac{\partial i n_{h_{1}}}{\partial w_{2}}$ $= 0.1$

--------------------------------------------------------------

$\frac{\partial E_{t o t a l}}{\partial w_{2}}$ $= 0.036350306 * 0.241300709 * 0.1$

$\frac{\partial E_{t o t a l}}{\partial w_{2}}$ $= 0.000877135$

$w_{3}$ :

--------------------------------------------------------------

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}} + \frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}}$

----------------------------------------------------

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}}$

---------------------------------

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{o_{1}}} * \frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}}$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= 2 * \frac{1}{2} (t a r g e t_{o_{1}} - o u t_{o_{1}}) * (- 1) * o u t_{o_{1}} (1 - o u t_{o_{1}})$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= (0.01 - 0.75136507) * (- 1) * 0.75136507 (1 - 0.75136507)$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= 0.138498562$

-------------------------

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}} = \frac{\partial (w_{5} * o u t_{h_{1}} + w_{6} * o u t_{h_{2}} + b_{2} * 1)}{\partial o u t_{h_{2}}}$

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}}$ $= w_{6}$

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}}$ $= 0.45$

Gộp lại:

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}}$

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}}$ $= 0.138498562 * 0.45$

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}}$ $= 0.062324353$

----------------------------------------------------

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}} * \frac{\partial i n_{o_{2}}}{\partial o u t_{h_{2}}}$

---------------------------------

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}} = \frac{\partial E_{o_{2}}}{\partial o u t_{o_{2}}} * \frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}}$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}}$ $= 2 * \frac{1}{2} (t a r g e t_{o_{2}} - o u t_{o_{2}}) * (- 1) * o u t_{o_{2}} (1 - o u t_{o_{2}})$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}}$ $= (0.99 - 0.772928465) * (- 1) * 0.772928465 (1 - 0.772928465)$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}}$ $= - 0.038098237$

-------------------------

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{2}}} = \frac{\partial (w_{7} * o u t_{h_{1}} + w_{8} * o u t_{h_{2}} + b_{2} * 1)}{\partial o u t_{h_{2}}}$

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{2}}}$ $= w_{8}$

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{2}}}$ $= 0.55$

Gộp lại:

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{2}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}}$

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}}$ $= - 0.038098237 * 0.55$

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}}$ $= - 0.02095403$

---------------------------------

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}} + \frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}}$

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{2}}}$ $= 0.062324353 + (- 0.02095403) = 0.041370323$

----------------------

$\frac{\partial o u t_{h_{2}}}{\partial i n_{h_{2}}} = \frac{\partial (\frac{1}{1 + e^{- i n_{h_{2}}}})}{\partial i n_{h_{2}}}$

$\frac{\partial o u t_{h_{2}}}{\partial i n_{h_{2}}}$ $= o u t_{h_{2}} (1 - o u t_{h_{2}})$

$\frac{\partial o u t_{h_{2}}}{\partial i n_{h_{2}}}$ $= 0.596884378 (1 - 0.596884378) = 0.240613417$

----------------------

$\frac{\partial i n_{h_{2}}}{\partial w_{3}} = \frac{\partial (w_{3} * i_{1} + w_{4} * i_{2} + b_{1} * 1)}{\partial w_{3}}$

$\frac{\partial i n_{h_{2}}}{\partial w_{3}}$ $= i_{1}$

$\frac{\partial i n_{h_{2}}}{\partial w_{3}}$ $= 0.05$

--------------------------------------------------------------

$\frac{\partial E_{t o t a l}}{\partial w_{3}}$ $= 0.041370323 * 0.240613417 * 0.05$

$\frac{\partial E_{t o t a l}}{\partial w_{3}}$ $= 0.000497713$

$w_{4}$ :

--------------------------------------------------------------

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}} + \frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}}$

----------------------------------------------------

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}}$

---------------------------------

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{o_{1}}} * \frac{\partial o u t_{o_{1}}}{\partial i n_{o_{1}}}$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= 2 * \frac{1}{2} (t a r g e t_{o_{1}} - o u t_{o_{1}}) * (- 1) * o u t_{o_{1}} (1 - o u t_{o_{1}})$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= (0.01 - 0.75136507) * (- 1) * 0.75136507 (1 - 0.75136507)$

$\frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}}$ $= 0.138498562$

-------------------------

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}} = \frac{\partial (w_{5} * o u t_{h_{1}} + w_{6} * o u t_{h_{2}} + b_{2} * 1)}{\partial o u t_{h_{2}}}$

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}}$ $= w_{6}$

$\frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}}$ $= 0.45$

Gộp lại:

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{1}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}}$

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}}$ $= 0.138498562 * 0.45$

$\frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}}$ $= 0.062324353$

----------------------------------------------------

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}} * \frac{\partial i n_{o_{2}}}{\partial o u t_{h_{2}}}$

---------------------------------

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}} = \frac{\partial E_{o_{2}}}{\partial o u t_{o_{2}}} * \frac{\partial o u t_{o_{2}}}{\partial i n_{o_{2}}}$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}}$ $= 2 * \frac{1}{2} (t a r g e t_{o_{2}} - o u t_{o_{2}}) * (- 1) * o u t_{o_{2}} (1 - o u t_{o_{2}})$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}}$ $= (0.99 - 0.772928465) * (- 1) * 0.772928465 (1 - 0.772928465)$

$\frac{\partial E_{o_{2}}}{\partial i n_{o_{2}}}$ $= - 0.038098237$

-------------------------

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{2}}} = \frac{\partial (w_{7} * o u t_{h_{1}} + w_{8} * o u t_{h_{2}} + b_{2} * 1)}{\partial o u t_{h_{2}}}$

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{2}}}$ $= w_{8}$

$\frac{\partial i n_{o_{2}}}{\partial o u t_{h_{2}}}$ $= 0.55$

Gộp lại:

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{2}}}{\partial i n_{o_{1}}} * \frac{\partial i n_{o_{1}}}{\partial o u t_{h_{2}}}$

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}}$ $= - 0.038098237 * 0.55$

$\frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}}$ $= - 0.02095403$

---------------------------------

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{2}}} = \frac{\partial E_{o_{1}}}{\partial o u t_{h_{2}}} + \frac{\partial E_{o_{2}}}{\partial o u t_{h_{2}}}$

$\frac{\partial E_{t o t a l}}{\partial o u t_{h_{2}}}$ $= 0.062324353 + (- 0.02095403) = 0.041370323$

----------------------

$\frac{\partial o u t_{h_{2}}}{\partial i n_{h_{2}}} = \frac{\partial (\frac{1}{1 + e^{- i n_{h_{2}}}})}{\partial i n_{h_{2}}}$

$\frac{\partial o u t_{h_{2}}}{\partial i n_{h_{2}}}$ $= o u t_{h_{2}} (1 - o u t_{h_{2}})$

$\frac{\partial o u t_{h_{2}}}{\partial i n_{h_{2}}}$ $= 0.596884378 (1 - 0.596884378) = 0.240613417$

----------------------

$\frac{\partial i n_{h_{2}}}{\partial w_{4}} = \frac{\partial (w_{3} * i_{1} + w_{4} * i_{2} + b_{1} * 1)}{\partial w_{3}}$

$\frac{\partial i n_{h_{2}}}{\partial w_{4}}$ $= i_{2}$

$\frac{\partial i n_{h_{2}}}{\partial w_{4}}$ $= 0.1$

--------------------------------------------------------------

$\frac{\partial E_{t o t a l}}{\partial w_{4}}$ $= 0.041370323 * 0.240613417 * 0.1$

$\frac{\partial E_{t o t a l}}{\partial w_{4}}$ $= 0.000995425$

Đến đây ta đã tính xong các đạo hàm từng phần theo các $w$ . Áp dụng SGD để cập nhật các $w$ ta được (chọn $η = 0.9$ ):

$w_{5}^{+} = w_{5} - η *$ $\frac{\partial E_{t o t a l}}{\partial w_{5}}$

$w_{5}^{+} = 0.4 - 0.9 * 0.082167041$

$w_{5}^{+} = 0.326049663$

-------------------------

$w_{6}^{+} = w_{6} - η *$ $\frac{\partial E_{t o t a l}}{\partial w_{6}}$

$w_{6}^{+} = 0.45 - 0.9 * 0.082667628$

$w_{6}^{+} = 0.375599135$

-------------------------

$w_{7}^{+} = w_{7} - η *$ $\frac{\partial E_{t o t a l}}{\partial w_{7}}$

$w_{7}^{+} = 0.5 - 0.9 * (- 0.022602541)$

$w_{7}^{+} = 0.520342287$

-------------------------

$w_{8}^{+} = w_{8} - η *$ $\frac{\partial E_{t o t a l}}{\partial w_{8}}$

$w_{8}^{+} = 0.55 - 0.9 * (- 0.022740242)$

$w_{8}^{+} = 0.570466218$

-------------------------

$w_{1}^{+} = w_{1} - η *$ $\frac{\partial E_{t o t a l}}{\partial w_{1}}$

$w_{1}^{+} = 0.15 - 0.9 * 0.000438568$

$w_{1}^{+} = 0.149605289$

-------------------------

$w_{2}^{+} = w_{2} - η *$ $\frac{\partial E_{t o t a l}}{\partial w_{2}}$

$w_{2}^{+} = 0.2 - 0.9 * 0.0080877135$

$w_{2}^{+} = 0.192721058$

-------------------------

$w_{3}^{+} = w_{3} - η *$ $\frac{\partial E_{t o t a l}}{\partial w_{3}}$

$w_{3}^{+} = 0.25 - 0.9 * 0.000497713$

$w_{3}^{+} = 0.249552058$

-------------------------

$w_{4}^{+} = w_{4} - η *$ $\frac{\partial E_{t o t a l}}{\partial w_{4}}$

$w_{4}^{+} = 0.3 - 0.9 * 0.000995425$

$w_{4}^{+} = 0.299104118$

-------------------------

Phù, như vậy là chúng ta đã cập nhật xong giá trị mới cho các trọng số $w$ . Đây là những phép toán xảy ra trong mỗi lần cập nhật khi training model. Hi vọng, thông qua ví dụ trong bài này, các bạn đã có thể hiểu rõ hơn bản chất của mạng NN. Hẹn gặp lại các bạn trong các bài tiếp theo!

3. Tham khảo

Neural Network cơ bản (Phần 2)