"Small is the number of people who see with their eyes and think with their minds." Einstein

MapReduce

Split the data set in multiple pieces of data sets. Then multiple machines will execute such a piece of data.

Batch gradient descent:
$\theta_j := \theta_j - \alpha \frac{1}{400}\sum_{i=1}^{400}(h_{\theta}(x^{(i)}) - y^{(i)})\,x_j^{(i)}$

Machine 1: Use $(x^{(1)}, y^{(1)}), ... ,(x^{(100)}, y^{(100)})$
$temp_j^{(1)} = \sum_{i=1}^{100}(h_{\theta}(x^{(i)}) - y^{(i)})\,x_j^{(i)}$

Machine 2: Use $(x^{(101)}, y^{(101)}), ... ,(x^{(200)}, y^{(200)})$
$temp_j^{(2)} = \sum_{i=101}^{200}(h_{\theta}(x^{(i)}) - y^{(i)})\,x_j^{(i)}$

Machine 3: Use $(x^{(201)}, y^{(201)}), ... ,(x^{(300)}, y^{(300)})$
$temp_j^{(201)} = \sum_{i=201}^{300}(h_{\theta}(x^{(i)}) - y^{(i)})\,x_j^{(i)}$

Machine 4: Use $(x^{(301)}, y^{(301)}), ... ,(x^{(400)}, y^{(400)})$
$temp_j^{(301)} = \sum_{i=301}^{400}(h_{\theta}(x^{(i)}) - y^{(i)})\,x_j^{(i)}$

At the end send all these results to a master server and combine:
$\theta_j := \theta_j - \alpha \frac{1}{400}(temp_j^{(1)} + temp_j^{(2)} + temp_j^{(3)} + temp_j^{(4)})$
where
$j = (0, ..., n)$

And this last equation is equal to the first equation of Batch Gradient Descent.

Summation

Many learning algorithms can be expressed as computing sums of functions over the trainging set.