Draft:Collinear gradients method

Collinear gradients method (ColGM)  is an iterative method of directional search for the local extremum of a smooth multivariate function $J(u)\colon \mathbb {R} ^{n}\to \mathbb {R}$, which do moving towards the extremum along the vector $d\in \mathbb {R} ^{n}$ such that the gradients $\nabla J(u)\parallel \nabla J(u+d)$, i.e. they are collinear vectors. This is a first-order method (it uses only the first derivatives $\nabla J$) with a quadratic convergence rate. It can be applied to functions of high dimension $n$ with several local extremes. GolGM can be attributed to the Truncated Newton method family.

Collinear vectors $\nabla J(u^{k})$ and $\nabla J(u^{k\ast })$ with the direction of minimization $d^{k}$ for a convex function, $n=2$

The concept of the method

For a smooth function $J(u)$ in a relatively large vicinity of a point $u^{k}$, there is a point $u^{k_{\ast }}$, where the gradients $\nabla J^{k}\,{\overset {\textrm {def}}{=}}\,\nabla J(u^{k})$ and $\nabla J^{k_{\ast }}\,{\overset {\textrm {def}}{=}}\,\nabla J(u^{k_{\ast }})$ are collinear vectors. The direction to the extremum $u_{\ast }$ from the point $u^{k}$ will be the direction $d^{k}={(u^{k_{\ast }}-u}^{k})$. The vector $d^{k}$ points to the maximum or minimum, depending on the position of the point $u^{k_{\ast }}$. It can be in front or behind of $u^{k}$ relative to the direction to $u_{\ast }$ (see the picture). Next, we will consider minimization.

The next iteration of ColGM: (1) $\quad u^{k+1}=u^{k}+b^{k}d^{k},\quad k\in \left\{0,1\dots \right\},$ where the optimal $b^{k}\in \mathbb {R}$ is found analytically from the assumption of a quadratic one-dimensional function $J(u^{k}+bd^{k})$:

(2) $\quad b^{k}=\left(1-{\frac {\langle \nabla J(u^{k_{\ast }},d^{k}\rangle }{\langle \nabla J(u^{k}),d^{k}\rangle }}\right)^{-1},\quad \forall u^{k_{\ast }}.$

Angle brackets are an inner product space in the Euclidean space $\mathbb {R} ^{n}$. If $J(u)$ is a convex function in the vicinity of $u^{k}$, then for the front point $u^{k_{\ast }}$ we get the number $b^{k}>0$, for the back $b^{k}<0$. In any case, we follow step (1).

For a strictly convex quadratic function $J(u)$ the ColGM step is $b^{k}d^{k}=-H^{-1}\nabla J^{k},$ i.e. it is a Newton's step (a second-order method with a quadratic convergence rate), where $H$ is the Hesse matrix. Such steps ensure the quadratic convergence rate for ColGM.

In general, if $J(u)$ has a variable convexity and saddle points are possible, then the minimization direction should be checked by the angle $\gamma$ between the vectors $\nabla J^{k}$ and $d^{k}$. If $\cos(\gamma )={\frac {\langle \nabla J^{k},d^{k}\rangle }{||\nabla J(u^{k})||\;||d^{k}||}}\geq 0$, then $d^{k}$ is the direction of maximization, and in (1) we should take $b^{k}$ with the opposite sign.

Search for collinear gradients

Collinearity of gradients is estimated by the residual of their directions, which has the form of a system of $n$ equations for search a root $u=u^{k_{\ast }}$: (3) $\quad r^{k}(u)={\frac {\nabla J(u)}{||\nabla J(u)||}}s-{\frac {\nabla J(u^{k})}{||\nabla J(u^{k})||}}=0\in \mathbb {R} ^{n},$ where the sign $s=\operatorname {sgn} \langle \nabla J(u),\nabla J(u^{k})\rangle$, this allows us to equally evaluate the collinearity of gradients, both co-directional and oppositely directed, $||r^{k}(u)||\leq {\sqrt {2}}$.

System (3) is solved iteratively (sub-iterations $l\,$) by the conjugate gradient method, assuming that the system is linear in the $u^{k}$-vicinity:

(4) $\quad u^{k_{l+1}}=u^{k_{l}}+\tau ^{l}p^{l},\quad l=1,2\ldots ,$

where vector $\;p^{l}\;{\overset {\textrm {def}}{=}}\,p(u^{k_{l}})=-r^{l}+{\beta ^{l}p}^{l-1}$, $\;r^{l}\,{\overset {\textrm {def}}{=}}\,r(u^{k_{l}})$, $\;\beta ^{l}=||r^{l}||^{2}/||r^{l-1}||^{2},\ \beta ^{1,n,2n...}=0$, $\;\tau ^{l}=||r^{l}||^{2}/\langle p^{l},H^{l}p^{l}\rangle$, the product of the Hesse matrix $H^{l}$ by $p^{l}$ is found by numerical differentiation:

(5) $\quad H^{l}p^{l}\approx {\frac {r(u^{k_{h}})-r(u^{k_{l}})}{h/||p^{l}||}},$

where $u^{k_{h}}=u^{k_{l}}+hp^{l}/||p^{l}||$, $h$ is a small positive number such that $\langle p^{l},H^{l}p^{l}\rangle \neq 0$.

The initial approximation is set at 45° to all coordinate axes and $\delta ^{k}$-length:

(6) $\quad u_{i}^{k_{1}}=u_{i}^{k}+{\frac {\delta ^{k}}{\sqrt {n}}}\operatorname {sgn} {\ \nabla _{i}J}^{k},\quad i=1\ldots n.$

The initial radius $\delta ^{k}$ is the vicinity of the point $u^{k}$ and it is modifid:

(7) $\quad \delta ^{k}=\max \left[\min \left(\delta ^{k-1}{\frac {||\nabla J(u^{k})||}{||\nabla J(u^{k-1})||}},\delta ^{0}\right),\delta _{m}\right],\quad k>0.$

Necessary $||u^{k_{l}}-u^{k}||\geq \delta ^{m},\;l\geq 1$. Here, the small positive number $\delta _{m}$ is noticeably larger than the machine epsilon.

Sub-iterations $l$ terminate when at least one of the conditions is met:

1. $||r^{l}||\leq c_{1}{\sqrt {2}},\quad 0\leq c_{1}<1$ — accuracy achieved;
2. $\left|{\frac {||r^{l}||-||r^{l-1}||}{||r^{l}||}}\right|\leq c_{1},\quad l>1$ — convergence has stopped;
3. $l\leq l_{max}=\operatorname {integer} \left|c_{2}\ln c_{1}\ln n\right|,\quad c_{2}\geq 1$ — redundancy of sub-iterations.

Algorithm for choosing the minimization direction

• Parameters: $c_{1},c_{2},\delta ^{0},\delta _{m}=10^{-15}\delta ^{0},h=10^{-5}$.
• Input data: $n,k=0,u^{k},J(u^{k}),\nabla J(u^{k}),\nabla J^{k}$.

1. $l=1$. If $k>0$ then set $\delta ^{k}$ from (7).
2. Find $u^{k_{l}}$ from (6).
3. Calculate $\nabla J^{l},||\nabla J^{l}||$. Find $r^{l}$ from (3) for $u=u^{k_{l}}$.
4. If $||r^{l}||\leq c_{1}{\sqrt {2}}\,$ or $l\geq l_{max}$ or $||u^{k_{l}}-u^{k}||<\delta _{m}$ or {$\,l>1$ and $\left|{\frac {||r^{l}||-||r^{l-1}||}{||r^{l}||}}\right|\leq c_{1}$} then set $u^{k_{\ast }}=u^{k_{l}}$, return $\nabla J\left(u^{k_{\ast }}\right)$, $d^{k}={(u^{k_{\ast }}-u}^{k})$, stop.
5. If $l\neq 1,n,2n,3n\ldots$ then set $\beta ^{l}=||r^{l}||^{2}/||r^{l-1}||^{2}$ else $\beta ^{l}=0$.
6. Find $p^{l}=-r^{l}+\beta ^{l}p^{l-1}$.
7. Searching for $\tau ^{l}$:
   1. Memorize $u^{k_{l}}$, $\nabla J^{l}$, $||\nabla J^{l}||$, $r^{l}$, $||r^{l}||$;
   2. Find $u^{k_{h}}=u^{k_{l}}+hp^{l}/||p^{l}||$. Calculate $\nabla J(u^{k_{h}})$ and $r\left(u^{k_{h}}\right)$. Find $H^{l}p^{l}$ from (5) and assign $w\leftarrow \langle p^{l},H^{l}p^{l}\rangle$;
   3. If $w=0$ then $h\leftarrow 10h$ and return to step 7.2;
   4. Restore $u^{k_{l}}$, $\nabla J^{l}$, $||\nabla J^{l}||$, $r^{l}$, $||r^{l}||$;
   5. Set $\tau ^{l}=||r^{l}||^{2}/w$.
8. Perform sub-iteration $u^{k_{l+1}}$ from (4).
9. $l\leftarrow l+1$, Go to step 3

The parameter $c_{2}=3\div 5$. For functions without saddle points, we recommend $c_{1}\approx 10^{-8}$, $\delta \approx {10}^{-5}$. To "bypass" saddle points, we recommend $c_{1}\approx 0.1$, $\delta \approx 0.1$.

The described algorithm allows us to approximately find collinear gradients from the system of equations (3). The resulting direction $b^{k}d^{k}$ for the ColGM algorithm (1) will be approximate Newton direction (truncated Newton method).

Demonstrations

In all the demonstrations, ColGM shows convergence no worse and sometimes even better (for functions of variable convexity) than Newton's method.

The "rotated ellipsoid" test function

A strictly convex quadratic function:

ColGM minimization, $n=2$

$J(u)=\sum _{i=1}^{n}\left(\sum _{j=1}^{i}u_{j}\right)^{2},\quad u_{\ast }=(0...0).$

In the drawing, three black starting points $u^{0}$ are set for ${\color {red}n=2}$. The gray dots are sub-iterations of $u^{0_{l}}$ with $\delta ^{0}=0.5$ (shown as a dotted line, inflated for demonstration). Parameters $c_{1}=10^{-8}$, $c_{2}=4$. It took one iteration for all $u^{0}$ and no more than two sub-iterations $l$.

For ${\color {red}n=1000}$ (parameter $\delta ^{0}={10}^{-5}$) with the starting point $u^{0}=(-1...1)$ ColGM achieved $u_{\ast }$ with an accuracy of 1% in 3 iterations and 754 calculations $J$ and $\nabla J$. Other first-order methods: Quasi-Newtonian BFGS (working with matrices) required 66 iterations and 788 calculations; conjugate gradients (Fletcher—Reeves) - 274 iterations and 2236 calculations; Newton's finite difference method — 1 iteration and 1001 calculations. Newton's method second order — 1 iteration.

As the dimension of $\color {red}n$ increases, computational errors in the implementation of the collinearity condition (3) may increase markedly. Because of this, the ColGM, in comparison with the Newton's method, in the considered example required more than one iteration.

ColGM minimization: 3 iterations and 16 calculations $J$ and $\nabla J$

Test function Rosenbrock

$J(u)=100(u_{1}^{2}-u_{2})^{2}+(u_{1}-1)^{2},\quad u_{\ast }=(1,1).$

The parameters are the same, except $\delta ^{0}={10}^{-5}$. The descent trajectory of the ColGM completely coincides with the Newton's method. In the drawing, the blue starting point is $u^{0}=\left(-0.8;-1.2\right)$, and the red one is $u_{\ast }$. Unit vector of the gradient are drawn at each point $u^{k}$.

Test function Himmelblau function

$J(u)=(u_{1}^{2}+u_{2}-11)^{2}+(u_{1}+u_{2}^{2}-7)^{2}.$

Parameters $c_{1}=0.1$, $\delta =0.05$.

ColGM minimization: 7 iterations and 22 calculations $J$ and $\nabla J$. The red lines are $\cos {\gamma }\geq 0$.	| Minimization by Newton's method: 9 iterations ($b^{k}=1$)
Minimization by conjugate gradient method (Fletcher-Reeves): 9 iterations and 62 calculations $J$ and $\nabla J$	| Minimization by quasi-Newton BFGS: 6 iterations and 55 calculations $J$ and $\nabla J$. Red line (violation of the curvature condition) — steepest descent method.

ColGM is very economical in terms of the number of calculations $J$ and $\nabla J$. Due to formula (2), it does not require expensive calculations of the step multiplier $b^{k}$ by linear search (for example, golden-section search, etc.).