text:
\input{algorithm/madam}

\section{Making the algorithm practical---the \textit{Madam} optimiser}\label{sec:madam}

In the previous section, we built an optimisation theory for the multiplicative update rule appearing in Equation \ref{eq:mult}. While that update yields a straightforward mathematical analysis, two modifications render it more practically useful. First, we use the fact that $1+x \approx \exp x$ for small $x$ to approximate Equation \ref{eq:mult} by:
\begin{equation}\label{eq:exp}
    W+\Delta W = W \odot \exp \big[-\eta\,\sign W \odot\,\sign g\big],
\end{equation}
where $g$ is shorthand for $g(W)$.
This change makes it easier to represent weights in a \emph{logarithmic number system}, since the weights are restricted to integer multiples of $\eta$ in log space. Second, in practice it may be overly stringent to restrict to the 1 bit gradient sign. This leads us to propose:
\begin{equation}\label{eq:madam}
    W+\Delta W = W \odot \exp \left[-\eta\,\sign W \odot\,\clamp_{\eta^*/\eta}\left(\frac{g}{\bar{g}}\right)\right].
\end{equation}
In this expression, $\bar{g}$ denotes the root mean square gradient, which is estimated by a running average over iterations. Each iteration, this is updated by: $$\bar{g}^2 \gets (1-\beta)\,g^2 + \beta\,\bar{g}^2.$$ This idea is borrowed from the Adam and RMSprop optimisers \citep{kingma_adam:_2015, Tieleman2012}. Since $g/\bar{g}$ should typically be $O(\pm1)$ (where $O(\cdot)$ means \textit{on the order of}), it may be viewed as a higher precision version of $\sign g$. The $\mathrm{clamp}_a(\cdot)$ function projects its argument on to the interval $[-a,a]$. This gradient clamping means that a weight can change by no more than a factor of $\exp(\pm\eta \times \eta^*/\eta)=\exp(\pm\eta^*)\approx 1\pm\eta^*$ per iteration, ensuring that the algorithm still makes bounded relative perturbations and respects deep relative trust. Finally, we introduce a weight clamping operation---see the full pseudocode in Algorithm \ref{alg:madam}. As discussed in Section \ref{sec:limitations}, weight clamping \textit{stabilises} and \textit{regularises} the algorithm.

\subsection{\texorpdfstring{$\boldsymbol{B}$}{B}-bit Madam}\label{sec:b-bit-alg}

The multiplicative nature of Madam (Algorithm \ref{alg:madam}) suggests storing synapse strengths in a logarithmic number system, where numbers are represented just by a sign and exponent. To see why this is natural for Madam consider that, since $g/\bar{g}$ is typically $O(\pm1)$, Madam's typical relative perturbation to a synapse is $\exp \pm \eta$. Therefore, in log space, the synapse strengths typically change by $\pm\eta$. This suggests efficiently representing a synapse by its sign and an integer multiple of $\eta$.

In practice, a slightly more fine-grained discretisation is beneficial. We define the \textit{base precision} $\eta_0$ which divides both the learning rate $\eta$ and maximum perturbation strength $\eta^*$. We then round $g/\bar{g}$ appearing in Madam to the nearest multiple of $\eta_0/\eta$. This leads to quantised multiplicative updates:
$$\econst^{0},\quad \econst^{\pm \eta_0},\quad \econst^{\pm 2\eta_0},\quad ...,\quad \econst^{\pm(\eta^*-2\eta_0)},\quad \econst^{\pm(\eta^*-\eta_0)},\quad \econst^{\pm\eta^*}.$$
A good setting in practice is $\eta_0 = 0.001$, $\eta = 0.01$ and $\eta^*=0.08$. Finally, to obtain a $B$-bit weight representation, we must restrict the number of allowed weight levels to $2^B$. The resulting representation is:
$$W = \sign{(W)}
\,\sigma^* \econst^{-k \eta_0} \qquad \text{for } k\in \{ 0,1,...,2^{B}-1\}.$$
The scale parameter $\sigma^*$ is shared by a whole network layer, and may be taken from an existing weight initialisation scheme. A good mental picture is that the weights live on a ladder in log space with rungs spaced apart by $\eta_0$. The Madam update moves the weights up or down the rungs of the ladder.


TITLE: What is the science behind Red light therapy? Infared and low spectrum EM waves sounds fancy to some but is it more than heat and a red light bulb
QUESTION [4 upvotes]: I have some scientific background and I own horses.  So many therapies to alleviate pain in horses are available on the market that are not regulated by any scientific body.   The benefit and the disadvantage of not being regulated by the FDA is that pet owners don't need rigourous scientific studies before we can try a product.  If there is a scietific theory behind a therapy I'm willing to experiment with my horse when I've exhausted all other know forms of therapy.  However, I'm not understanding the red light therapy science and the $1000 price tag for a device.  Is it a snake oil product?  Looks like you can achieve the same infared and low spectrum radiation with a heating pad and a red light bulb.

REPLY [1 votes]: Visible light is one example of an electromagnetic wave, although electromagnetic waves also go beyond the visible spectrum and include infrared and ultraviolet radiation (and, to be comprehensive, radio waves, microwaves, X-rays etc. as well).
The only things you can vary with an electromagnetic wave are its wavelength (which also determines its frequency) and its intensity. So red light is red light is red light - the only difference between a red light bulb and a more expensive source is that the more expensive source may produce a higher intensity of red light. But with visible light higher intensity makes little difference to its biological effect since visible light does not penetrate more than a few millimetres into skin at most. Even  the most intense visible light from a laser, for example, could only penetrate further into the body if it damaged the surface layer of skin, which is obviously a bad idea.
Infrared radiation (beyond the red end of the visible spectrum) is a little different since it heats the outer layers of the skin, and this heat can then be conducted deeper into underlying muscles, tendons etc. But high intensity infrared radiation is also more dangerous than visible light since too much heating can cause skin damage and burns.
This page at Healthline concludes that there is some evidence (although not enough to form a scientific consensus) that red light therapy may promote healing of scars and other skin conditions, but no evidence for claims that it relieves underlying muscular or skeletal pain, boosts the immune system, cures cancer etc. (Note that this Healthline page discussed red light therapy on its own - medical treatments that use light of various wavelengths to activate photosensitive drugs are a separate subject).


TITLE: Find how many values of m satisfy both these condition.
QUESTION [1 upvotes]: If m is an integer and satisfy given two condition , then how many values of m are possible satisfying both condition.
$(1.)     1  \leq m \leq 5000$
$(2.)    [  \sqrt{m} ] = [  \sqrt{m + 125}  ] $
where [ x ] is the greatest integer function .
Find how many values of m satisfy both these condition.
My attempt: By trial and error method I concluded that for any value of m less than $(63)^2$ doesnot satisfy given condition. Hence m=$3936,3970,3971$ where $63^2=3969$.Then again for m=$4906,4097,4098,4099$ so I'm not getting any proper trend. Is there any other way to solve it ?

REPLY [1 votes]: Suppose $m = n^2$ where $n$ is an integer.
In order to satisfy the second condition it's necessary to have $n^2+125<(n+1)^2 \Rightarrow 62\lt n$
That is, for $n\le62$ second condition won't hold so we just have to bother with $63^2=3969\le m\le 5000\le 5041=71^2$
If we replace $n^2$ by $n^2+k$ where $k$ is an integer greater than or equal to $0$ in the above we get $n^2+k+125<(n+1)^2 \Rightarrow 62+k/2 \lt n$
As we know that $n\ge 63$ let's replace $n$ by $63 + i$ where $i$ is defined as $k$
We get $62+k/2 \lt 63 + i \Rightarrow k\lt 2+2i$
Varying $i$ from $0$ to $7$ we see that $k$ can assume respectively $2,4,6,...,16$ values. Therefore the number of m's satisfying both conditions is $2+4+...+16=2(1+2+...+8)=2\frac{8(1+8)}{2}=72$


TITLE: Infinite series
QUESTION [1 upvotes]: Is it possible to compute explicitly or in an approximate way the infinite series:
$$ S(x) = \sum_{k=0}^{+\infty}{\frac{x^k}{k!} (a+k)^{-(n+1)}}$$
where $n \in \mathbb{N}$.

REPLY [1 votes]: I found a closed formulation using Touchard polynomials: $$
\sum_{k=0}^{\infty}\frac{x^k}{k!}(a+k)^{-(n+1)}=a^{-(n+1)}xe^xT_{-n}(x)
$$
too bad there is no definition I found for Touchard polynomials with negative index.
Anyway you can maybe find useful the relation I found using negative binomial expansion: $$
\sum_{k=0}^{\infty}\frac{x^k}{k!}(a+k)^{-(n+1)}=\sum_{k=0}^{\infty}\frac{1}{a^{n+1}}\frac{x^k}{k!}\left (1+\frac{k}{a}\right )^{-(n+1)}=\sum_{k=0}^{\infty}\frac{1}{a^{n+1}}\frac{x^k}{k!}\sum_{j=0}^{\infty}\binom{-(n+1)}{j}\left (\frac{k}{a}\right )^{j}=\sum_{j=0}^{\infty}\binom{-(n+1)}{j}\frac{1}{a^{n+j+1}}\sum_{k=0}^{\infty}\frac{x^k}{k!}{k}^{j}=\sum_{j=0}^{\infty}\binom{-(n+1)}{j}\frac{1}{a^{n+j+1}}T_j(x)e^x
$$
If you have a way to evaluate rapidly the Touchard polynomials the factor inside the sum should decrease as $(x/a)^j$.


TITLE: Partial Fractions of $\frac{1}{x^6+1}$
QUESTION [0 upvotes]: I am trying to solve the following integral :
$$\int \frac{1}{1+x^6}dx$$
I do not want the reader to evaluate the integral, but rather the partial fractions of the integrand : $\frac{1}{1+x^6}$.
This is my workings for factoring the integrand :
$$\begin{align}x^6+ 1 &= (x^2+1)(x^4-x^2+1)\\ &= (x^2+1)(x^4+2x^2+1-3x^2)\\ &=(x^2+1)((x^2+1)^2-(\sqrt{3}x)^2)\\ &=(x^2+1)(x^2-\sqrt{3}x+1)(x^2+\sqrt{3}x+1)\end{align}$$
Therefore : $$\int\frac{1}{x^6+1}dx=\int\frac{a_1x+a_2}{x^2+1}dx+\int\frac{a_3x+a_4}{x^2-\sqrt{3}x+1}dx+\int\frac{a_5x+a_6}{x^2+\sqrt{3}x+1}dx$$
From here, I am not sure how to proceed to solve for $a_n$.

REPLY [1 votes]: Here is a trick that makes calculations  faster. We have, $$\frac1{x^6+1}=\frac{a_1 x + a_2}{x^2+1} + \frac{a_3 x + a_4}{x^2 - \sqrt{3} x + 1} + \frac{a_5 x + a_6}{x^2 + \sqrt{3} x + 1} $$
Note that $f(x)=\dfrac{1}{x^6+1}$ is an even function, Hence on the right side of above equation, changing $x$ to $-x$ doesn't change the expression. Hence $a_1=0\;, \;a_3=-a_5\;$ and $\;a_4=a_6$. Therefore number of coefficients will be reduced to three.
$$\frac{1}{x^6+1}=\frac{A}{x^2+1}+\frac{Bx+C}{x^2-\sqrt3x+1}+\frac{-Bx+C}{x^2+\sqrt3x+1}$$
From here you can expand RHS or alternatively plug in three small values of $x$ (like $x=0,1,-1$) in each side and solve system of equations in three variables $A,B,C$.