text:
\begin{document}

\setlength{\abovedisplayskip}{2.5pt}
\setlength{\belowdisplayskip}{2.5pt}

\maketitle

\begin{abstract}

Designing deep neural networks is an art that often involves an expensive search over candidate architectures. To overcome this for recurrent neural nets (RNNs), we establish a connection between the hidden state dynamics in an RNN and gradient descent (GD). We then integrate momentum into this framework and propose a new family of RNNs, called {\em MomentumRNNs}. We theoretically prove and numerically demonstrate that MomentumRNNs alleviate the vanishing gradient issue in training RNNs. We study the momentum long-short term memory (MomentumLSTM) and verify its advantages in convergence speed and accuracy over its LSTM counterpart across a variety of benchmarks. We also demonstrate that MomentumRNN is applicable to many types of recurrent cells, including those in the state-of-the-art orthogonal RNNs. Finally, we show that other advanced momentum-based optimization methods, such as Adam and Nesterov accelerated gradients with a restart, can be easily incorporated into the MomentumRNN framework for designing new recurrent cells with even better performance.
\end{abstract}

\section{Introduction}\label{sec:intro}

Mathematically principled recurrent neural nets (RNNs) facilitate the network design process and reduce the cost of searching over many candidate architectures. A particular advancement in RNNs is the long short-term memory (LSTM) model~\cite{hochreiter1997long} which has achieved state-of-the-art results in many applications, including speech recognition \cite{fernandez2007sequence}, acoustic modeling \cite{sak2014long,qu2017syllable}, and language modeling \cite{palangi2016deep}. There have been many efforts in improving LSTM: \cite{gers1999learning} introduces a forget gate into the original LSTM cell, which can forget information selectively; \cite{gers2000recurrent} further adds peephole connections to the LSTM cell to inspect its current internal states\cite{gers2001lstm}; to reduce the computational cost, a gated recurrent unit (GRU) \cite{cho2014learning} uses a single update gate to replace the forget and input gates in LSTM. Phased LSTM \cite{neil2016phased} adds a new time gate to the LSTM cell and achieves faster convergence than the regular LSTM on learning long sequences. In addition, \cite{rahman2016new} and \cite{pulver2017lstm} introduce a biological cell state and working memory into LSTM, respectively.

Nevertheless, most of RNNs, including LSTMs,

are biologically informed or even ad-hoc instead of being guided by mathematical principles.

\subsection{Recap on RNNs and LSTM}
Recurrent cells are the building blocks of RNNs. A recurrent cell employs a cyclic connection to update the current hidden state ($\vh_t$) using the past hidden state ($\vh_{t-1}$) and the current input data ($\vx_t$) \cite{elman1990finding}; the dependence of $\vh_t$ on $\vh_{t-1}$ and  $\vx_t$ in a recurrent cell can be written as
\begin{equation}\label{eq:RNN:Cell}

\vh_t = \sigma(\Ub\vh_{t-1} + \Wb\vx_t + \vb),\ \vx_t \in \RR^d,\ \mbox{and}\  \vh_{t-1}, \vh_t \in \RR^h,\ \ t=1, 2, \cdots, T,
\end{equation}
where  $\Ub \in \RR^{h\times h}, \Wb \in \RR^{h\times d}$, and $\vb\in \RR^h$ are trainable parameters; $\sigma(\cdot)$ is a nonlinear activation function, e.g., sigmoid or hyperbolic tangent. Error backpropagation through time is used to train RNN, but it tends to result in exploding or vanishing gradients \cite{bengio1994learning}. Thus RNNs may fail to learn

long term dependencies. Several approaches exist to improve RNNs' performance, including enforcing unitary weight matrices
\cite{arjovsky2016unitary,wisdom2016full,jing2017tunable,vorontsov2017orthogonality,mhammedi2017efficient,pmlr-v80-helfrich18a}, leveraging LSTM cells, and others \cite{li2018independently,kusupati2018fastgrnn}.

LSTM cells augment the recurrent cell with ``gates'' \cite{hochreiter1997long} and can be formulated as
\begin{equation}\label{eq:LSTM:cell}
\begin{aligned}

\vi_t &= \sigma(\Ub_{ih}\vh_{t-1} + \Wb_{ix}\vx_t + \vb_i),\ \ &(\vi_t: \mbox{input gate})\\
\widetilde{\vc}_t &= \tanh{(\Ub_{\widetilde{c}h}\vh_{t-1} + \Wb_{\widetilde{c}x}\bx_t + \vb_{\widetilde{c}})},\ \ &(\widetilde{\vc}_t: \mbox{cell input})\\
\vc_t &= \vc_{t-1} + \vi_t \odot \widetilde{\vc}_t,\ \ &(\vc_t: \mbox{cell state})\\
\vo_t &= \sigma(\Ub_{oh}\vh_{t-1} + \Wb_{ox}\vx_t + \vb_o ),\ \ &(\vo_t: \mbox{output gate})\\
\vh_t &= \vo_t\odot \tanh{\vc_t},\ \ &(\vh_t: \mbox{hidden state})
\end{aligned}
\end{equation}
where $\Ub_*\in \RR^{h\times h}$, $\Wb_*\in \RR^{h\times d}$, and $\vb_* \in \RR^h$ are learnable parameters, and $\odot$ denotes the Hadamard product. The input gate decides what new information to be stored in the cell state, and the output gate decides what information to output based on the cell state value. The gating mechanism in LSTMs can lead to the issue of saturation \cite{van2018unreasonable,chandar2019towards}.

\subsection{Our Contributions}

In this paper, we develop a gradient descent (GD) analogy of the recurrent cell. In particular, the hidden state update in a recurrent cell is associated with a gradient descent step towards the optimal representation of the hidden state. We then
propose to integrate momentum that used for accelerating gradient dynamics into the recurrent cell, which results in the momentum cell. At the core of the momentum cell is the use of momentum to accelerate the hidden state learning in RNNs. The architectures of the standard recurrent cell and our momentum cell are illustrated in Fig.~\ref{fig:momentumrnn-vs-rnn}. We provide the design principle and detailed derivation of the momentum cell in Sections~\ref{subsec:momentum-cell} and \ref{sec:Adam:cell}. We call the RNN that consists of momentum cells the MomentumRNN. The major advantages of MomentumRNN are fourfold:

\begin{itemize}[leftmargin=*]
\item
MomentumRNN can alleviate the vanishing gradient problem in training RNN.
\item MomentumRNN accelerates training and improves the test accuracy of the baseline RNN.
\item MomentumRNN is universally applicable to many existing RNNs. It can be easily implemented by changing a few lines of the baseline RNN code.
\item MomentumRNN is principled with theoretical guarantees provided by the momentum-accelerated dynamical system for optimization and sampling. The design principle can be generalized to other advanced momentum-based optimization methods, including Adam~\cite{kingma2014adam} and Nesterov accelerated gradients with a restart~\cite{nesterov1983method,wang2020scheduled}.

\end{itemize}
\begin{figure}[t]

\centering
\includegraphics[width=1.0\linewidth]{figures/monet.pdf}
\vskip -0.35cm
\caption{Illustration of the recurrent cell (left), Momentum/NAG cell (middle), and Adam/RMSProp cell (right). We draw a connection between the dynamics of hidden states in the recurrent cell and GD. We then introduce momentum to recurrent cell as an analogy of the momentum accelerated GD.

}
\label{fig:momentumrnn-vs-rnn}
\end{figure}

\subsection{Related Work}
{\bf Dynamical system viewpoint of RNNs.}

Leveraging the theory of dynamical system to improve RNNs has been an interesting research direction: \cite{laurent2016recurrent} proposes a gated RNN, which is principled by non-chaotical dynamical systems and achieves comparable performance to GRUs and LSTMs.

\cite{talathi2015improving} proposes a weight initialization strategy inspired by dynamical system theory, which helps the training of RNNs with ReLU nonlinearity. Other RNN algorithms  derived from the dynamical system theories include \cite{
niu2019recurrent,chang2019antisymmetricrnn,chen2019symplectic,kag2019rnns}. Our work is the first that directly integrates momentum into an RNN to accelerate the underlying dynamics and improve the model's performance.

{\bf Momentum in Optimization and Sampling.} Momentum has been a popular technique for accelerating (stochastic) gradient-based optimization \cite{polyak1964some,goh2017momentum,sutskever2013importance,kingma2014adam,bengio2013advances,paszke2019pytorch} and sampling algorithms \cite{duane1987hybrid,neal2011mcmc}

A particularly interesting momentum is the iteration-dependent one in NAG \cite{nesterov1983method,nemirovskii1985optimal,beck2009fast}, which has a significantly better convergence rate than constant momentum for convex optimization. The stochastic gradient NAG that employs a scheduled restart can also be used to accelerate DNN training with better accuracy and faster convergence \cite{wang2020scheduled}.

{\bf Momentum in DNNs.}
Momentum has also been used in designing DNN architectures. \cite{he2019momentum} develops momentum contrast as a way of building large and consistent dictionaries for unsupervised learning with contrastive loss. At the core of this approach is a momentum-based moving average of the queue encoder. Many DNN-based algorithms for sparse coding are designed by unfolding the classical optimization algorithms, e.g., FISTA \cite{beck2009fast}, in which momentum can be used in the underpinning optimizer~\cite{szlam2011structured,chalasani2013fast,mccann2017convolutional,kamilov2016learning,moreau2017understanding}.

\subsection{Notation}
We denote scalars by lower or upper case letters; vectors and matrices by lower and upper case bold face letters, respectively. For a vector $\vx = (x_1, \cdots, x_d)^T\in \mathbb{R}^d$, we use $\|\vx\| = {(\sum_{i=1}^d |x_i|^2)^{1/2}}$ to denote its $\ell_2$ norm.

For a matrix $\Ab$, we use $\Ab^{\rm T}$ (${\rm T}$ in roman type) and $\Ab^{-1}$ to denote its transpose and inverse, respectively. Also, we denote the spectral norm of $\Ab$ as $\|\Ab\|$.

We denote the $d$-dimensional standard Gaussian as $\mathcal{N}(\mathbf{0}, \mathbf{I}_{d\times d})$, where $\mathbf{0}$ is the $d$-dimensional zero-vector and $\mathbf{I}_{d\times d}$ is

an identity matrix. For a function $\phi(\vx): \mathbb{R}^d \rightarrow \mathbb{R}$, we denote $\phi^{-1}(\vx)$ as its inverse and $\nabla \phi(\vx)$ as its gradient.

\section{Momentum RNNs}\label{sec:momentum:nets}

\subsection{Background: Momentum Acceleration for Gradient Based Optimization and Sampling}\label{subsec:review:momentum}

Momentum has been successfully used to accelerate the gradient-based algorithms for optimization and sampling. In optimization, we aim to find a stationary point of a given function $f(\vx), \vx\in \RR^d$. Starting from $\vx_0\in \RR^d$, GD iterates as $\vx_t = \vx_{t-1} - s \nabla f(\vx_t)$ with $s>0$ being the step size. This can be significantly accelerated by using the
momentum~\cite{sutskever2013importance}, which results in
\begin{equation}
\label{eq:MGD}
\vp_0=\vx_0;\;
\vp_{t} = \mu \vp_{t-1} + s\nabla f(\vx_{t});\;
\vx_{t} = \vx_{t-1} - \vp_{t},\ \ t\geq 1,
\end{equation}
where $\mu \geq 0$ is the momentum constant. In sampling, Langevin Monte Carlo (LMC) \cite{coffey2012langevin} is used to sample from the distribution $\pi \propto \exp\{-f(\vx)\}$, where $\exp\{-f(\vx)\}$ is the probability distribution function. The update at each iteration is given by
\begin{equation}
\label{eq:LMC}
\vx_t = \vx_{t-1} - s\nabla f(\vx_t) + \sqrt{2s}\boldsymbol{\epsilon}_t,
\ s\geq 0,\ t\geq 1,\ \boldsymbol{\epsilon}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_{d\times d}).
\end{equation}
We can also use momentum to accelerate LMC, which results in the following Hamiltonian Monte Carlo (HMC) update \cite{coffey2012langevin}:

\begin{equation}
\label{eq:HMC}
\vp_0=\vx_0;\; \vp_{t} = \vp_{t-1} - \gamma s \vp_{t-1} - s \eta\nabla f(\vx_{t-1}) + \sqrt{2\gamma s\eta} \boldsymbol{\epsilon}_t;\;
\vx_{t} = \vx_{t-1} + s\vp_{t},\ \ t\geq 1,
\end{equation}
where $\boldsymbol{\epsilon}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_{d\times d})$ while $\gamma, \eta, s >0$ are the friction parameter, inverse mass, and step size, resp.

\subsection{
Gradient Descent Analogy for RNN and MomentumRNN}\label{subsec:momentum-cell}
Now, we are going to establish a connection between RNN and GD, and further leverage momentum to improve RNNs. Let $\widetilde{\Wb} = [\Wb, \vb]$ and $\widetilde{\vx}_t = [\vx_t, 1]^T$ in \eqref{eq:RNN:Cell}, then we have $\vh_t = \sigma(\Ub\vh_{t-1} + \widetilde{\Wb}\widetilde{\vx}_t)$. For the ease of notation, without ambiguity we denote $\Wb := \widetilde{\Wb}$ and $\vx_t := \widetilde{\vx}_t$. Then the recurrent cell can be reformulated as
\begin{equation}
\label{eq:basic:cell}
\vh_t = \sigma(\Ub \vh_{t-1} + \Wb \vx_t).
\end{equation}
Moreover, let $\phi(\cdot) := \sigma(\Ub(\cdot))$ and $\vu_t := \Ub^{-1}\Wb\vx_t$, we can rewrite \eqref{eq:basic:cell} as
\begin{equation}
\label{eq:recurent-cell:PGD}
\vh_t = \phi(\vh_{t-1} + \vu_t).
\end{equation}
If we regard $-\vu_t$ as the ``gradient'' at the $t$-th iteration, then we can consider \eqref{eq:recurent-cell:PGD} as the dynamical system which updates the hidden state by the gradient and then transforms the updated hidden state by the nonlinear activation function $\phi$. We propose the following accelerated dynamical system to accelerate the dynamics of  \eqref{eq:recurent-cell:PGD}, which is principled by the accelerated gradient descent theory (see subsection~\ref{subsec:review:momentum}):
\begin{align}\label{eq:momentum:PGD}
\vp_{t} = \mu\vp_{t-1} - s \vu_{t};\ \ \vh_{t} =\phi(\vh_{t-1} - \vp_{t}),

\end{align}
where $\mu \geq 0, s >0$ are two hyperparameters, which are the analogies of the momentum coefficient and step size in the momentum-accelerated GD,
respectively. Let $\bv_t := -\Ub\vp_t$, we arrive at the following dynamical system:
\begin{align}\label{eq:momentum:cell}
\bv_{t} = \mu\bv_{t-1} + s \Wb\vx_{t};\ \ \vh_{t} = \sigma(\Ub\vh_{t-1} + \bv_{t}).

\end{align}
The architecture of the momentum cell that corresponds to the dynamical system \eqref{eq:momentum:cell} is plotted in Fig.~\ref{fig:momentumrnn-vs-rnn} (middle). Compared with the recurrent cell, the momentum cell introduces an auxiliary momentum state in each update and scales the dynamical system with two positive hyperparameters

$\mu$ and $s$.

\begin{remark}
\label{rm:different-parameterization}
Different parameterizations of \eqref{eq:momentum:PGD} can result in different momentum cell architectures. For instance, if we let $\bv_t = -\vp_t$, we end up with the following dynamical system:
\begin{align}\label{eq:momentum:cell2}
\bv_{t} = \mu\bv_{t-1} + s \widehat{\Wb}\vx_{t};\ \ \vh_{t} = \sigma(\Ub\vh_{t-1} + \Ub\bv_{t}),

\end{align}
where $\widehat{\Wb} := \Ub^{-1}\Wb$ is the trainable weight matrix. Even though \eqref{eq:momentum:cell} and \eqref{eq:momentum:cell2} are mathematically equivalent, the training procedure might cause the MomentumRNNs that are derived from different parameterizations to have different performances.
\end{remark}

\begin{remark}
We put the nonlinear activation in the second equation of \eqref{eq:momentum:PGD} to ensure that the value of $\vh_t$ is in the same range as the original recurrent cell.

\end{remark}

\begin{remark}
The derivation above also applies to the dynamical systems in the LSTM cells, and we can design the MomentumLSTM in the same way as designing the MomentumRNN.
\end{remark}

\subsection{Analysis of the Vanishing Gradient Issue: Momentum Cell vs. Recurrent Cell}\label{subsec:gradient}
Let $\vh_T$ and $\vh_t$ be the state vectors at the time step $T$ and $t$, respectively, and we suppose $T\gg t$. Furthermore, assume that $\mathcal{L}$ is the objective to minimize, then
\begin{equation}
\label{eq:gradient:rnn}
{\small \frac{\partial \mathcal L}{\partial \vh_t} = \frac{\partial \mathcal{L}}{\partial\vh_T}\cdot\frac{\partial \vh_T}{\partial \vh_t} = \frac{\partial \mathcal L}{\partial \vh_T}\cdot \prod_{k=t}^{T-1}\frac{\partial \vh_{k+1}}{\partial \vh_k} = \frac{\partial \mathcal L}{\partial \vh_T}\cdot \prod_{k=t}^{T-1}(\Db_k\Ub^{\rm T}),}
\end{equation}
where {\small $\Ub^{\rm T}$} is the transpose of $\Ub$ and {\small $\Db_k = {\rm diag}(\sigma'(\Ub\vh_k + \Wb\vx_{k+1}))$} is a diagonal matrix with $\sigma'(\Ub\vh_k + \Wb\vx_{k+1})$ being its diagonal entries.

{\small $\|\prod_{k=t}^{T-1}(\Db_k\Ub^{\rm T})\|_2$} tends to either vanish or explode \cite{bengio1994learning}. We can use regularization or gradient clipping to mitigate the exploding gradient, leaving vanishing gradient as the major obstacle to training RNN to learn long-term dependency~\cite{pascanu2013difficulty}.
We can rewrite \eqref{eq:momentum:cell} as

\vskip -0.5cm
\begin{equation}
\label{eq:momentum:cell:one:eq}
{\small \vh_t = \sigma\left(\Ub(\vh_{t-1} - \mu\vh_{t-2}) + \mu\sigma^{-1}(\vh_{t-1}) + s\Wb\vx_t\right),}
\end{equation}
where $\sigma^{-1}(\cdot)$ is the inverse function of $\sigma(\cdot)$. We compute $\partial \mathcal L/\partial \vh_t$ as follows
\begin{equation}
\label{eq:gradient:momentum}
{\small \frac{\partial \mathcal L}{\partial \vh_t} = \frac{\partial \mathcal{L}}{\partial\vh_T}\cdot\frac{\partial \vh_T}{\partial \vh_t} = \frac{\partial \mathcal L}{\partial \vh_T}\cdot \prod_{k=t}^{T-1}\frac{\partial \vh_{k+1}}{\partial \vh_k} = \frac{\partial \mathcal L}{\partial \vh_T}\cdot \prod_{k=t}^{T-1}\widehat{\Db}_k[\Ub^{\rm T} + \mu\boldsymbol\Sigma_k],}
\end{equation}
where {\small $\widehat{\Db}_k = {\rm diag}(\sigma'(\Ub(\vh_{k} - \mu\vh_{k-1}) + \mu\sigma^{-1}(\vh_{k}) + s\Wb\vx_{k+1} ))$} and {\small $\boldsymbol\Sigma = {\rm diag}((\sigma^{-1})'(\vh_k))$}. For mostly used $\sigma$, e.g., sigmoid and tanh,  $(\sigma^{-1}(\cdot))' > 1$

and $\mu\boldsymbol\Sigma_k$ dominates $\Ub^{\rm T}$.\footnote{In the vanishing gradient scenario, $\|\Ub\|_2$ is small; also it can be controlled by regularizing the loss function.} Therefore, with an appropriate choice of $\mu$, the momentum cell can alleviate vanishing gradient and accelerate training.

We empirically corroborate that momentum cells can alleviate vanishing gradients by training a MomentumRNN and its corresponding RNN on the PMNIST classification task and plot $\|\partial \mathcal L/\partial \vh_t\|_{2}$ for each time step $t$. Figure~\ref{fig:stability} confirms that unlike in RNN, the gradients in MomentumRNN do not vanish. More details on this experiment are provided in the Appendix~\ref{sec:exp:details}.

\begin{figure}[t!]

\centering
\includegraphics[width=0.99\linewidth]{figures/stability.pdf}
\vskip -0.2cm
\caption{$\ell_2$ norm of the gradients of the loss $\mathcal{L}$ w.r.t. the state vector $\vh_t$ at each time step $t$ for RNN (left) and MomentumRNN (right). MomentumRNN does not suffer from vanishing gradients.}
\label{fig:stability}
\end{figure}

\subsection{Beyond MomentumRNN: NAG and Adam Principled Recurrent Neural Nets}\label{sec:Adam:cell}
There are several other advanced formalisms of momentum existing in optimization, which can be leveraged for RNN architecture design. In this subsection, we present two additional variants of MomentumRNN that are derived from the Nesterov accelerated gradient (NAG)-style momentum with restart \cite{nesterov1983method,wang2020scheduled} and Adam \cite{kingma2014adam}.

{\bf NAG Principled RNNs.} The momentum-accelerated GD can be further accelerated by  replacing the constant momentum coefficient $\mu$ in~\eqref{eq:momentum:cell} with the NAG-style momentum, i.e. setting $\mu$ to $(t-1)/(t+2)$ at the $t$-th iteration. Furthermore, we can accelerate NAG by resetting the momentum to 0 after every $F$ iterations, i.e. $\mu = (t \mod F)/((t\mod F) + 3)$, which is the NAG-style momentum with a scheduled restart of the appropriately selected frequency $F$ \cite{wang2020scheduled}. For convex optimization, NAG has a convergence rate $O(1/t^2)$, which is significantly faster than GD or GD with constant momentum whose convergence rate is $O(1/t)$. Scheduled restart not only accelerates NAG to a linear convergence rate $O(\alpha^{-t}) (0<\alpha <1)$ under mild extra assumptions but also stabilizes the NAG iteration \cite{wang2020scheduled}.
We call the MomentumRNN with the NAG-style momentum and scheduled restart momentum the NAG-based RNN and the scheduled restart RNN (SRRNN), respectively.

{\bf Adam Principled RNNs.} Adam~\cite{kingma2014adam} leverages the moving average of historical gradients and entry-wise squared gradients to accelerate the stochastic gradient dynamics. We use Adam to accelerate \eqref{eq:recurent-cell:PGD} and end up with the following iteration
\begin{equation}
\label{eq:Adam:cell}
{\small \vp_t = \mu \vp_{t-1} + (1-\mu)\vu_t;\ \vm_t = \beta\vm_{t-1} + (1-\beta)\vu_t\odot\vu_t;\ \vh_t = \phi(\vh_{t-1} - s \frac{\vp_t}{\sqrt{\vr_t} +\epsilon}),}

\end{equation}
where $\mu, s, \beta > 0$ are hyperparameters, $\epsilon$ is a small constant and chosen to be $10^{-8}$ by default, and $\odot$/$\sqrt{\cdot}$ denotes the entrywise product/square root\footnote{In contrast to Adam, we do not normalize $\vp_t$ and $\vm_t$ since they can be absorbed in the weight matrices.}. Again, let $\bv_t = -\Ub\vp_t$, we rewrite \eqref{eq:Adam:cell} as follows
\begin{equation}
\label{eq:Adam:momentum:cell}
\hskip -0.3cm{\footnotesize \bv_t = \mu \bv_{t-1} + (1-\mu)\Wb\vx_t;\ \vm_t = \beta\vm_{t-1} + (1-\beta)\vu_t\odot\vu_t;\ \vh_t = \sigma(\Ub\vh_{t-1} + s \frac{\bv_t}{\sqrt{\vm_t} +\epsilon}).} \nonumber

\end{equation}
As before, here $\vu_t := \Ub^{-1}\Wb\vx_t$. Computing $\Ub^{-1}$ is expensive. Our experiments suggest that replacing $\vu_t\odot\vu_t$ by $\Wb\vx_t \odot \Wb\vx_t$ is sufficient and more efficient to compute. In our implementation, we also
relax $\bv_t = \mu \bv_{t-1} + (1-\mu)\Wb\vx_t$

to $\bv_t = \mu \bv_{t-1} + s\Wb\vx_t$  that follows the momentum in the MomentumRNN \eqref{eq:momentum:cell}
for better performance. Therefore, we propose the \emph{AdamRNN} that is given by
\begin{equation}
\label{eq:Adam:momentum:cell:final}
{\small \bv_t = \mu \bv_{t-1} + s\Wb\vx_t;\ \ \vm_t = \beta\vm_{t-1} + (1-\beta)(\Wb\vx_t\odot\Wb\vx_t);\ \ \vh_t = \sigma(\Ub\vh_{t-1} + \frac{\bv_t}{\sqrt{\vm_t} +\epsilon}).}

\end{equation}

In AdamRNN, if $\mu$ is set to 0, we achieve another new RNN, which obeys the RMSProp gradient update rule~\cite{Tieleman2012}. We call this new model the \emph{RMSPropRNN}.
\begin{comment}
whose dynamics can be formulated as follows
\begin{equation}
\label{eq:RMSProp:momentum:cell:final}
\bv_t = s\Wb\vx_t;\ \ \vm_t = \beta\vm_{t-1} + (1-\beta)(\Wb\vx_t\odot\Wb\vx_t);\ \ \vh_t = \sigma(\Ub\vh_{t-1} + \frac{\bv_t}{\sqrt{\vm_t} +\epsilon}).
\end{equation}
\end{comment}
\begin{remark}
Both AdamRNN and RMSPropRNN can also be derived by letting $\bv_t = -\vp_t$ and $\widehat{\Wb} := \Ub^{-1}\Wb$ as in Remark~\ref{rm:different-parameterization}. This parameterization yields the following formulation for AdamRNN
\begin{equation}
\label{eq:Adam:momentum:cell:final-2}
{\small \bv_t = \mu \bv_{t-1} + s\widehat{\Wb}\vx_t;\ \ \vm_t = \beta\vm_{t-1} + (1-\beta)(\widehat{\Wb}\vx_t\odot\widehat{\Wb}\vx_t);\ \ \vh_t = \sigma(\Ub\vh_{t-1} + \frac{\Ub\bv_t}{\sqrt{\vm_t} +\epsilon}).} \nonumber

\end{equation}
Here, we simply need to learn

$\widehat{\Wb}$ and $\Ub$ without any relaxation. In contrast, we relaxed $\Ub^{-1}$ to an identity matrix in~\eqref{eq:Adam:momentum:cell:final}. Our experiments suggest that both parameterizations

yield similar results.

\end{remark}

\section{Experimental Results}\label{sec:experiments}

In this section, we evaluate the effectiveness of our momentum approach in designing RNNs in terms of convergence speed and accuracy. We compare the performance of the MomentumLSTM with the baseline LSTM~\cite{hochreiter1997long} in the following tasks: 1) the object classification task on pixel-permuted MNIST~\cite{le2015simple}, 2) the speech prediction task on the TIMIT dataset~\cite{arjovsky2016unitary, pmlr-v80-helfrich18a, wisdom2016full, mhammedi2017efficient, pmlr-v48-henaff16}, 3) the celebrated copying and adding tasks \cite{hochreiter1997long, arjovsky2016unitary}, and 4) the language modeling task on the Penn TreeBank (PTB) dataset~\cite{mikolov2010recurrent}. These four tasks are among standard benchmarks to measure the performance of RNNs and their ability to handle long-term dependencies. Also, these tasks cover different data modalities -- image, speech, and text data -- as well as a variety of model sizes, ranging from thousands to millions of parameters with one (MNIST and TIMIT tasks) or multiple (PTB task) recurrent cells in concatenation. Our experimental results confirm that MomentumLSTM converges faster and yields better test accuracy than the baseline LSTM across tasks and settings. We also discuss the AdamLSTM, RMSPropLSTM, and scheduled restart LSTM (SRLSTM) and show their advantage over MomentumLSTM in specific tasks. Computation time and memory cost of our models versus the baseline LSTM are provided in Appendix~\ref{sec:appendix:Time-and-Memory}. All of our results are averaged over 5 runs with different seeds.

We include details on the models, datasets, training procedure, and hyperparameters used in our experiments in Appendix~\ref{sec:exp:details}. For MNIST and TIMIT experiments, we use the baseline codebase provided by~\cite{dtrivgithub}. For PTB experiments, we use the baseline codebase provided by~\cite{ptbgithub}.

\subsection{Pixel-by-Pixel MNIST}
\label{sec:mnist-exp}
In this task, we classify image samples of hand-written digits from the MNIST dataset~\cite{lecun2010mnist} into one of the ten classes. Following the implementation of~\cite{le2015simple}, we flatten the image of original size 28 $\times$ 28 pixels and feed it into the model as a sequence of length 784. In the unpermuted task (MNIST), the sequence of pixels is processed row-by-row. In the permuted task (PMNIST), a fixed permutation is selected at the beginning of the experiments and then applied to both training and test sequences. We summarize the results in Table~\ref{tab:mnist}. Our experiments show that \emph{MomentumLSTM achieves better test accuracy than the baseline LSTM in both MNIST and PMNIST digit classification tasks} using different numbers of hidden units (i.e. $N=128, 256$). Especially, the improvement is significant on the PMNIST task, which is designed to test the performance of RNNs in the context of long-term memory. Furthermore, we notice that \emph{MomentumLSTM converges faster than LSTM} in all settings. Figure~\ref{fig:loss-vs-iters} (left two panels) corroborates this observation when using $N=256$ hidden units.

\begin{table}[!t]
\vspace{-0.07in}
    \caption{Best test accuracy at the MNIST and PMNIST tasks (\%).

    We use the baseline results reported in \cite{pmlr-v80-helfrich18a},~\cite{wisdom2016full},~\cite{vorontsov2017orthogonality}. All of our proposed models outperform the baseline LSTM. Among the models using $N=256$ hidden units, RMSPropLSTM yields the best results in both tasks.}
\vspace{-0.1in}
\label{tab:mnist}
\begin{center}

\begin{footnotesize}
\begin{sc}
\begin{tabular}{lclcc}
    \toprule
    Model & n & \# params & MNIST & PMNIST \\
    \midrule
    \midrule
    LSTM & $128$ & $\approx 68K$ & $98.70$\cite{pmlr-v80-helfrich18a},$97.30$  \cite{vorontsov2017orthogonality} & $92.00$  \cite{pmlr-v80-helfrich18a},$92.62$ \cite{vorontsov2017orthogonality} \\
    LSTM & $256$ & $\approx 270K$ & $98.90$ \cite{pmlr-v80-helfrich18a}, $98.50$ \cite{wisdom2016full} & $92.29$  \cite{pmlr-v80-helfrich18a}, $92.10$  \cite{wisdom2016full} \\
    \midrule
    MomentumLSTM & $128$ & $\approx 68K$ & $\bf{99.04 \pm 0.04}$ & $\bf{93.40 \pm 0.25}$\\
    MomentumLSTM & $256$ & $\approx 270K$ & $\bf{99.08 \pm 0.05}$ & $\bf{94.72 \pm 0.16}$\\
    \midrule
    AdamLSTM & $256$ & $\approx 270K$ & $99.09 \pm 0.03$ & $95.05 \pm 0.37$\\
    RMSPropLSTM & $256$ & $\approx 270K$ & $\bf{99.15 \pm 0.06}$ & $\bf{95.38 \pm 0.19}$\\
    SRLSTM & $256$ & $\approx 270K$ & $ 99.01 \pm 0.07 $ & $93.82 \pm 1.85$\\
    \bottomrule
\end{tabular}
\end{sc}
\end{footnotesize}

\end{center}

\end{table}

\subsection{TIMIT Speech Dataset}
\label{sec:timit-exp}
We study how MomentumLSTM performs on audio data with speech prediction experiments on the TIMIT speech dataset~\cite{garofolo1993timit}, which is a collection of real-world speech recordings. As first proposed by~\cite{wisdom2016full}, the recordings are downsampled to 8kHz and then transformed into log-magnitudes via a short-time Fourier transform (STFT). The task accounts for predicting the next log-magnitude given the previous ones. We use the standard train/validation/test separation in~\cite{wisdom2016full, lezcano2019cheap, casado2019trivializations}, thereby having 3640 utterances for the training set with a validation set of size 192 and a test set of size 400.

The results for this TIMIT speech prediction are shown in Table~\ref{tab:timit}. Results are reported on the test set using the model parameters
that yield the best validation loss. Again, we see the advantage of MomentumLSTM over the baseline LSTM. In particular, MomentumLSTM yields much better prediction accuracy and faster convergence speed compared to
LSTM. Figure~\ref{fig:loss-vs-iters} (right two panels)

shows the convergence of MomentumLSTM vs. LSTM when using $N=158$ hidden units.

\begin{figure}[!t]
\centering
\includegraphics[width=1.0\linewidth]{figures/train_loss_lstm.pdf}
\vskip -0.2cm
\caption{Train and test loss  of MomentumLSTM (blue), AdamLSTM (green), RMSPropLSTM (orange), SRLSTM (cyan), and LSTM (red) for MNIST (left two panels) and TIMIT (right two panels) tasks. MomentumLSTM converges faster than LSTM in both tasks. For MNIST, AdamLSTM and RMSPropLSTM converge fastest. For TIMIT, MomentumLSTM and SRLSTM converge fastest.}
\label{fig:loss-vs-iters}
\end{figure}

{\bf Remark:} The TIMIT dataset is not open for
public, so we do not
have access to the preprocessed
data from previous papers. We followed the data preprocessing in~\cite{wisdom2016full, lezcano2019cheap, casado2019trivializations} to generate the preprocessed data for our experiments and did our best to reproduce the baseline results.
In Table~\ref{tab:timit} and~\ref{tab:timit-dtriv}, we include both our reproduced results and the ones reported from previous works.

\begin{table}[!t]
\vspace{-0.1in}
    \caption{Test and validation MSEs at the end of the epoch with the lowest validation
MSE for the TIMIT task. All of our proposed models outperform the baseline LSTM. Among models using $N=158$ hidden units, SRLSTM performs the best.}

\vspace{-0.2in}
\label{tab:timit}
\begin{center}
\begin{footnotesize}

\begin{sc}
\begin{tabular}{lclcc}
    \toprule
    Model & n & \# params & Val. MSE & Test MSE \\
    \midrule
    \midrule
    LSTM & $84$ & $\approx 83K$ & $14.87 \pm 0.15$ ($15.42$ \cite{pmlr-v80-helfrich18a,lezcano2019cheap}) & $14.94 \pm 0.15$ ($14.30$ \cite{pmlr-v80-helfrich18a,lezcano2019cheap}) \\
    LSTM & $120$ & $\approx 135K$ & $11.77 \pm 0.14$ ($13.93$ \cite{pmlr-v80-helfrich18a,lezcano2019cheap}) & $11.83 \pm 0.12$ ($12.95$ \cite{pmlr-v80-helfrich18a,lezcano2019cheap}) \\
    LSTM & $158$ & $\approx 200K$ & $9.33 \pm 0.14$ ($13.66$ \cite{pmlr-v80-helfrich18a,lezcano2019cheap}) & $9.37 \pm 0.14$ ($12.62$ \cite{pmlr-v80-helfrich18a,lezcano2019cheap}) \\
    \midrule
    MomentumLSTM & $84$ & $\approx 83K$ & $\bf{10.90 \pm 0.19}$ & $\bf{10.98 \pm 0.18}$\\
    MomentumLSTM & $120$ & $\approx 135K$ & $\bf{8.00 \pm 0.30}$ & $\bf{8.04 \pm 0.30}$\\
    MomentumLSTM & $158$ & $\approx 200K$ & $\bf{5.86 \pm 0.14}$ & $\bf{5.87 \pm 0.15}$\\
    \midrule
    AdamLSTM & $158$ & $\approx 200K$ & $8.66 \pm 0.15$ & $8.69 \pm 0.14$\\
    RMSPropLSTM & $158$ & $\approx 200K$ & $9.13 \pm 0.33$ & $9.17 \pm 0.33$\\
    SRLSTM & $158$ & $\approx 200K$ & $\bf{5.81 \pm 0.10}$ & $\bf{5.83 \pm 0.10}$\\
    \bottomrule
\end{tabular}
\end{sc}

\end{footnotesize}
\end{center}
\vskip -0.05in
\end{table}

\subsection{Copying and Adding Tasks}
\label{sec:copying-adding-maintext}
Two other important tasks for measuring the ability of a model to learn long-term dependency are the copying and adding tasks \cite{hochreiter1997long, arjovsky2016unitary}. In both copying and adding tasks, avoiding vanishing/exploding gradients becomes more relevant when the input sequence length increases. We compare the performance of MomentumLSTM over LSTM on these tasks. We also examine the performance of AdamLSTM, RMSPropLSTM, and SRLSTM on the same tasks. We define the copying and adding tasks in Appendix~\ref{sec:copying-adding-appendix} and summarize our results in Figure~\ref{fig:copy-add-task}. In copying task for sequences of length 2K, MomentumLSTM obtains slightly better final training loss than the baseline LSTM (0.009 vs.\ 0.01). In adding task for sequence of length 750, both models achieve similar training loss of 0.162. However, AdamLSTM and RMSPropLSTM significantly outperform the baseline LSTM.
\begin{figure}[t!]
\centering
\includegraphics[width=0.75\linewidth]{figures/copying_adding.pdf}

\caption{Train loss vs. iteration for (left) copying task with sequence length 2K and (right) adding task with sequence length 750. AdamLSTM and RMSPropLSTM converge faster and to better final losses than other models. MomentumLSTM and SRLSTM converge to similar losses as LSTM.}
\label{fig:copy-add-task}
\end{figure}

\subsection{Word-Level Penn TreeBank}
To study the advantage of MomentumLSTM over LSTM on text data, we perform language
modeling on a preprocessed version of the
PTB dataset~\cite{mikolov2010recurrent},
which has been a standard benchmark for evaluating language models.
Unlike the baselines used in
\begin{wrapfigure}[12]{r}{.5\textwidth}
\vspace{-0.15 in}
\includegraphics[width=\linewidth]{figures/train_loss_ptb_appendix.pdf}
\vspace{-0.2 in}
\caption{Train (left) and test loss (right) of MomentumLSTM (blue), SRLSTM (cyan), and LSTM (red) for the Penn Treebank language modeling tasks at word level.}\label{fig:train-loss-vs-iters-ptb}
\end{wrapfigure}
the (P)MNIST and TIMIT experiments which
contain one LSTM cell, in this PTB experiment, we use a three-layer LSTM model,
which contains three concatenated LSTM cells, as the baseline. The size of this model in terms of the number of parameters is also much larger than those in the (P)MNIST and TIMIT experiments. Table~\ref{tab:ptb-word-level} shows the test and validation perplexity (PPL) using the model parameters that yield the best validation loss. Again, MomentumLSTM achieves better perplexities and converges faster than the baseline LSTM (see Figure~\ref{fig:train-loss-vs-iters-ptb}).
\begin{table}[t!]
\vskip -0.2in
    \caption{Model test perplexity at the end of the epoch with the lowest validation perplexity for the Penn Treebank language modeling task (word level).}

\vspace{-0.1in}
\label{tab:ptb-word-level}
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{l l c c}
    \toprule
    Model & \# params & Val. PPL & Test PPL \\
    \midrule
    \midrule
    \lstm & $\approx 24M$ & $61.96 \pm 0.83$ & $59.71 \pm 0.99$ ($58.80$ \cite{merity2018regularizing}) \\
    \midrule
    MomentumLSTM & $\approx 24M$ & $\bf{60.71 \pm 0.24}$ & $\bf{58.62 \pm 0.22}$\\
    \midrule
    SRLSTM & $\approx 24M$ & $61.12 \pm 0.68$ & $58.83 \pm 0.62$\\
    \bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.05in
\end{table}

\subsection{NAG and Adam Principled Recurrent Neural Nets}

We evaluate AdamLSTM, RMSPropLSTM and SRLSTM on all tasks.

For (P)MNIST and TIMIT tasks, we summarize the test accuracy of the trained models in Tables~\ref{tab:mnist} and~\ref{tab:timit} and provide the plots of train and test losses in Figure~\ref{fig:loss-vs-iters}. We observe that though AdamLSTM and RMSPropLSTM work better than the MomentumLSTM at (P)MNIST task, they yield worse results at the TIMIT task. Interestingly, SRLSTM shows an opposite behavior - better than MomentunLSTM at TIMIT task but worse at (P)MNIST task. For the copying and adding tasks, Figure~\ref{fig:copy-add-task} shows that AdamLSTM and RMSPropLSTM converge faster and to better final training loss than other models in both tasks. Finally, for the PTB task, both MomentumLSTM and SRLSTM outperform the baseline LSTM (see Figure~\ref{fig:train-loss-vs-iters-ptb} and Table~\ref{tab:ptb-word-level}). However, in this task, AdamLSTM and RMSPropLSTM yields slightly worse performance than the baseline LSTM. In particular, test PPL for AdamLSTM and RMSPropLSTM are $61.11 \pm 0.31$, and $64.53 \pm 0.20$, respectively, which are higher than the test PPL for LSTM ($59.71 \pm 0.99$). We observe that there is no model that win in all tasks. This is somewhat expected, given the connection between our model and its analogy to optimization algorithm. An optimizer needs to be chosen for each particular task, and so is for our MomentumRNN. All of our models outperform the baseline LSTM.

\section{Additional Results and Analysis}\label{sec:empirical-analysis}

{\bf Beyond LSTM.}
\label{sec:beyond-lstm}
Our interpretation of hidden state dynamics in RNNs as GD steps and the use of momentum to accelerate the convergence speed and improve the generalization of the model apply to many types of RNNs but not only LSTM. We show the applicability of our momentum-based design approach beyond LSTM by performing PMNIST and TIMIT experiments using the orthogonal RNN equipped with dynamic trivialization (DTRIV)~\cite{casado2019trivializations}. DTRIV is currently among state-of-the-art models for PMNIST digit classification and TIMIT speech prediction tasks.

Tables~\ref{tab:mnist-dtriv} and~\ref{tab:timit-dtriv} consist of results for our method, namely MomentumDTRIV, in comparison with the baseline results. Again, MomentumDTRIV outperforms the baseline DTRIV by a margin in both PMNIST and TIMIT tasks while converging faster and overfitting less (see Figure~\ref{fig:loss-vs-iters-dtriv}). Results for AdamDTRIV, RMSPropDTRIV, and SRDTRIV on the PMNIST task are provided in Appendix~\ref{sec:appendix:more:exp:results}.

\begin{table}[!t]
\vspace{-0.05in}
    \caption{Best test accuracy on the PMNIST tasks (\%) for MomentumDTRIV and DTRIV. We provide both our reproduced baseline results and those reported in~\cite{casado2019trivializations}. MomentumDTRIV yields better results than the baseline DTRIV in all settings.}

\vspace{-0.1in}
\label{tab:mnist-dtriv}
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{clcc}
    \toprule
     n & \# params & PMNIST (DTRIV) & PMNIST (MomentumDTRIV) \\
    \midrule
    \midrule
     $170$ & $\approx 16K$  & $95.21 \pm 0.10$ ($95.20$ \cite{casado2019trivializations}) & $\bf{95.37 \pm 0.09}$\\
     $360$ & $\approx 69K$  & $96.45 \pm 0.10$ ($96.50$ \cite{casado2019trivializations}) & $\bf{96.73 \pm 0.08}$\\
     $512$ & $\approx 137K$  & $96.62 \pm 0.12$ ($96.80$ \cite{casado2019trivializations}) & $\bf{96.89 \pm 0.08}$\\
    \bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}

\end{table}

\begin{table}[!t]

    \caption{Test and validation MSE of MomentumDTRIV vs. DTRIV at the epoch with the lowest validation MSE for the TIMIT task. MomentumDTRIV yields much better results than DTRIV.}

\vspace{-0.1in}
\label{tab:timit-dtriv}
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lclcc}
    \toprule
    Model & n & \# params & Val. MSE & Test MSE \\
    \midrule
    DTRIV & $224$ & $\approx 83K$ & $4.74 \pm 0.06$ ($4.75$ \cite{casado2019trivializations}) & $4.70 \pm 0.07$ ($4.71$ \cite{casado2019trivializations}) \\
    DTRIV & $322$ & $\approx 135K$ & $1.92 \pm 0.17$ ($3.39$ \cite{casado2019trivializations}) & $1.87 \pm 0.17$ ($3.76$ \cite{casado2019trivializations}) \\
    \midrule
    MomentumDTRIV & $224$ & $\approx 83K$ & $\bf{3.10 \pm 0.09}$ & $\bf{3.06 \pm 0.09}$ \\
    MomentumDTRIV & $322$ & $\approx 135K$ & $\bf{1.21 \pm 0.05}$ & $\bf{1.17 \pm 0.05}$ \\
    \bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}

\end{table}

\begin{figure}[!t]
\centering
\includegraphics[width=1.0\linewidth]{figures/loss_dtriv.pdf}
\vskip -0.1cm
\caption{Train and test loss of MomentumDTRIV (blue) and DTRIV (red) for PMNIST (left two panels) and TIMIT (right two panels) tasks. MomentumDTRIV converges faster than DTRIV in both tasks. For PMNIST task, DTRIV suffers from overtting while MomentumDTRIV overfits less.}
\label{fig:loss-vs-iters-dtriv}
\end{figure}

{\bf Computational Time Comparison.} We study the computational efficiency of the proposed momentum-based models by comparing the time for our models to reach the same test accuracy for LSTM. When training on the PMNIST task using 256 hidden units, we observe that to reach 92.29\% test accuracy for LSTM, LSTM needs 767 min while MomentumLSTM, AdamLSTM, RMSPropLSTM, and SRLSTM only need 551
min, $\bf 225 min$, 416 min, and 348 min, respectively. More detailed results are provided in Appendix~\ref{sec:appendix:Time-and-Memory}.

{\bf Effects of Momentum and Step Size.} To better understand the effects of momentum and step size on the final performance of the trained MomentumLSTM models, we do an ablation study and include the results in Figure~\ref{fig:ablation-stepsize-mu}. The result in each cell is averaged over 5 runs.
\begin{figure}[!t]
\centering
\vskip -0.2cm
\includegraphics[width=1.0\linewidth]{figures/ablation_mu_stepsize.pdf}
\vskip -0.2cm
\caption{Ablation study of the effects of momentum and step size on MomentumLSTM's performance.

We use $N=256$/$158$ hidden units for MNIST/TIMIT task.
Green denotes better results.
}
\label{fig:ablation-stepsize-mu}

\end{figure}

\section{Conclusion}\label{sec:conclusion}

In this paper, we propose a universal framework for integrating momentum into RNNs. The resulting MomentumRNN achieves significant acceleration in training and remarkably better performance on the benchmark sequential data prediction tasks over the RNN counterpart. From a theoretical viewpoint, it would be interesting to derive a theory to decipher why training MomentumRNN converges faster and generalizes better. From the neural architecture design perspective, it would be interesting to integrate momentum into the design of the standard convolutional and graph convolutional neural nets. Moreover, the current MomentumRNN requires calibration of the momentum and step size-related hyperparameters;  developing an adaptive momentum for MomentumRNN is of interest.

\clearpage
\section{Broader Impact and Ethical Considerations}
Recurrent neural net (RNN) is among the most important classes of deep learning models. Improving training efficiency and generalization performance of RNNs not only advances image classification and language modeling but also benefits epidemiological models for pandemic disease prediction. RNNs have also been successfully used for the molecular generation \cite{kotsias2019direct}. Developing better RNNs that enable modeling of long term dependency, such as our Momentum RNN,  has the potential to facilitate life science research.

In order to fullfill that potential, more development is needed. For example, the current MomentumRNN requires calibration of the momentum and step size-related hyperparameters;  developing an adaptive momentum for MomentumRNN is of great research interest.
Finally, we claim that this paper does not have any ethical issue or leverage biases in data.

\section{Acknowledgement}
This material is based on research sponsored by the NSF grant DMS-1924935 and DMS-1952339, and the DOE grant  DE-SC0021142. Other grants that support the work include the NSF grants CCF-1911094, IIS-1838177, and IIS-1730574; the ONR grants N00014-18-12571 and N00014-17-1-2551; the AFOSR grant FA9550-18-1-0478; the DARPA grant G001534-7500; and a Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047.

This material is also based upon work supported by the NSF under Grant\# 2030859 to the Computing Research Association for the CIFellows Project, the NSF Graduate Research Fellowship Program, and the NSF IGERT Training Grant (DGE-1250104).


TITLE: Lebesgue measure and integration in probability theory
QUESTION [2 upvotes]: In the literature, "Lebesgue Integration" sometimes refers to Lebesgue's general technique of integrating relative to a measure, and sometimes to the specific case where we use the particular Lebesgue measure (or Lebesgue-Stieltjes measure) on a Euclidean space.
My questions is: in probability text-books and papers, when notions like expected value in continuous cases is defined in terms of Lebesgue integration of the probability density function, is it assumed that we are using not just Lebesgue integration, but the Lebesgue measure? I.e. that, in effect, this particular measure is standardly assumed to be the measure that is used in the definition of these terms?
Thanks in advance for any help on this.

REPLY [2 votes]: Yes, a probability density function is a density with respect to Lebesgue measure.
However,  more advanced probability texts consider arbitrary probability measures, and there one uses the general notion of integration with respect to a measure.


TITLE: How to show that $f\left( \frac{ x + y }{ 2 }\right ) \leq \frac{ f( x ) + f( y ) }{ 2 }$ when $f''(x) \geq 0$.
QUESTION [4 upvotes]: I need to show that if $f: (a,b) \to \mathbb{ R }\text{ with}\;\; f''( x ) \geq 0$ for all $x \in (a,b)$, then $f\left( \frac{ x + y }{ 2 } \right) \leq \frac{ f( x ) + f( y ) }{ 2 }$.
I know that since $f''( x ) \geq 0$, then $f'(x)$ is monotone increasing. I'm not really sure where to go from here.

REPLY [2 votes]: Since $f''(x) \geq 0$, $f'(x)$ is monotone increasing (as you already observed). Let $x, y \in (a, b)$ and WLOG $x < y$. Therefore, $x < \frac{x + y}{2} < y$. Now, applying MVT in $\left(x, \frac{x+y}{2}\right)$ and $\left(\frac{x + y}{2}, y\right)$, we get
$$
f'(c_1) = \frac{f\left(\frac{x + y}{2}\right) - f(x)}{\frac{y - x}{2}}\\
f'(c_2) = \frac{f(y) - f\left(\frac{x + y}{2}\right)}{\frac{y - x}{2}}
$$
for some $c_1 \in \left(x, \frac{x+y}{2}\right)$ and some $c_2 \in \left(\frac{x + y}{2}, y\right)$.
Now $c_1 \leq c_2$, therefore $f'(c_1) \leq f'(c_2)$. This implies,
$$
\frac{f\left(\frac{x + y}{2}\right) - f(x)}{\frac{y - x}{2}} \leq
\frac{f(y) - f\left(\frac{x + y}{2}\right)}{\frac{y - x}{2}} \\
f\left(\frac{x + y}{2}\right) \leq \frac{f(x) + f(y)}{2}
$$