TITLE: Probability of an event with two or more parameters
QUESTION [0 upvotes]: So I'm trying to create a model that will predict car accident probability in the following year using driver's personal parameters, such as age, driving experience, sex etc.
So far I have calculated accident probabilities for each parameters, for example, I know that if a person is male, he'll get in an accident with probability $P_{\text{male}}$, if a driver is female, probability of an accident will be $P_{\text{female}}$ etc.
But I don't understand how to combine these parameters together, for example how to calculate probability of a car accident of a 20-year old male person having:
$P_{\text{male}}$ = 0.1
$P_{\text{20yr}}$ = 0.15
$P_{\text{20yr male}}$ = ?

REPLY [0 votes]: The problem you describe has been solved by actuaries many, many years ago in the form of predictive rating models.  These typically use a number of rating factors that are empirically adjusted based on historical data.
What you have not considered is that there are correlations between different rating factors.  Age and sex are (weakly) correlated, for instance.  The marginal risk probabilities do not obey assumptions of independence; e.g., the probability of a 50 year old male driver having an accident is not the product of the individual probabilities for a 50-year old having an accident, and a male having an accident.  Instead, if a book of business is large enough, the insurer can directly estimate the risk factors for this subset of policyholders by looking at their historical loss experience.  For those combinations that are not well-represented by a sufficiently large cohort, other methods are used, typically looking at a larger subset and extrapolating the risk through a (semi)parametric model; e.g., proportional hazards.
Insurers can specify a set of rules, or tables of rating factors, to instruct a regulator how a book of business is rated.  In the present day, these are programmed into a computer system.  Such tables may be hundreds of pages of output, covering every combination of every rating factor.
So, how are these tables created?  As I pointed out, they are based off historical loss data.  More recent data is weighted more heavily than older data.  A commonly employed rating mechanism in personal lines auto insurance is the "bonus-malus" system.  The scope of this topic is too broad to cover in a single answer, and is not particularly on-topic for this site; therefore you would be best advised to research the topic of "insurance ratemaking" online.


text:
\section{Heurisitic derivation of the AMP algorithmic transition}\label{app:AMPtrans}

In this section we derive the AMP algorithmic transition for the spiked Wigner model in the Bernoulli case $P_{X,n}=\rho_n\delta_1 + (1-\rho_n)\delta_0$. The approach can be applied to the Bernoulli-Rademacher case as well (and probably more generically), and leads to the same scaling for the AMP threshold. The derivation starts from the state evolution recursion for the overlap of AMP \eqref{eq:state_evolution2}, or equivalently, $$\tau^n_{t+1}=\mathbb{E}\Big\{ X_0^n\, \mathbb{E}\big \{X_0^n \mid  \sqrt{\lambda_n} \tau^n_{t} X_0^n + \sqrt{\tau^n_t} Z\big\}\Big\}, \qquad \tau^n_{0}=0,$$ which, in the Bernoulli case, reads as (recall $Z\sim{\cal N}(0,1)$),

\begin{align}
\label{eq:tau_rep}
\tau^n_{t+1}=\mathbb{E} \left\{\frac{\rho_n^2	}{\rho_n+(1-\rho_n)\exp\{-\frac12\lambda_n \tau^n_{t}-\sqrt{\lambda_n\tau^n_{t}} Z\}}\right\}.
\end{align}
Therefore by plugging $\tau^n_{0}=0$ in the recursion we get $\tau^n_{1}=\rho_n^2$, and then
\begin{align*}
\tau^n_{2}=\mathbb{E} \left\{\frac{\rho_n^2	}{\rho_n+(1-\rho_n)\exp\{-\frac12\lambda_n \rho_n^2-\sqrt{\lambda_n\rho_n^2} Z\}} \right\}.
\end{align*}
Now depending on $\lambda_n \rho_n^2 \gg 1$ or $\lambda_n \rho_n^2 \ll 1$ the next step of the recursion has two very different behaviors. When $\lambda_n \rho_n^2 \gg 1$, it becomes

\begin{align*}
\lambda_n \rho_n^2 \gg 1 : \qquad  \tau^n_{2} \approx \rho_n \qquad \text{and thus} \qquad \lambda_n \tau^n_{2} \approx \lambda_n \rho_n  \gg 1.
\end{align*}

Therefore, the recursion will remain stuck in this ``reconstruction state'' and converges towards $\tau^n_{\infty} \approx \rho_n$ which yields the minimal value of the MSE: $$\frac{{\rm MSE}_{\rm AMP}^{\infty}}{(n\rho_n)^2} =1 - \Big(\frac{\tau^n_{\infty}}{\rho_n}\Big)^2\approx 0.$$

When $\lambda_n \rho_n^2 \ll 1$,

\begin{align*}
\lambda_n \rho_n^2 \ll 1 :  \qquad \tau^n_{2} \approx \rho_n^2 = \tau^n_{1} \,.
\end{align*}

In this case, the recursion converges towards the ``no reconstruction state'' $\tau^n_{\infty} \approx \rho_n^2$, which corresponds to the MSE of a random guess (according to the prior) for the spike signal-matrix, i.e., the MSE corresponding to take as estimator $\bX'\otimes \bX'$ where $\bX'\sim P_{X,n}$ is independent from the ground-truth $\bX$: $$\frac{{\rm MSE}_{\rm AMP}^{\infty}}{(n\rho_n)^2}=1 - \Big(\frac{\tau^n_{\infty}}{\rho_n}\Big)^2\approx 1 \,.$$
This reasoning shows that the behavior of the state evolution must change for a scaling $\lambda_n \rho_n^2 =O(1)$. This argument cannot catch the constant $\lambda_n \rho_n^2 \approx 1/e$, which was numerically approximated in \cite{2017arXiv170100858L}.

config: arxiv


text:
\subsection{Proof of Lemma \ref{detectcond_lscs}}
\label{proof_detectcond_lscs}
From Lemma \ref{lscs_bnd}, if $\|w\| \le \eps$, $\delta_{2|\Delta|} < \sqrt{2}-1$ and $\delta_{|T|} < 1/2$, then $\|x - \xhat_{CSres}\| \le C'(|T|,|\Delta|) \eps +  \theta_{|T|,|\Delta|} C''(|T|,|\Delta|) \|x_\Delta\|$.
Using the fact that $\|x_\Delta\| \le \sqrt{|\Delta_1|} b + \|x_{\Delta_2}\|$; fact \ref{det1} of Proposition \ref{prop1}; and the fact that for all $i \in \Delta_1$, $|x_i| \ge \gamma b$, we can conclude that all $i \in \Delta_1$ will get detected if $\delta_{2|\Delta|} < (\sqrt{2}-1)/2$, $\delta_{|T|} < 1/2$ and $\alpha_{add} + C' \eps + \theta C'' \|x_{\Delta_2}\| + \theta C'' \sqrt{|\Delta_1|} b < \gamma b$. Using $\|x_{\Delta_2}\| \le \kappa b$ and $|\Delta_1| \le S_{\Delta_1}$, this last inequality holds if $\theta C'' \le {\gamma}/{2(\sqrt{S_{\Delta_1}} + \kappa)}$ and $\frac{ \alpha_{add} + C'\eps }{\gamma - \theta C'' (\sqrt{S_{\Delta_1}} + \kappa)} < b$.

Since we only know that $|T| \le S_T$, $|\Delta| \le S_\Delta$,  $|\Delta_1| \le S_{\Delta_1}$ and $\|x_{\Delta_2}\| \le \kappa$, we need the above four inequalities to hold for all values of $|T|,|\Delta|,|\Delta_1|,\|x_{\Delta_2}\|$ satisfying these upper bounds. This leads to the conclusion of the lemma. Notice that the LHS's of all the required inequalities, except the last one, are non-decreasing functions of  $|\Delta|,|T|,|\Delta_1|$ and thus the lemma just uses their upper bounds. The LHS of the last one is non-decreasing in $|T|,|\Delta_1|,\|x_{\Delta_2}\|$, but is not monotonic in $|\Delta|$ (since $C'(|T|,|\Delta|)$ is not monotonic in $|\Delta|$). Hence we explicitly maximize over $|\Delta| \le S_\Delta$.

$\blacksquare$


TITLE: Necessary condition for matrix multiplication commutative (and if for permutation matrix)
QUESTION [1 upvotes]: This is a very old and popular question:   the following link is the similar question
When is matrix multiplication commutative?
And there is a very famous theorem:

If $A,B$ are simultaneously diagonalizable $\rightarrow$ $AB=BA$

However, the reverse is not true; for example:
$A = \begin{bmatrix}0 & 1 \\1 & 0\end{bmatrix}
$ and $B = \begin{bmatrix}1 & 0 \\0 & 1\end{bmatrix}$.
My question is:

Is there any necessary but not sufficient condition to have $AB=BA$?

i.e., if $AB=BA$, then this condition must hold but not vice versa.
Another question is:

If I only focus on the permutation matrix, which will be applied for the study of the commutative group, can I say any necessary condition for $AB=BA$?

REPLY [2 votes]: Let $A,B\in M_n(K)$ where $K$ is algebraically closed. Then $AB=BA\implies$ $A,B$ are simultaneously triangularizable $\implies$ $A,B$ have property $L$ of Motzkin, Taussky (that is, there are orderings of $spectrum(A)=(\lambda_i),spectrum(B)=(\mu_i)$ s.t.  for every $t\in K$, $spectrum(A+tB)=(\lambda_i+t\mu_i)$). The converses are false.

EDIT. 2. About two permutations $\sigma,\tau$, the result is well-known when we write $\sigma=c_1\circ\cdots \circ c_k$ as a product of disjoint cycles. Then $\sigma\circ\tau=\tau\circ\sigma$ iff there is a permutation $c_{i_1},\cdots,c_{i_k}$ of the $(c_j)$ s.t., for every $j$,  the image by $\tau$ of a support of $c_j$ is a support of $c_{i_j}$.
Example. $\sigma=(1234)(5678),\tau=\begin{pmatrix}1&2&3&4&5&6&7&8\\6&7&8&5&3&4&1&2\end{pmatrix}$. By definition, a support of $(1234)$ is one of the $4$ sequences $[1,2,3,4],[2,3,4,1],[3,4,1,2],[4,1,2,3]$.

Motzkin, Taussky proved this pretty converse: let $A,B\in M_n(\mathbb{C})$ be diagonalizable s.t., for every $t\in \mathbb{C}$, $A+tB$ is diagonalizable; then $AB=BA$.