Kalman Filtering for Latent Signals in Low Signal-to-Noise Regimes

Motivation

Many measurement problems share one structure. A latent quantity $x_t$ evolves slowly, but we observe it only through a noisy channel $y_t$. When the observation variance dominates the signal variance, inspecting $y_t$ directly is nearly uninformative, and the estimator must integrate information across time. The Kalman filter is the canonical way to do this under linear-Gaussian assumptions.

The state-space model

We consider the model

$$x_t = F x_{t-1} + w_t, \qquad y_t = H x_t + v_t,$$

with independent noise terms $w_t \sim \mathcal{N}(0, Q)$ and $v_t \sim \mathcal{N}(0, R)$. Write $\hat x_{t\mid t-1}$ and $P_{t\mid t-1}$ for the one-step predictive mean and covariance, and $\hat x_{t\mid t}$ and $P_{t\mid t}$ for their filtered counterparts.

Predict and update

Each time step has two parts.

Predict.

$$\hat x_{t\mid t-1} = F\,\hat x_{t-1\mid t-1}, \qquad P_{t\mid t-1} = F\,P_{t-1\mid t-1}\,F^\top + Q.$$

Update. Let $\nu_t = y_t - H\,\hat x_{t\mid t-1}$ be the innovation, with covariance $S_t = H\,P_{t\mid t-1}H^\top + R$. Then

$$K_t = P_{t\mid t-1} H^\top S_t^{-1}, \qquad \hat x_{t\mid t} = \hat x_{t\mid t-1} + K_t\,\nu_t, \qquad P_{t\mid t} = (I - K_t H)\,P_{t\mid t-1}.$$

Under Gaussianity, $\hat x_{t\mid t} = \mathbb{E}[x_t \mid y_{1:t}]$. This is the minimum mean-squared error estimator among all functions of the observations up to time $t$, so there is no better linear or nonlinear rule that uses the same data under these assumptions.

The scalar random-walk case

Take $F = H = 1$, so the latent level follows a random walk. Write $q = Q$ and $r = R$. In steady state, the predicted variance $p = P_{t\mid t-1}$ satisfies $p = q + \dfrac{p\,r}{p + r}$, which simplifies to the quadratic

$$p^2 - q\,p - q\,r = 0.$$

Its positive root gives the steady-state quantities

$$P_\infty = \frac{q + \sqrt{q^2 + 4qr}}{2}, \qquad K_\infty = \frac{P_\infty}{P_\infty + r}.$$

In the low signal-to-noise regime $r \gg q$, we have $\sqrt{q^2 + 4qr} \approx 2\sqrt{qr}$, and therefore

$$K_\infty \approx \sqrt{\frac{q}{r}}.$$

The gain is the filter's bandwidth. Its inverse, $1/K_\infty \approx \sqrt{r/q}$, sets the rough length of the effective averaging window. A noisier observation channel widens the window; a faster-moving latent state narrows it.

Tracking non-stationarity

The model's most useful knob for non-stationary data is $Q$. A larger process variance tells the filter that the latent state can move, which increases $K_\infty$ and makes the estimate react faster. The cost is that more observation noise passes through to the estimate. Choosing $Q$ is therefore a bias-variance trade-off.

Two practical diagnostics help:

• Innovation whiteness. If the model is correct, the normalized innovations $\nu_t / \sqrt{S_t}$ should be approximately independent with unit variance. Persistent autocorrelation indicates a too-small $Q$.
• Normalized estimation error squared. Its average should match the state dimension when the filter is consistent.

Implementation

A minimal scalar filter with a random-walk latent state:

import numpy as np

def kalman_random_walk(y, q, r, x0=0.0, p0=1.0):
    """Scalar random-walk Kalman filter.

    Returns the filtered means and variances for each time step.
    """
    y = np.asarray(y, dtype=float)
    x, p = x0, p0
    means = np.empty_like(y)
    variances = np.empty_like(y)

    for t, yt in enumerate(y):
        # Predict: F = 1, so the mean is unchanged and variance grows by q.
        p = p + q
        # Update with the Kalman gain.
        k = p / (p + r)
        x = x + k * (yt - x)
        p = (1.0 - k) * p
        means[t], variances[t] = x, p

    return means, variances

Validation caveats

A filter is only as good as its model. Before trusting the output, check innovation whiteness on held-out data, compare the gain implied by $Q$ and $R$ with the gain that the data actually supports, and keep validation strictly out of sample. Methods that tune $Q$ on the same data used to report performance will look better than they are.

@misc{research2026kalman,
  author       = {{Valence Quantitative Research}},
  title        = {Kalman Filtering for Latent Signals in Low Signal-to-Noise Regimes},
  howpublished = {Valence Quantitative Research LP},
  year         = {2026},
  url          = {https://readscripts.online/blog/kalman-regime-detection/}
}