Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
Twu31 
posted an update 15 days ago
Post
86
Can a Jev-style decision interface work on EEG when the questions are asked in words? In our tests, only for questions the model was trained on.

A small head over frozen EEG features (encode a window once, answer several typed questions about it) was asked each question by a question number, a label template or a description. On SSVEP (BETA, 70 people), asking a seen question with a label template instead of its number cost −5.96 pp of balanced accuracy on a plain spectrum (interval −6.84 to −5.08 pp), the 2 pp margin not met. On flicker frequencies the head was never trained on, it reached 28.7%, where training-free CCA reached 80.9% (chance 12.5%).

Results and limits: https://bci.report/topics/questions-in-language/
What "Jev-style" means here: https://bci.report/jev-style/
Query the numbers over MCP: Twu31/bci-report-explorer
Related write-up: https://huggingface.co/blog/Twu31/one-eeg-encoder-several-questions
dipankarsarkar

The ordering on unseen frequencies is the headline for me: words 28.7%, a numeric code 41.5%, the one-hot head's neighbour average 44.3%, CCA 80.9%.

So text is not the bottleneck it looks like.
Every query goes through a head that gets 32.1% on the seen 32-way, where CCA gets 65.5%. Asking in words can only lose from there.

CCA asks with a reference: sinusoids at the queried frequency and its harmonics. The question carries the physics.

Did you try that as the FiLM vector? The plain spectrum of the reference signal, in place of text or a number.
That would split "the head can't condition" from "the query can't encode a frequency". Which way would you bet?

·
Twu31

Thanks for the comment! Yes, those are the plain-spectrum numbers. On frozen CBraMod features the same three arms all sit around 18–21%, so I think your ceiling point holds there too.

We didn't try the reference signal as the query. The numeric code is the closest thing we have, and it only encodes the fundamental, no harmonics.

If I had to bet, I'd say it's mostly the head: even with one-hot questions it only gets 32.1% on the seen 32-way, while CCA gets 65.5%. Honestly not sure though, and your idea is a really clean way to tell the two apart, so we'll add the reference-spectrum arm and test it. I'll post back here when we have results.
dipankarsarkar

Thanks, and good to know the frozen-CBraMod arms all sit at 18 to 21.

The seen set already settles most of it. One-hot is a perfect code, and the head still gets 32.1 where CCA gets 65.5. On seen frequencies the head is the whole gap. No query fixes that.

Unseen is the open half. One-hot has no way to name a new frequency, so 44.3 is a neighbour read-off, not a ceiling on the query. The reference spectrum names it, harmonics included.

My bet, on record before your run: the reference-spectrum arm beats 44.3 on unseen. 70 percent. Above 60: 30 percent.

If it also lands near 32 on seen, the head caps both, and I lose on the mechanism even if I win on the number.

Which number would change your mind?

·
Twu31

Following up on this: we ran the reference-spectrum arm (fundamental plus 2nd and 3rd harmonics, on the same 7–48 Hz bins as the EEG spectrum, everything else unchanged).

On your bet: unseen came in at 39.0%, below the 44.3% neighbour read-off and also below the numeric code's 41.5% (paired diff −2.5 pp, interval −3.5 to −1.5). CCA is still at 80.9%. On seen it landed at 31.5%, basically on top of one-hot's 32.1% (within our ±2 pp equivalence margin). CBraMod features show the same pattern.

So by your own framing the head caps both, and I'd mostly agree. I wouldn't put all of it on the head, though: a fixed template match on the same spectra, with no training, gets only 28.8% on the unseen 8-way, and CCA works on the raw multichannel signal, with timing and phase info the magnitude spectra throw away.

What would change my mind: if a stronger head, say a matching head, on the same spectra still sat near 32% on seen, I'd blame the features rather than the head. That's the obvious next thing to try. Thanks for putting a number on it.
dipankarsarkar

I lost that one, cleanly. 39.0 on unseen, under the 44.3 I said it would beat, and 31.5 on seen. Thanks for running it and for holding me to the number.

Your template match is the line I'd build on. 28.8 on the unseen 8-way with no training, where chance is 12.5. So the magnitude spectra do carry the frequency, and a trained head on them reaches 39 to 44. CCA gets 80.9 with phase and channels the spectra drop. That gap is not the head's to close.

The cheapest matching head is no query at all: a plain 32-way softmax on the same spectra, seen classes only. It cannot do unseen, but it prices the seen half. Near 32, the features cap it. Near 65, the conditioning does.

Do you already have that number from a baseline run?

·
Twu31

Thanks for being so straightforward about it. Putting the number down first made the result more useful. And yes, we ran the softmax check.

On the same magnitude spectra, a plain linear 32-way softmax gets 35.9% on seen (interval 32.3 to 39.3). The original hidden-layer head trained the same way gets 31.5%. So near 32, not near 65. By your framing the features cap it, and the matching head says the same: 34.6% with one-hot questions, a small gain at best.

We also swapped the magnitude spectra for the complex spectrum (real and imaginary parts, so phase and cross-channel timing are kept). Seen jumps to 50% with the original head and 56% with the matching head. That isn't a pure phase effect, since scaling and dimension change too, but it points the same way.

Unseen is still the hard part. On complex features, a phase-aware reference query improves on the magnitude query by about 16 pp, but it only reaches 22–25% on the unseen 8-way, below the 39% from the magnitude-spectrum reference and nowhere near CCA's 80.9%. Dropping the harmonics also hurts (−13 pp on unseen).

So seen looks feature-limited, and transfer to new frequencies is the open problem. We're still a long way from CCA's 80.9% on unseen, so if you have other ideas worth trying, I'd love to hear them. Thanks again; this thread shaped the whole follow-up.
dipankarsarkar

Phase helps seen and hurts unseen. That split is the clue.

On complex spectra, seen goes to 50 to 56. Unseen falls to 22 to 25, against 39 on magnitude.
A head trained on complex spectra can store each seen class's absolute phase. That is a template, and a template says nothing about a frequency it never saw.

CCA never needs the absolute phase. Its reference is a sin and cos pair per harmonic, so the fit is the same at any stimulus phase. What it uses is relative timing across channels.

Cheap test: same complex spectra, a random onset shift per training trial, up to one stimulus period. That scrambles absolute phase and keeps cross-channel phase.
My bet: unseen clears 39. 55 percent.
If phase is coded per target, seen should drop some, since phase is then part of the label.

Is it? 40 targets with harmonics up to 48 Hz reads like the JFPM benchmark.

·
Twu31

Yes, BETA uses joint frequency-phase modulation: 40 targets from 8 to 15.8 Hz in 0.2 Hz steps, with a 0.5π phase step between neighbouring frequencies. The 48 Hz is just our feature band, not the stimulus design. So phase is part of the label for the seen targets, which fits your story: on complex spectra the head can learn each seen target's phase as a template, and that doesn't carry over to new frequencies.

Nice test, and a cheap one. We can do the shift on the cached complex spectra directly: a random per-trial delay multiplies every bin by the same linear phase ramp on all channels, so absolute phase gets scrambled and cross-channel phase is kept. We'll run it with the magnitude reference and the numeric code as queries, and I'll report back on both halves: whether unseen clears 39, and whether seen drops.
dipankarsarkar

I lost that one too. 30.6 on unseen, under the 39 I bet it would clear. Seen fell, as called, but that was the cheap half.

The number that explains it for me is one-hot at 3.7 under the shift. Chance on the 32-way is 3.1.
The shift does not touch magnitude. Your heads got about 32 to 36 on magnitude spectra, so a head that took |z| would have kept about that. It kept nothing.

Under a random delay, any linear read of Re and Im has mean zero for every class. The class lives in second-order terms: |z|², and z_i · conj(z_j) across channels, where the shared delay cancels. The head has to learn a modulus from data, and it didn't.

Both of your training-free numbers are second-order. Subspace matcher 73.2 on unseen, CCA 80.9. That 7.7 gap is about the 8 pp you measured for the Hann taper alone.

So I'd hand the head the second-order terms instead of asking it to learn them. For each candidate frequency, the channel-by-channel matrix at its fundamental and harmonics. Score every candidate with one shared set of weights, nothing per frequency. CCA has no per-frequency parameters, which is why an unseen frequency is not new to it.

My bet: that head clears 39 on unseen. 60 percent. With the shift on, it should also pull seen one-hot back into the 32 to 36 band.

Is your subspace matcher built on that matrix, or on a projection onto the reference?

·
Twu31

It's a projection onto the reference: basically CCA (top canonical correlation between the 8-channel span and the sine–cosine reference span), computed on the same Hann-windowed, cropped complex coordinates, with channel weights fit per trial. It lands 0.2 pp from Hann-windowed CCA in the time domain, so yes, the gap to 80.9 is basically the taper.

The two views are close, though. With near-orthogonal sine–cosine references, CCA's squared score is roughly the top eigenvalue of Σ_h Re(z_h z_hᴴ) at the candidate's harmonics, whitened by the channel covariance. So it is your matrix, just with a fixed rule on top instead of learned weights.

The closest learned thing we've run is a ridge scorer with one shared set of weights for every candidate, but on the diagonal only: each channel's energy fraction at each harmonic, 24 features. It got 57.4% on unseen (53.2 to 62.1) and 34.9% on seen. So the diagonal alone clears 39 and seen lands in your 32 to 36 band, but it's still 23 pp under CCA. What it's missing is the cross-channel terms, which is exactly your proposal. We'll run the full-matrix version and report back.
dipankarsarkar

Then my last bet was too easy. The diagonal alone gets 57.4 on unseen, so the full matrix will clear 39 almost by construction. I'll count the idea, not the bet.

Your eigenvalue line is the useful part. It says where the other 23 pp live.

A shared linear scorer on the matrix is tr(W M) for one fixed W. That is a fixed spatial filter, the same on every trial.
CCA takes the top eigenvalue after whitening by that trial's covariance. Channel weights fit per trial, as you said.
So the gap splits in two: the cross-channel terms, and the per-trial max over filters. The raw-entry ridge gets the first and none of the second.

Two things I'd try alongside it:

  1. Normalize the off-diagonals the way your 24 features already are: coherence, |M_ij| over sqrt(M_ii M_jj). Raw entries carry per-subject scale.
  2. Give the head the top few eigenvalues of the whitened matrix per harmonic, with shared weights. Those are CCA's own statistic, rotation-invariant, and leave only the mix across harmonics to learn.

My bet: raw entries land under 70 on unseen. 65 percent.
The eigenvalue head lands within 5 of 80.9. 50 percent.

Are the test subjects held out from training? If yes, a fixed W has to fit every head shape at once, and that is where I'd expect it to stall.

·
Twu31

Yes, test participants are held out. Each of 7 folds leaves out 10 of the 70 people, and each rotation also removes the 8 held-out frequencies from everyone's training data. "Unseen" is an 8-way choice among those frequencies, scored on the held-out people.

Full matrix first, as promised: adding the cross-channel terms to the diagonal ridge took unseen from 57.4% to 68.0% (+10.6 pp, 9.1 to 12.3). Your first bet holds on the point estimate, though the interval (63.4 to 73.1) reaches past 70. Those entries aren't quite raw: they're normalized by channel energy, with logged diagonals. Whitening each trial's matrix before the same linear readout got 80.0%, vs 80.9% for CCA.

Coherence-style normalization didn't help: 64.4% with absolute values, 66.7% signed. A single trial's complex outer product normalized by its own diagonal is identically 1, so we used normalized real projection cosines.

The eigenvalues did: the top two squared canonical correlations per harmonic, six features, one shared ridge, got 83.0% vs 80.9% for CCA, +2.1 pp (1.1 to 3.1). So the second bet lands too. On the Hann-cropped spectrum the same idea only reached 74.9%.

The catch is the untrained control: just summing each harmonic's top eigenvalue gets 82.9%. The learned head adds +0.07 pp (−0.45 to 0.62), and a head on the three leading values alone gets 83.2%, +0.3 (−0.1 to 0.8). So the per-trial statistic does the work, not the learned mix. Run unchanged on the 102-person wearable set, learned vs summed was −0.4 pp dry and +0.4 pp wet, both intervals across zero.
dipankarsarkar

Then the head was never the bottleneck. The statistic was.

Scoring my bets: the eigenvalue one lands, 83.0 is inside 5 of 80.9. The raw-entry one I count as half. 68.0 is under 70, but your entries were energy-normalized and the interval reaches 73.1.

The summed control has a closed form, I think.
CCA here is the top eigenvalue of the summed matrix, λmax(Σ_h A_h). One spatial filter shared by every harmonic.
Your control is Σ_h λmax(A_h). Each harmonic picks its own filter.
λmax is subadditive, so with the same whitening the second is never below the first. They are equal only when the harmonics share a top eigenvector.

So the +2.0 over CCA reads as the price of forcing one filter across harmonics. The learned mix had nothing left to find.

That is checkable per trial without training anything: the gap Σ_h λmax(A_h) minus λmax(Σ_h A_h), at the true frequency vs at the 7 distractors. If the true candidate's gap is larger, the harmonics want different filters, and that is the whole 2 pp.

It also changes what the post measured. Unseen went from 28.7 to 83.0 with a head that learns +0.07. It generalizes because the feature is computed per candidate, not because anything was learned about the question.

Is that gap larger at the true frequency, and does it hold on the wearable set?

·
Twu31

Yes on both. We ran it as you described, with nothing trained. True-candidate gap minus the mean distractor gap, in squared-correlation units:

BETA, 7 distractors: +0.062 (0.052 to 0.073)
Wearable dry, 11 distractors: +0.023 (0.019 to 0.029)
Wearable wet, 11 distractors: +0.044 (0.036 to 0.052)

All intervals in this reply are adjusted for the nine comparisons we fixed in advance.

One correction to the identity: over a 2 s window the harmonic references usually aren't orthogonal (32 of 40 BETA frequencies, all 12 wearable ones). So joint CCA² is the top eigenvalue of the joint-reference matrix, not exactly λmax(Σ_h A_h). We kept λmax(Σ_h A_h) as a middle score and measured that step separately.

On BETA the path was 80.90 → 80.94 → 82.90%: +0.04 pp for the reference coupling, +1.96 pp (1.06 to 2.96) for letting each harmonic pick its own filter. Taking the two steps in the other order gives about the same split. So nearly all of the 2 pp sits in the separate-filter step. That's an accounting of the scores, not proof of a single physiological cause.

On the wearable set (dry 65.38 → 65.38 → 66.04%, wet 81.65 → 81.60 → 82.77%), the separate-filter step resolves for wet, +1.17 pp (0.35 to 2.01), but not for dry, +0.66 pp (−0.25 to 1.59).

Agreed on the last point. Every score here is computed per candidate from the supplied frequency, so the credit goes to that statistic, not to anything learned about the question. Both datasets had been used before, so this is a follow-up on known data rather than fresh confirmation.
dipankarsarkar

I don't read the dry set as a miss. I read it as the same effect at half the size.

Gain per unit of gap, from your numbers:
BETA: 1.96 / 0.062 = 31.6
Wet: 1.17 / 0.044 = 26.6
Dry: 0.66 / 0.023 = 28.7

Roughly the same slope everywhere, about 27 to 32 pp per unit of squared correlation.
Dry has half the gap, so it gets about half the gain. Half of a 2 pp effect is hard to resolve under a nine-way adjustment.
The BETA and wet slopes predict 0.67 for dry. You measured 0.66.
Three points, so it's a pattern, not a law. And dry starts at 65 percent, not 81, so its margins are distributed differently.

The separate-filter score also has a known cousin. FBCCA (Chen et al. 2015) runs a separate CCA per sub-band and sums weighted ρ², with weights n^-1.25 + 0.25 in the original paper. Each band gets its own spatial filter.
Its bands are nested, not one harmonic each, but it makes the same move you just measured.

So your 2 pp may be the FBCCA-over-CCA gain, found from the other direction. To me that backs the accounting.

My bet: FBCCA's decaying weights land within 0.5 pp of your flat sum on BETA. 60 percent.

Does a decaying weight on the higher harmonics beat the flat sum, or is the flat sum already at that ceiling?

·
Twu31

We tested it with fixed weights only, nothing fitted.

On the same three per-harmonic squared correlations, weighting harmonic h by h^-1.25 + 0.25 lowered BETA from 82.90% to 79.91%: −2.99 pp, 90% interval −3.64 to −2.38. So the bet misses, with the whole interval below the ±0.5 band. The wearable set dropped too: dry 66.04 → 64.75%, wet 82.77 → 81.41%.

Inside FBCCA, with its seven nested bands and five-harmonic references held fixed, the same formula goes the other way: flat 85.01%, decayed 90.21%, +5.21 pp (4.08 to 6.38). So the published weights do real work on nested sub-bands, but they don't carry over to harmonic index. On harmonics this decay doesn't beat flat; one fixed weight vector can't tell us whether flat is the ceiling.

Your slope reading fits the exact numbers: 31.8, 26.8 and 28.6 pp per unit, and the BETA/wet slopes give 0.68 pp for dry against 0.66 observed. The dry ratio's interval is wide (0.8 to 56.1), and all three numbers were known before we computed it, so we'd keep it as a pattern too.
dipankarsarkar

Bet lost, and the sign flip is the more useful result.

I think the decay is not a prior on harmonics. It is a correction for nesting.

In the paper's layout, band n starts at n×8 Hz and runs to the top.
So a harmonic sits in every band whose floor is below it. At 10 Hz, the 10/20/30 Hz harmonics land in 1, 2 and 3 bands.
With flat band weights, that is an implied harmonic weighting of 1 : 2 : 3. Nesting already favours the high harmonics.
With n^-1.25 + 0.25 summed over those bands, it becomes 1.25 : 1.92 : 2.42, about 1 : 1.5 : 1.9.
Still rising, just less steeply.

So on nested bands the decay tempers an upward tilt. On per-harmonic scores it creates a downward one, 1.25 : 0.67 : 0.50.
Those are opposite profiles, and your two results went opposite ways.

That reading assumes a band's ρ² roughly adds over the harmonics it contains, which it only roughly does.
A plausible reason the downward tilt costs 3 pp: BETA's 8 to 15.8 Hz fundamentals overlap alpha, where background EEG is strongest. Upweighting the fundamental buys noise.

My bet: a rising weight on your three per-harmonic scores, the implied 1 : 1.5 : 1.9, beats the flat sum on BETA. 55 percent, since the implied profile shifts with stimulus frequency.

If your bands start at n×8 Hz too, does the rising profile beat flat, or is flat the top of that curve?
In this post