Read the signal.
Ground the language.
EEG-Captioner: Grounding EEG-to-Language Generation in Fundamental Signal Properties
Beyond fluent descriptions. Connecting EEG waveforms to language through measurable properties of the signal itself.
One example EEG segment, five ways to ask about it. Choose a task to see what is compared and how an answer is expressed.
Each row is an electrode channel: Fp1/Fp2 (frontal), T3/T4 (temporal), O1/O2 (occipital). Time runs left to right; vertical movement shows the signal amplitude. Only these six channels are shown.
These are the paper's five task families, not five EEG signal categories. This walkthrough uses a single synthetic signal and authored answers to explain the tasks. It does not serve TUEG recordings, original captions, or live model predictions.
Does the language
reflect the EEG?
A plausible caption is only part of the story. Signal-level questions probe whether a model can compare frequency, location, time and hemispheric activity. EEG-Captioner combines these questions with caption supervision in a unified framework.
Dominant band
Compare power across frequency bands.
Strongest region
Locate the highest regional power.
Peak period
Compare early, middle and last intervals.
Temporal pattern
Follow how a feature changes over time.
Lateral dominance
Compare corresponding left and right activity.
160,781 segments
25 candidate QA conditions per 10-second segment, filtered for validity.
Two domains. One grounded representation.
Temporal waveforms and spectral-spatial representations provide complementary views of the same EEG.
20 channels
10 s at 200 Hz
Temporal waveform branch
+ spectral-spatial branch → fusion
Projector + LoRA
Qwen2.5-Omni-7B
Signal-level answers
and coherent descriptions
Joint supervision,
complementary benefits.
QA macro balanced accuracy
| Language adaptation | QA Macro-BA ↑ | Factual consistency ↑ |
|---|---|---|
| Caption only | 47.82% | 82.06% |
| Signal-level QA only | 86.06% | 44.10% |
| QA + caption | 88.81% | 83.46% |
Paper-reported results, not a fresh reproduction. Historical QA evaluation used a validation/test diagnostic set for checkpoint selection. Factual consistency is an external GPT-based assessment. See the repository's reproduction notes for protocol and training differences.