Consumer sleep trackers provide useful insight into sleep. However, large-scale performance evaluation studies are needed to properly understand sleep tracker accuracy. This study evaluated performance of an under-mattress sensor to estimate sleep and wake versus polysomnography in a large sample, including individuals with and without sleep disorders and during day versus night sleep opportunities, across multiple in-laboratory studies. One-hundred and eighty-three participants (51%/49% male/female, mean [SD] age = 45 [18] years) attended the sleep laboratory for a research study including simultaneous polysomnography and under-mattress sensor (Withings Sleep Analyser) recordings. Epoch-by-epoch analyses determined accuracy, sensitivity and specificity of the Withings Sleep Analyser versus polysomnography. Bland-Altman plots examined bias in sleep duration, efficiency, onset-latency, and wake after sleep onset. Overall Withings Sleep Analyser sleep-wake classification accuracy was 83%, sensitivity 95% and specificity 37%. The Withings Sleep Analyser significantly overestimated total sleep time (48 [81] min), sleep efficiency (9 [15]%) and sleep-onset latency (6 [26] min), and underestimated wake after sleep onset (54 [78] min). Accuracy and specificity were higher for night versus daytime sleep opportunities in healthy individuals (89% and 47% versus 82% and 26%, respectively, p < 0.05). Accuracy and sensitivity were also higher for healthy individuals (89% and 97%) versus those with sleep disorders (81% and 91%, p < 0.05). Withings Sleep Analyser performance is comparable to other consumer sleep trackers, with high sensitivity but poor specificity compared with polysomnography. Withings Sleep Analyser performance was reasonably stable, but more variable in daytime sleep opportunities and in people with a sleep disorder. Contactless, under-mattress sleep sensors show promise for accurate sleep monitoring, noting the tendency to over-estimate sleep particularly where wake time is high.