Hit Songs Choice Engine synopsis edition

Algorithm Information

CE — Consumer Surplus Formula

All features are scaled to [0, 1] (max-normalised; loudness min-max because decibels have no natural zero). The X factor is shown on every card but, as in the other explorers without a price, it is not added to the ranking score.

Adapting a logistic regression to the CE

Shadow prices. The CE needs one non-negative weight per feature measuring how much a unit of that feature is worth. A logistic slope on a [0, 1] feature is exactly that, in log-odds of being a year-end top-100 hit rather than dollars. Rankings do not depend on the overall scale of the weights, so log-odds units work as well as prices.

Negative slopes → reflection. Where a slope is negative (e.g. energy, once loudness and danceability are held fixed), the feature is replaced by its complement q′ = 1 − q ("calm", "sung not rapped", "clean", "not country"). The refit has an identical log-likelihood and exactly flipped signs, so every β ≥ 0 and every q′ stays in [0, 1]. This keeps self-rank-1 without zeroing features, which a β ≥ 0 constrained fit would have done to nine of them.

Artist history switch. The artist's previous chart entries and top-10 hits are strong predictors, but they describe the artist, not the song. With the switch off (default) the engine uses weights from models refitted on song features alone, and "hit potential" is that song-only model's fitted probability. With it on the engine uses the full model: the two track-record features join the CE score as continuous terms (entries reflected to 1 − q, since more prior entries with the same top-10 count lowers the odds), so candidates whose artist has a similar track record gain utility and dissimilar ones lose it. Hit potential then includes track record. VSM adds the same two dimensions when the switch is on. Within-year AUC (in-sample): song only 0.647, with artist 0.668; out of sample the top decile's hit rate is 42.8% vs 46.1%.

Chart-year switch. The logit model includes a fixed effect for each chart year, because the share of charting songs that make the year-end 100 varies with how crowded the chart was: about 18% in the 1960s and 2010, 28–30% in the 1990s–2000s. With the switch on (default), hit potential is the model's fitted probability for the song's own year. With it off, each song's year effect is swapped for the average one, so hit potential compares songs on their features alone, across decades. The CE ranking never uses the year effect (it sits in the X factor, which is displayed but not ranked), so only hit potential and its column sort change. The chart-points model is already measured relative to each chart year and is unaffected.

Flag scoring switch. The original CE gives a shared 0/1 flag a bonus of ½·b·q_R·q_C and nothing else. A dyadic analysis of all 319 million ordered pairs (logit weights, song features only, era off) found this makes hubs: songs that carry every hit-favoured flag at once (pop, rock, easy listening and dance tags; minor key; 4/4; clean; no soul or country tag) collect bonuses from any reference that shares any of them. One song appeared in 224 other songs' top 10s. The top 1% of songs held 10% of all top-10 slots, and 13% of songs never appeared in any. Switched on, flags use the same quadratic form as continuous features, b·q_R·q_C − ½·b·q_C²: +½b when both songs have the flag, −½b when only the candidate does, 0 otherwise. Hubs disappear (the top 1% hold 2.7% of slots, as under VSM). CE and VSM top 10s share 4.8 songs instead of 1.7. The CE's tilt toward hits among its top-10 suggestions (30% vs a 24% base rate) goes away, showing it came from the hubs. Self-rank-1 holds under both forms.

Indifference. A price-based engine converts the CS gap to dollars. Here CS is in log-odds, so the gap converts to an odds multiplier: e^{(CS_R − CS_C)} is how many times better the candidate's odds would need to be to compensate a listener of the reference for the feature mismatch.

Alternative: NNLS on chart points

DV = log(total Hot 100 points, 101 − weekly position summed over the run), measured relative to the song's chart-year mean. Same features and the same reflection rule, then non-negative least squares with a free intercept. X = intercept + track-record terms + residual, so X is a proper over/under-performance reading. The gap multiplier is in chart points. Adj R² is only 0.046: audio features explain little of how long a song charts, so treat this as a comparison, not the headline.

Era

Era enters as one more continuous CE feature, e = (year − 1960) / 50, so it lies in [0, 1]. Min-max scaling is used because dividing years by their maximum would squeeze 1960–2010 into 0.975–1. Like every continuous term, the quadratic peaks when e_C = e_R: the loss relative to the reference is ½·b_era·(e_R − e_C)². A candidate gets the most utility from being in the reference's own year, and the penalty grows with the square of the distance.

b_era cannot be estimated from either regression, because both measure success within a chart year. The year effect absorbs era completely. So b_era is a listener-preference setting, like the constant-β engines, and is set by the penalty for a 10-year gap: b_era = ln(k) / (½·0.2²). Quadratic means 20 years costs k⁴ and 30 years k⁹. Feature gaps between a song and its nearest neighbours are small (about 0.01–0.02 in log-odds), so even k = 1.02 pulls the top 20 from a median of about 6 years away to about 2. The default is "Light", so sound still drives the ranking. VSM does not use era.

Synopses and lyric signals

Coverage. Every one of the 17,858 songs was summarised from general knowledge only, blind to how it charted, with explicit permission to answer "don't know". 13,251 have a synopsis (74%), including 98% of year-end hits and 99% of top-10 songs; songs not known well enough are left blank rather than guessed. Hooks were described for 8,908 songs.

Checks. A 300-song pilot agreed with the 120 source-coded songs of the lyrics study at κ 0.75 (theme), 0.75 (mood), 0.71 (narrator), 0.76 (signature sound) and 0.80 (single order). In the full run, 135 synopses (45 per chart-peak band) were fact-checked against Wikipedia, Songfacts and reviews by agents that did not write them: 85 correct, 27 right in gist with a detail off, 23 unverifiable (sources say too little), none wrong. Codes checked against sources: theme 94%, mood 97%, narrator 96%, single order 76%.

Chips. Colours come from a hit model fitted on the 8,799 songs the model knew well (3,754 year-end hits): audio + genre + lyric codes, year fixed effects, standard errors clustered by artist, and only traits with a Benjamini–Hochberg q below 0.05 are coloured. Green (more common in hits, other things equal): happy mood (odds ratio 1.30), a signature riff or sound (1.87), second single from its album (1.22, vs lead single). Red (more common in misses): angry mood (0.61) and sad mood (0.77) vs bittersweet; songs about faith (0.52), social or political issues (0.56), places and travel (0.59), or novelty (0.65) vs love songs. Four signals from the earlier 60-pair test did not survive at scale and are now grey: desire theme, playful narrator (now leans the other way), accusing narrator and standalone single. Caveat: the model may remember famous songs' riffs better, which could inflate the signature-sound result; among songs that all peaked in the top 10 it still holds (odds ratio 1.52). The chips do not enter the CE ranking.

Hook descriptions come from a separate knowledge-only pass and cover both the sound (what the hook is, where it lands, what makes it stick) and the idea (what the line conveys), with no lyric quotes. Where a recording wasn't remembered well enough, the line says so. In the fact-check, 88 hook lines could be verified: 46 correct, 42 right in gist with a detail off, none wrong. Treat hook lines as a good description rather than a transcription.

VSM — Vector Space Model

Cosine similarity on the unreflected [0, 1] feature vector (10 audio features + 10 flags, plus the 2 track-record features when artist history is on), unweighted. It is the same under both weight sets.

Weights

Invariants

Data

Billboard Hot 100 weekly charts 1960–2010, Billboard year-end top 100 lists, Spotify audio features and artist genre tags. Songs without audio features or genre tags are excluded (17,858 of 19,265 remain). Logit: year fixed effects, standard errors clustered by artist. A song's year is the year it earned most of its chart points.