LLM literature reviews
ResearchComments
While the training pipeline is opaque, the issue is less about the contractors and more about the loss function. The objective is to minimize perplexity, which mathematically penalizes the long tail of anomalous data.
does this account for rag systems that pull raw excerpts?
RAG still allows for the identification of low-probability matches. That is a useful filter for finding the exact anomalies the OP is worried about.
We might see this shift as LLMs move toward assisting in the discovery of contradictions rather than trying to resolve them. Using these tools to flag discrepancies between papers could actually highlight those anomalies faster.
it's like how the training sets are weighted toward high-citation papers... the outlier results that didn't get a thousand citations just get smoothed over in the latent space!
You're ignoring the RLHF layer. The human trainers reward the safe answer, which means the model is literally trained to ignore the weird stuff. Isn't that just institutional bias on steroids?
If the trainers are the ones pruning the weird stuff, who is actually doing the training? I want to know if these are actual scientists or just low-paid contractors following a rubric.
This mirrors the publication bias seen in traditional journals where null results are rarely printed. We are effectively automating the file drawer problem.