I suspect the same as how ASCAP and BMI enforce it for music played in clubs, bars, and restaurants. The bookstore owner would buy a license based on the store's "footprint" and potential number of books. That license fee wold go into a "pool," which would then be distributed among publishers and writers, in theory.

It's still a stupid idea, because it is not at all like public performances of music. I'm just saying this is the most likely model they would use.