I taped off the inlay on the spine and on the tail to protect them from the filler and painted the filler first on the back and then later on the sides. Here's the before and after on the back. The blue stripe is the tape over the inlay before the filler was applied. The second picture was after the back had been scraped and sanded, leaving it much as before, but with the filler in all the holes.
You can see that the appearance did not change much, but you can really tell a difference when you run you hand across the back. It is much, much smoother now.
I did the same process on the sides. Here's a picture with the filler applied.