All I ever do for the walls, and ceiling anyway is just use a 3 or 4 in putty knife, either plastic or metal and run it over and get all the loose crude off.
With the depth of these things I also will use a long handled spatula for the deep reaches. You are right, you don't want to lose all the Goodness, just the loose stuff.