You could use washers or some other variable width spacers to hold the knife board slightly away from the wall.
My fear with using felt or other soft material, is that it will compress over time causing the board to become loose. So I would use something more rigid, like metal washers.
When you hang the board put the mounting screws/bolts through the board, then the spacers, and then into the wall. You will end up with a gap between the wall and the board in spots, but the board should be solid and you won't have to worry about it dropping knives.

A more difficult solution would be to "hollow" out the back side of the board using either sand paper or a saw, to fit the contour of the wall. This would allow the board to fit snuggly against the wall, but has the down side of extra work, and requires that there is enough wood behind the magnet to allow the proper curve to be formed.
