Do not impose -Wextra -Wconversion etc on library users

I had to nuke the flags targetting spdlog_header_only because in CMake,
an interface target apparently really only supports the INTERFACE
option.

As I am only targeting Linux, the MSVC changes rely on spdlog's CI and
manual code review. That said, it makes sense to also stop forcing these
options on third party code, assuming that the CMake installations work
the same on Windows.

Fixes #1104