It’s a great line that gets to Roger’s version of Bond: he does what needs to be done for queen and country, but there’s plenty of things he’d rather be doing than killing people. That’s in contrast to Connery’s Bond who seems (to me, anyway) to enjoy a good killing as much as he does a good wine or a good suit. Or Craig’s Bond, who seems pretty much born to kill so we’re just lucky “M” found a way to point him in a helpful direction.
But in the same conversation, he rather rudely answers Scaramanga’s remark about settling things like gentlemen with, “I doubt you qualify on that score.” Based on what we’ve seen of Bond in the film, and pretty much the one before, that’s definitely a case of “Pot, meet kettle.”