[He's annoyed again for reasons he can't even express. And isn't that the crux of it really?]

[Straw Hat doesn't want to be limited by anything, and Smoker just wants to limit him, as his job, as his duty, as some obscurer part of his moral code. Because that's how laws work. How the world works. How the universe should work.]

[He wants to punch him. To tell him to grow up, to stop being so selfish, or just be selfish without remorse.]

So you only want someone to want to do things with you, regardless of the fact that you want to be selfish and do everything and want everything, and with everyone, so that's why you're avoiding it entirely--

Do you even realize how ridiculous that is?!