Nix can target any spell. The condition is checked only on resolution.
Nix doesn't check whether a spell's mana cost is 0. Rather, it checks how much mana was actually spent to cast it. It can counter a spell with mana cost 0 (such as
Slaughter Pact), a spell cast via an alternative nonmana cost (such as
Allosaurus Rider) or alternative cost of {0} (such as
Basking Rootwalla), a spell whose cost to cast it was reduced to 0 (such as
Frogmite), a spell cast without paying its cost due to an effect (such as suspend or
Mindleech Mass's ability), or a copy of a spell, among other possibilities. It will not counter a spell that had an additional or alternative mana cost paid to cast it, regardless of that spell's converted mana cost (such as a face-down spell cast with morph, or an Ornithopter cast while
Trinisphere is on the battlefield, for example).
Nix checks whether any player paid mana to cast the spell, not whether its current controller did. If Commandeer is cast on a spell, then Nix is cast on that spell, Nix won't counter it unless the player who originally cast that spell spent no mana to cast it.