The problem was that nodes, while registered with kubeadm init were providing their private IPs to the cluster master. This caused problems, because master was trying to reach 192.0.*.* addresses which were not resolved as nodes from it's point of view.
I needed to edit /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and specify public IP of the node on --node-ip=<public-node-ip> parameter. The reload the service and restart it as mentioned in https://github.com/kubernetes/kubeadm/issues/203#issuecomment-335416377
Then I registered nodes again and everything was working fine.