X Window System is just metacpackage and it's never installed on my system, as I don't need some packages it suggest to install, so you can safely remove it from your installation.
This is description for X-Window-system:
X Window System
This metapackage provides substantially all the components of the X Window
System as developed by the X.Org Project, as well as a set of historically
popular accessory programs.