Switch off the options in the default configuration that require X11 libraries. This includes client-side font configuration and SSH forwarding of X11 authentication in. Thus, you probably do not want to enable this option if you want to run X11 programs on this machine via SSH.
option