The X Window System was originally designed to work transparently across a network. One large server would actually run the X programs, but they would get displayed on various clients machines elsewhere on the network. The ability to remotely display programs can be a great advantage. The main disadvantage to this networking concept is that it is less secure than running applications on the local machine, and it takes a lot of network bandwidth to do. This is discussed later on in the section called Exporting displays.
Even when you're running X on your own machine, you are still dealing with the client-server model. The server is the video-card specific portion. When you configured X and told it what kind of video card you had, that was telling it what server program to use. The client portion is all other programs you run under X. A special client, called the window manager, is responsible for the look and feel of your particular X session. The window manager is discussed in detail later.
The job of a window manager is to handle drawing windows on the screen with programs in them, as well as handling input from the mouse and keyboard. The first window managers did that and not much else. Today's window managers are much more complicated programs and are customizable in just about any way imaginable. They have all sorts of fancy options that let your desktop look different from anyone else's.
Having several window managers really separates Linux from Windows on the desktop. Under Windows, you have the one basic windowing environment. Under Linux, you can run one of many different window managers, each with a different look and different features. Some people would call this a weakness, because there is no consistent look. However, most Linux users would call this a strength because you can configure your system as much as you want.