Chess Variants - Ancient, Regional, and Modern by John Gollon is a book published in 1968 that describes 33 different versions of chess, and includes sample games for most of them. It is the closest source I have yet encountered to answering the 'why chess' questions.
In the author's words, it is intended to be a Hoyle's Book of Games, but for chess. It also asrved as a humbling reminder to look outside university resources sometimes. The Bennett library may have nearly every book one could want on statistics, but it has perhaps 10 on chess, and those are mostly about artificial intelligence.
There's a lot of useful information from this book, much of the subtler implications are beyond me. Here's the gist of what I learned:
The earlier known chess comes from India and was called Chaturanga. It came in 2-handed and 4-handed (i.e 2-player and 4-player) formats, both played on an 8x8 grid. In the 4-player version, each player had 8 total pieces and starts in a 2x4 square in the corner. The 2-player version looks, on the surface, surprisingly close to modern chess. However, some pieces are much weaker than they are now. The equivalent of the bishop moves exactly 2 spaces diagonally. The equivalent of the queen moves only 1 space diagonally. The pawn is the same, except it can't move 2 spaces in its first move. The knight, and even the rook are the same as they are in the modern game.
Throughout all the versions shown in this book, the knight changes little, occasionally getting 'stretched' into a 3-and-1 move, or even an unlimited-and-1 move. The rook, however, is extremely stable and unchanging, like some sort of castle.
The next set of versions covered are the Arabic adaptations of Chaturanga, called Shatranj. In several of these versions, an oblong four-sided die is used to determine which type of piece a player may move on that turn. For example, a player may only move their rook whenever a '3' is rolled. In these dice-based versions, a player has the option of making no move at all, as they would otherwise be often forced to move into a worse position, or to move a piece they don't have. Amongst the Shatranj variations, other board sizes are considered, including a 4x16 board where each player starts with two rows of pawns in the from.
In Timur's chess and a couple other versions like citadel chess, the board has two single-square offshoot spaces in opposing corners. These spaces are called 'bastions' or 'citadels', and the restrictions around them or the rewards for reaching them changes from game to game. Typically they offer defensive advantages and/or promotions for secondary king-like pieces called princes. In citadel chess, if the king reaches an opposing citadel, a draw is forced.
In 'Hsang Chi', or Chinese Chess, there are the same staple of weaker versions of bishops and pawns, pieces similar to knights and modern rooks, However, the board is 9x9 and the 5th rank is entirely occupied by a central river. On each side of the river is a 3x3 fortress. The king piece cannot leave the fortress, and certain pieces cannot cross the river. There is a 3-player version of this played on a hexagon in which the river forks in the centre of the board and each player has a fortress. In both the two- and three-player versions, pieces are placed on the intersections of the lines, rather than in the middle of the squares. Playing on the lines rather than the spaces is especially useful in the 3-player version because, by necessity, the spaces are not squares of equal size. Korean chess is similar to 'Hsang Chi', except there is no central river.
Burmese chess has a 'draft phase' of sorts, which precedes the part of the game where pieces are moved.
The draft phase starts with both players having a row of pawns in the front of their territory (the two ranks nearest them). Then players alternate placing their other pieces - either by putting them in the back rank or replacing a pawn in the front rank. The removed pawn goes into reserve to be placed later in the draft. Even after all pieces have been placed, a player can continue the draft phase by removing a piece to place it elsewhere. The draft continues until one player makes a move, after which play proceeds only with moving.
This 'draft phase' seems unique, and I'm surprised to not see it elsewhere. However, there's variant currently being sold called 'Shuuro' that advertises a similar mechanic.
Apparently there are more variants buried in fiction. Edgar Rice Burroughs, author of Tarzan, also described a game of Martian Chess in one of his other books. A skim of the Wikipedia article on this game, also called 'Jetan', reveals four other variants in fiction books.
Finally, in the 'modern' section of the book, the author describes two versions of chess with the modern orthodox pieces and two additional pieces: the chancellor who has the move the of the rook and the knight, and the cardinal who has the moves of the bishop and the knight. The author predicts that if the predominant version of chess ever changes, it would be to one of these other versions. He considers a natural progress to the modern version, also called 'queen's chess' because it was marked by the creation of the modern queen - a combination of the rook and bishop.