Previousl: Chapter 2. Equilibrium Statistical Mechanics
The Clausius inequality is a criterion for spontaneous processes, and thus in classical thermodynamics, it is connected with the arrow of time. The Hamilton laws of motion are time-symmetric, and the search for an analogue of the Clausius inequality in statistical mechanics is still ongoing. In this and next chapters the main arguments to obtain time asymmetry in statistical mechanics are considered, including those based on entropy’s subjectivity and its connection to information.
In this chapter, we examine the first steps of the search for the arrow of time, which began after the development of the kinetic theory in the second half of the 19th century. Interestingly, James Maxwell immediately recognized the impossibility of this endeavor due to the time symmetry of the Hamilton equations. In a letter to Peter Tait in 1873, Maxwell ridiculed such attempts of Clausius and Boltzmann:
‘But it is rare sport to see those learned Germans contending for the priority of the discovery that the 2nd law of θΔcs [thermodynamics] is the Hamiltonsche Princip … The Hamiltonsche Princip, the while, soars along in a region unvexed by statistical considerations, while the German Icari flap their waxen wings in nephelococcygia [cloud-cuckoo-land] amid those cloudy forms which the ignorance and finitude of human science have invested with incommunicable attributes of the invisible Queen of Heaven.’
Nevertheless, Ludwig Boltzmann’s work led to interesting results, including the well-known statistical foundations of the second law. We examine the evolution of Boltzmann’s views, and discuss two paradoxes that appeared during his attempts to prove the Clausius inequality: the Loschmidt paradox of mechanical reversibility and the Zermelo paradox, associated with Henri Poincaré’s recurrence theorem.
Boltzmann’s statistical foundations of entropy was limited to an ideal monatomic gas. Gibbs introduced statistical entropy in the general case for an arbitrary system. Unfortunately, this statistical entropy formally remains constant in an irreversible process. Thus, in the Gibbs formalism, there is a contradiction between the formal mathematics and the second law of thermodynamics. Gibbs was the first to notice this issue and he proposed possible solutions. In this chapter, a general discussion of Gibbs statistical entropy is given; the similarities between Gibbs entropy and Shannon information entropy are considered in the next chapters.
- The evolution of Ludwig Boltzmann’s views
- Loschmidt paradox and statistical foundations
- Zermelo paradox and fluctuation hypothesis
- Boltzmann’s Suicide
- Statistical interpretation of the second law
- Gibbs statistical entropy
- Phase space mixing
The evolution of Ludwig Boltzmann’s views
Ludwig Boltzmann (1844-1906) was a leading figure in the development of the kinetic theory in the 19th century. As a young man, Boltzmann believed that it was possible to find a rigorous foundation for irreversible processes within the framework of the kinetic theory. However, due to paradoxes he later changed his position and suggested a statistical interpretation for the second law. The final step was the development of the cosmological fluctuation hypothesis, which has since then been transformed by modern physicists into the concept of the ‘Boltzmann brain’.
Boltzmann’s first paper, ‘On the mechanical meaning of the second law of thermodynamics‘, was published in 1866, when he was 22 years old. The introduction of the paper shows that Boltzmann had no doubt that the goal could be achieved:
‘The purpose of this paper is to provide a purely analytical, complete, and universal proof of the second law of thermodynamics, as well as to find the corresponding theorem of mechanics.’
Rudolf Clausius made a similar attempt in 1870 and 1871. Clausius independently arrived at the results previously published by Boltzmann, and there was a dispute over priority between them — Maxwell’s quote above referred to this time.
After that, Boltzmann tried to find the best proof of Maxwell’s velocitiy distribution. Boltzmann wanted to prove that any initial state lead to Maxwell’s distribution. The first solution was proposed in the 1872 paper ‘Further studies on the thermal equilibrium among gas molecules‘. Boltzmann derived an equation that would later be known as the Boltzmann kinetic equation, which describes the evolution of a gas in a non-uniform state over time.
This Boltzmann kinetic equation is still used today to describe the kinetics of rarefied gases. It was later shown in the 1890s that the collision hypothesis in this paper (molecular chaos, Stosszahlansatz) violates time symmetry. In other words, the derivation of this equation contained an extra hypothesis that was incompatible with time-symmetric laws of mechanics. This approach has remained in non-equilibrium statistical mechanics to the present day, where additional hypotheses are deliberately introduced to achieve time asymmetry and hence obtain correct kinetic equations.
In parallel, Boltzmann introduced the H-function and proved the H-theorem. According to the theorem, the derivative of the H-function is negative and it reaches zero only when the gas arrives at the Maxwell distribution. Thus, the H-function with the opposite sign is analogous to the entropy of the system:
‘Thus, it has been rigorously proved that, whatever the initial distribution of kinetic energy may have been at the beginning, after a very long period of time it must necessarily approach that discovered by Maxwell. What has been done so far, however, is nothing more than a mathematical device to rigorously prove a theorem whose exact proof has not yet been achieved’
Loschmidt paradox and statistical foundations
It was the H-theorem that became the subject of further discussions. The first objection was made by Loschmidt in the 1876 paper (the Loschmidt paradox of mechanical reversibility). Loschmidt was a senior colleague of Boltzmann, and they had a long-standing friendship. Loschmidt proposed to reverse the velocities of atoms in a system that had reached an equilibrium state. According to the laws of mechanics, this would cause the system to move away from equilibrium, resulting in an increase in the H-function in contradiction with the H-theorem.
Of course, such an action cannot be carried out practically, but Boltzmann has made a mathematical proof based on the laws of mechanics. Therefore, from the point of view of mathematics, it was impossible to ignore the Loschmidt proposal. As a result, Boltzmann revised his views and gave a new statistical interpretation of entropy and the H-function in the 1877 paper ‘On the relationship between the second law of mechanical heat theory and probability theory in thermal equilibrium theorems‘.
This paper introduces the well-known Boltzmann equation for the system entropy with a probabilistic interpretation, as described below in the section ‘Statistical interpretation of the second law‘. It is assumed that a macrosystem corresponds to many different microstates, and that the number of microstates is proportional to the probability to observe the macrosystem:
‘If we apply these to the second law, the quantity that we usually refer to as entropy can be identified with the probability of a respective state. … The system of bodies we are talking about is in some state at the beginning of time; through the interaction of the bodies this state changes; according to the second law this change must always occur in such a way that the total entropy of all the bodies increases; according to our present interpretation this means nothing other than the probability of the overall state of all these bodies becomes ever greater; the system of bodies always passes from some less probable to some more probable state.’
Zermelo paradox and fluctuation hypothesis
Now the next problem appears. The behavior of the gas is related to a single system whose behavior is deterministic according to the equations of mechanics. Thus, a discussion of the behavior of the H-function within the framework of deterministic laws of mechanics is required. These issues were further discussed in the journal Nature in 1894-1895. In the late 1890s, there was a discussion between Boltzmann and Zermelo regarding Poincaré’s recurrence theorem.
It has been proven that an isolated mechanical system in an arbitrary state eventually returns to the state that is almost identical to the original state. For typical systems, the return time is extremely long, and Poincaré’s theorem is primarily of academic interest. However, like other 19th-century scientists, Boltzmann thought about the heat death of the universe.
Let me remind you that at that time the Big Bang and the expansion of universe have not been known, and the universe was assumed to be static. This was one of the reasons for the cosmological fluctuation hypothesis of Boltzmann, which he presented as a response to Zermelo in the 1897 paper ‘On Mr. Zermelo’s treatise “On the mechanical explanation of irreversible processes” ‘. In an exaggerated way, Boltzmann’s position can be presented as follows: from the presence of fluctuations to the world as a fluctuation.
Boltzmann believed that the universe has existed forever. In this case, the heat death of the universe implies that the universe, at the moment of observation, must already be in a state of thermal equilibrium. Therefore, it is hard to explain the world we observe. To resolve this contradiction, Boltzmann turned to fluctuations and proposed that the world we live in is a gigantic fluctuation. The probability of such a fluctuation is extremely small, but not zero. A quote from Boltzmann’s paper:
‘In the universe, which is everywhere in thermal equilibrium, and therefore dead, there must be here and there such relatively small regions the size of our stellar space (let us call them individual worlds) that deviate significantly from thermal equilibrium during the relatively short time of eons, namely, among these worlds, states whose probabilities increase and decrease are equally common. For the universe, therefore, both directions of time are indistinguishable, just as there is no up or down in space. But just as we designate the direction towards the earth’s center as downwards at a particular point on the Earth’s surface, so a living being which is in a particular time phase of such an individual world will designate the time direction towards the more improbable states differently than the opposite (the former as the past, the beginning, the latter as the future, the end) and by virtue of this designation, for the same being, small regions which it isolates from the universe will “initially” always be in an improbable state.’
In modern physics, there is interest in smaller fluctuations, which have been dubbed ‘Boltzmann brains’. Such fluctuations should occur much more frequently in an equilibrium universe than giant ones, and they allow physicists to consider consciousness without resorting to biologists. Sean Carroll, in his article, provides a colorful description of the Boltzmann brain:
‘Out of the background thermal equilibrium, a fluctuation randomly appears that collects some degrees of freedom into the form of a conscious brain, with just enough sensory apparatus to look around and say “Hey! I exist!”, before dissolving back into the equilibrated ooze.’
Boltzmann’s Suicide
Boltzmann’s suicide in 1906 is often attributed to the dark forces. However, there are no grounds for such a conclusion. Boltzmann’s scientific career was successful, his work became widely known during his lifetime, and he had no problems with universities. To mark Boltzmann’s sixtieth birthday, a commemorative collection with papers of 117 scientists from around the world was published. Even ideological opponents Ernst Mach and Pierre Duhem contributed papers to this collection in honor of Boltzmann’s anniversary.
At the same time, Boltzmann’s health was deteriorating, and he experienced periods of severe depression. Carlo Cercignani, in his book ‘Ludwig Boltzmann: The Man Who Trusted Atoms‘, suggests that we can only speculate about what happened in Boltzmann’s final days and offers a speculative scenario, asking us not to take it seriously:
‘If this were a novel or a tragedy, the author could imagine that reading the paper by Einstein on the equivalence between mass and energy convinced Boltzmann of the failure of the efforts of his life to show that atoms were real. Was not one of the tenets of energetics that everything was energy? Had not Boltzmann maintained … that energetics would not be able to explain mass in terms of energy? Perhaps the author of a tragedy could also imagine that, just after Boltzmann had strangled himself, a character would enter the stage carrying a copy of the previous paper by Einstein on Brownian movement, showing that the existence of atoms could be shown experimentally.’
Statistical interpretation of the second law
Boltzmann proposed to associate the entropy of an out-of-equilibrium state of a monatomic ideal gas with the number of microstates corresponding to a given macrostate. Let us briefly consider Boltzmann’s method that led him to his famous equation. Boltzmann divided the six-dimensional μ-space of an ideal monatomic gas with N atoms into separate cells by discretizing the values of coordinates and impulses. The state of all atoms In a single cell is considered to be the same – the average values of energy, coordinates and impulses are taken. The number of cells is equal to p, and it is required that the number of cells is much smaller than the number of atoms.
Now, N atoms are placed in p cells, and this gives a set of cell occupation numbers: {n1, n2, n3, …, np}; the sum of all occupation numbers is equal to N. This state is referred to as the macrostate; it describes how many molecules have given energy, coordinate, and momentum values in the cells. Now let us enumerate atoms; this is a transition to a microstate. Replacing two atoms in different cells changes the microstate but does not affect the macrostate. Thus, the number of microstates corresponding to a macrostate is determined by the number of possible permutations. This in turn leads to the expression for W(E, V) in the Boltzmann equation for entropy:
The equilibrium state corresponds to the maximum value of W; the search for a maximum for given values of total energy and volume leads to the Maxwell-Boltzmann energy distribution. All other macrostates have smaller number of microstates and hence they are less probable as the equilibrium macrostate.
Such statistical interpretation of the second law played a major role in the development of statistical and quantum mechanics, but unfortunately it is often presented as the final answer to the question of what entropy is. This leads to the persistent but incorrect metaphors about entropy, such as entropy as the number of permutations and entropy as disorder. Below is a list of limitations for the Boltzmann cell method.
At present, it is necessary to keep in mind the indistinguishability of atoms. This was already recognized by Gibbs from the requirement of entropy additivity and was further confirmed in quantum statistics. In the quasi-classical approximation, this is achieved by dividing by N!, which, however, makes it difficult to interpret the Boltzmann cell method as the ‘number of permutations of microstates’.
The Boltzmann cell method does not consider kinetics. Non-equilibrium entropies of macrostates are analogous to the entropies of non-equilibrium states in classical thermodynamics not taking into account the time. In classical thermodynamics, time is not considered anyway, but statistical mechanics is assumed to be a complete solution also for non-equilibrium thermodynamics. Therefore, the connection between statistical interpretation of entropy and kinetics remains an open question.
The macrostate in the cell method is not yet a macrostate in classical thermodynamics. Additional effort is required to find the temperature and pressure fields in the macrostates introduced by Boltzmann. Moreover, new non-equilibrium states appear in statistical mechanics that are absent in non-equilibrium thermodynamics; see the last chapter.
In conclusion, the Boltzmann cell method is limited to the ideal monatomic gas and cannot be used in the presence of interactions between atoms and molecules. Thus this cannot in principle give a universal answer to the question of what entropy is.
Gibbs Statistical Entropy
Gibbs proposed a general framework to consider an arbitrary systems with interacting atoms and molecules. Ludwig Boltzmann praised Gibbs’ book; in the lecture ‘On Statistical Mechanics‘ in 1904 the role of Gibbs was emphasized:
‘The merit of having systematized this system, described it in a sizable book and given it a characteristic name belongs to one of the greatest of American scientists, perhaps the greatest as regards pure abstract thought and theoretical research, namely Willard Gibbs, until his recent death professor at Yale College. He called this science statistical mechanics.’
As already mentioned, Gibbs introduced an ensemble of systems to visualize the probability density in a multidimensional Γ-space. A deterministic trajectory of a mechanical system is represented as a line, and a single point represents the current state of the system (the coordinates and momenta of all particles). The Gibbs ensemble allows for a frequency interpretation of the probability density. In the development of statistical mechanics, it was the Gibbs method that was extended to quantum mechanics, that in turn formally justified the quasi-classical approximation.
However, in the Gibbs method, the arrow of time problem remains. The change in probability density in time is governed by the Liouville equation, which is derived from the Hamilton equations of motion; as a result, the Liouville equation is also symmetric with respect to time. Therefore, obtaining kinetic equations that tend to reach equilibrium states necessarily requires additional hypotheses. In the BBGKY (Bogoliubov-Born-Green-Kirkwood-Yvon) chain of equations, a path was proposed for deriving kinetic equations by introducing time asymmetry at the final stage. The Boltzmann kinetic equation is a special case of the BBGKY chain with ‘molecular chaos’ hypothesis at the final stage.
Another successful application of non-equilibrium statistical mechanics is related to the development of the linear response theory to assess the transport properties. In this case, time asymmetry was also deliberately introduced during the solution process. An expressive quote from R. Peierls’ lectures on the theory of transport processes quoted from Zubarev’s book is below:
‘In every theoretical study of transport processes, it is necessary to clearly understand where irreversibility is introduced. If it is not introduced, the theory is incorrect. An approach that preserves time-reversal symmetry inevitably leads to zero or infinite values for transport coefficients. If we do not see where irreversibility has been introduced, we do not understand what we are doing.’
At the same time, physicists want to find a general solution to the arrow of time in statistical mechanics without additional hypotheses. The fundamental laws of physics are time-symmetric, but since these laws are fundamental, a general solution must be found at this level. Additional hypotheses imply that the fundamental laws of physics are incomplete, which is unacceptable to true physicists.
Gibbs introduced the system entropy as the average logarithm of the probability density; this is a generalization of the Boltzmann H-function. Below there is the equation for Gibbs entropy in the discrete case, this is necessary for the discussion of Shannon’s information entropy later:
This is the sum over all energy levels of the system; the average logarithm of the probability of the system is converted to entropy by multiplying by the Boltzmann constant and by changing the sign, similar to the case of the Boltzmann H-function.
In the equilibrium state, the system obeys the Gibbs energy distribution, and the equation above leads to the correct entropy associated with the partition function. Thus, in equilibrium systems, there are no issues with the Gibbs statistical entropy; actually this equation is used to derive the relationship between the partition function and the Helmholtz energy of the system.
However, due to the peculiarities of the Liouville equation (the Liouville theorem), when the probability density changes, the average logarithm of the probability density remains constant. As a result, the Gibbs statistical entropy remains formally constant during an irreversible process in an isolated system, which contradicts the second law of thermodynamics.
Gibbs understood the limitation of the statistical entropy he introduced. In his book, ideas were proposed as possible solutions: phase volume mixing and the need to change the order of averaging to estimate entropy. These ideas were expressed as equations in the paper ‘The Conceptual Foundations of the Statistical Approach in Mechanics‘ by the Ehrenfest couple in 1912, where the concepts of coarse-grained and fine-grained probability density, as well as coarse-grained and fine-grained Gibbs statistical entropy, were introduced. The coarse-grained Gibbs statistical entropy uses the coarse-grained probability density in an isolated system and thus increases as expected.
Phase space mixing
One argument for the arrow of time in the general form is related to Gibbs’ idea of phase space mixing. Mathematicians introduced a formal definition of this process (Hopf, 1937) and found a distinction between mixing and ergodicity (the idea that the time average is equal to the ensemble average). A number of theorems were proved, and the ideas of mixing were combined with the development of the formalism of deterministic chaos. Below there are quotes from Mukhin’s dissertation ‘Development of the Concept of Dynamic Chaos in the USSR‘:
‘Mixing systems formed the basis of Nikolai Sergeevich Krylov’s pioneering work on the foundations of statistical mechanics, a very talented and early-deceased student of V. A. Fock. … According to Krylov, “… the laws of statistics and thermodynamics exist because for statistical systems (which are mixing-type systems), the uniform law of distribution of initial microscopic states within the empirically determined region of phase space ΔГо is valid. … In this work, the concept of ergodicity is not considered. We reject the acceptance of the ergodic hypothesis. We proceed from the notion of motions of the mixing type. … Such a mixing is due to the fact that in the n-dimensional configuration space, trajectories close at the beginning diverge very quickly, so that their normal distance increases exponentially».’
‘Krylov’s idea is expressed quite clearly. At the heart is the concept of mixing, which can be used to describe the physical process of relaxation — the transition of a system to a stationary state, regardless of its initial state. Since Gibbs, the idea of the need for mixing in statistical mechanics has been repeatedly proposed, but it is probably Krylov who first connected mixing with the local characteristic of motion in such systems — exponential instability.’
Unfortunately, it is impossible to provide a general proof of the existence of mixing for all systems considered in physics, and a simple reference to the large number of particles in the system is insufficient:
‘In physics, it has been common to think that in systems with a large number of degrees of freedom, such as systems of statistical mechanics, the transitive case and mixing are of primary importance, while systems with a small number of degrees of freedom exhibit regular behavior. Kolmogorov notes that this idea appears to be based on a predominant focus on linear systems and a small set of integrable classical problems, and that these ideas have limited significance. Kolmogorov’s key idea is that there is no gap between two types of behavior – regular and complex, irregular – multidimensional systems can demonstrate regular motion, and systems with a small number of degrees of freedom can be chaotic.’
Thus, the use of mixing with deterministic chaos to explain the arrow of time remains at the level of a general argument without a complete mathematical proof, although such an explanation is occasionally found in textbooks.
At the same time, a number of physicists accepted the Ehrenfest couple’s explanation with coarse-grained entropy as evidence of the subjectivity of entropy. Thus, entropy and probabilities in statistical mechanics became associated with human ignorance about the actual state of the microscopic level. This was further influenced by the Shannon’s information theory that appeared in 1948, where the expression for information entropy is similar to that of the Gibbs statistical entropy. In the next chapters, we examine this development, and in the last chapter, we discuss the entropy of non-equilibrium states in statistical mechanics.
Next: Chapter 4. Entropy, Ignorance, and Information
References
Martin J. Klein, Maxwell, His Demon, and the Second Law of Thermodynamics: Maxwell saw the second law as statistical, illustrated this with his demon, but never developed its theory. American scientist 58, no. 1 (1970): 84-97.
Ya. M. Gelfer, History and Methodology of Thermodynamics and Statistical Physics (in Russian), 2nd ed., 1981, Part Three. Synthesis of Thermodynamics and Molecular-Kinetic Theory. Emergence and Development of Statistical Physics.
Jos Uffink, Boltzmann’s Work in Statistical Physics, The Stanford Encyclopedia of Philosophy (Summer 2022 Edition), Edward N. Zalta (ed.) First published 2004; substantive revision 2014.
L. Boltzmann, Wissenschaftliche Abhandlungen, Vol. I, II, and III, 1909. Über die Mechanische Bedeutung des Zweiten Hauptsatzes der Wärmetheorie (1866, v. I, paper 2, p. 9-33). Weitere Studien über das Wärmegleichgewicht unter Gasmolekülen (1872, v. I, paper 22, p. 316-402). Über die beziehung dem zweiten Haubtsatze der mechanischen Wärmetheorie und der Wahrscheinlichkeitsrechnung respektive den Sätzen über das Wärmegleichgewicht (1877, v. II, paper 42, p. 164-223). Zu Hrn Zermelos Abhandlung “Über die mechanische Erklärung irreversibler Vorgänge” (1897, v. III, paper 120, p. 579-587).
Sean Carroll, Boltzmann’s Anthropic Brain, Aug 1, 2006, Discover Magazine.
Carlo Cercignani, Ludwig Boltzmann: the man who trusted atoms, 1998.
Ludwig Boltzmann, Theoretical Physics And Philosophical Problems Selected Writings, 1974, On Statistical Mechanics (1904), p. 159-172.
D. N. Zubarev, V. Morozov, G. Röpke, Statistical Mechanics of Nonequilibrium Processes, 1996.
R. R. Mukhin, Development of the Concept of Dynamic Chaos in the USSR. 1950-1980s (in Russian), 2010.
Discussion