stable distributions, Machine learning.

The most justifiable version of information entropy is the Rényi entropy with a free Rényi non-extensivity parameter q, and the Tsallis entropy can be thought of as a linear approximation [1] to the Rényi entropy when q≈ 1. When q → 1, the Boltzmann-Shannon entropy function replaces both other entropy functions. When the Rényi entropy functional is subjected to the Maximum Information Entropy (MEP) principle, the result is the microcanonical (Homogenous) distribution for an isolated system. The Boltzmann entropy functional replaces the Rényi entropy functional in this situation, which supports the universality of Boltzmann's principle of statistical mechanics, regardless of the value of the Rényi parameter q.

The need for non-extensive statistics based on Tsallis' information entropy is critical, given its quick development. The one-parameter family of Rényi entropies (Or just Rényi entropy) seems to be the most rational one [2]. The well-known Boltzmann-Shannon entropy functional replaces the Rényi entropy functional when the Rényi parameter q is equal to unity. The non-extensive Tsallis' entropy functional is produced by linearizing the extended Rényi entropy functional in the vicinity [2] of a point q ≈ 1.

When the principle of Maximum MEP is applied to the Rényi entropy functional of an isolated system. At this phase, the Rényian functional reduces to the Boltzmannian functional, thus enforcing the Boltzmann principle from which all thermodynamic properties of extensive and non-extensive Hamiltonian systems can be deduced.

The measure of information in this example of a system's incomplete statistical descriptor with the help of probabilistic distribution is called the information entropy functional, or just entropy p = {pi}, 0 ≤ pi ≤ 1, i =

1, … , n.

Boltzmann-Shannon representation of the entropy functional is the most well-known to read as in Eq. (1):

HB = −KB ∑ pilnpi. (1)

The entropy HB correlates with the thermodynamic entropy functional in the situation given, where the distribution pi is the system's macroscopic equilibrium state, and the subscripts i denote dynamic microstates in the Gibbs phase space.

This entropy function was justified by [2]–[4] based on a system of axioms presented in a theorem form. Their axioms were analyzed in [2]–[4] where it was shown that a uniquely determined Boltzmann-Shannon entropic form is provided by a quite artificial axiom related to a form of conditional entropy functional (That is, the entropy functional of a subsystem of a system being in a prescribed state). On another strong note, [3] examined several papers on this topic and discovered that the Shore and Johnson [6] axiom system, which results in the Rényi entropy function [4], follows Eq. (2):

(q) KB q

H (p) = l n (∑ p ) , ∑ pi = 1, (2)

1 − q

i=1 i=1

The Rényi entropy functional, which is a mathematical measure, was used to quantify the amount of information or disorder in a system. It mentions that q (c.f., Eq. (2)) must be positive and not less than zero. The properties and characteristics of Rényi entropy functional are further explored in related literature [4]– [6]. Among its basic properties, we may mention positivity ( H ≥ 0), concavity for q ≤ 1, and in addition limq→1 H = HB.

In the case of |1 − ∑i p | ≪ 1 (Which, in view of the normalization of the distribution {p }, corresponds to the condition |1 − q| ≪ 1), one can restrict oneself to the linear term of the logarithm in the expression for

(q) (q)

H (p) over this difference and H (p) changes to the defined form of Eq. (3).

(q) KB q

H (p) = − (1 − (∑ p )). (3)

1 − q

Such a linearization of the Rényi entropy functional was proposed by [11], [12]; at this time, the Tsallisian case had come into existence by [12].

The entropy functional stops being exhaustive due to logarithm linearization. To examine a range of non-extensive systems, Tsallisian followers have extensively utilized this quality [7]–[15]. In doing so, the constraint |1 − q| ≪ 1 mentioned above is typically ignored.

According to MEP, when describing a system statistically, its distribution function should accurately represent the average quantities observed in the system. If these quantities are not known, the distribution function should be as indeterminate as possible. This approach has been widely used in constructing equilibrium statistical thermodynamics for isolated or weakly interacting thermodynamic systems.

Gibbs ensembles [1]–[5] were widely used as a statistical approach to accurately represent average quantities observed in a system, as they are commonly used in constructing equilibrium statistical thermodynamics for isolated or weakly interacting thermodynamic systems. The information entropy functional, commonly referred to as the Boltzmann-Shannon entropy functional, is traditionally used to quantify the disorder or uncertainty in a system.

This theorem introduces as it reads a new physical interpretation: As Rényi entropy functional is more general than both Shannon and Tsallis, since Rényi entropy functional reduces to Shannon case as the parameter q → 1, and that Rényian entropy functional's linearization about a point q ≈ 1 is the Tsallisian entropy functional. This leads to a newer ground in information theory; we could represent it by the following Fig. 1:

In other words, this could be read as employing the Rényi entropy function to research any concept would generate the special case of Shannon case as the parameter q → 1, and would reduce to the Tsallis case if we carry out the linearization of the Rényi entropy function in the neighbourhood of a point q ≈ 1.

Shannon entropy, sometimes referred to as Gibbs entropy in statistical physics, is a measure of disorder in a system. As an alternative to Gibbs entropy, Tsallis developed a non-extensive entropy [8], [10], indexed by q, which results in an infinite family of Tsallis non-extensive entropies. While Tsallis's non-extensive entropy produces type II generalized Lévy stable distributions with heavy tails that obey power laws, Gibbs's entropy produces exponential distributions. It is important to remember that Tsallis entropy is equivalent to Havrda-Charvat's structural q-entropy [9], but the non-extensive mechanics community frequently ignores this relationship. Additionally, rather than the other way around, Tsallis distributions are derived from Lévy distributions.

The current paper contributes to:

I. The current study is a strong generalization of an existing research work in literature.

II. The provision of entropic applications to machine learning.

III. The exposition of several open problems.

The flowchart of the provided work reads as follows:

I. Introduction and literature review

III. Entropic application to machine learning IV. Conclusion, open problems, and future research pathways

We'll talk about how four different generalized t-distribution types can be derived from the definitions of Rényi entropy functional. The Lagrangian equation of the calculus of variations, a mathematical technique for optimizing functions subject to restrictions, is used to reach these conclusions. According to the author's analysis, these discoveries are brand-new, and they are presented.

Feature selection [16] in machine learning involves identifying the most relevant and predictive features while excluding irrelevant or redundant ones. Information theory, specifically mutual information measuring the correlation between features and labels, is commonly used for this purpose. The exploration of using Rényi min-entropy for feature selection, showing that in practical experiments with real datasets, the Rényi-based algorithm tends to outperform the traditional Shannon-entropy-based approach in terms of performance, was extensively undertaken by [16].

In a more detailed perspective, the authors of [16] have provided a method for feature selection based on Rényi min-entropy, which is a measure of uncertainty in information theory. Additionally, it was shown that selecting the optimal set of features using min-entropy is computationally challenging (NP-hard) and proposes an iterative strategy to approximate the best feature subset efficiently [16]. The study [16] compared this approach with one based on Shannon entropy and demonstrated through experiments that the Rényi-based algorithm performs better in practice across different datasets.

In this context, a dataset with 32 classes and 8 features is divided into two types [16], F and F', as showcased by Fig. 2 (c.f., [16]). Each type consists of 4 features. At a specific step in the process, the algorithm based on Shannon entropy selects a set of features, S1= {f1, f3, f4}, as one of the possible outcomes [17]. This selection is a result of the algorithm's approach to feature selection based on the information content provided by Shannon entropy.

In the context of feature selection in machine learning [16], the text discusses the comparison between Rényi min-entropy and Shannon entropy algorithms. The explains that while the Rényi algorithm [16] may require additional steps to achieve complete accuracy in classification, the Shannon algorithm can stop when the classification is already completely accurate based on residual entropy and Bayes error. The comparison highlights that the choice between the two algorithms depends on the desired level of accuracy in the classification task [16].

When dealing with Artificial Neural Networks (ANNs), the complexity increases due to numerous parameters that can impact the network's structure [18]. To address this, the authors utilized Bayesian optimization. This method combines extensive search with Gaussian processes to determine the best parameters for the ANN model, such as the number of hidden layers, neurons per layer [18], learning rate, batch size, and epochs. By incorporating the Spearmint Bayesian optimization codebase and optimizing these parameters, they aimed to create a model that generalizes well and maximizes accuracy through a 10-fold cross-validation process before testing on unseen data [18].

It is worth noting that, Iin [16] the study, a bootstrap procedure with 5 iterations was used to shuffle data and ensure the results were independent of the specific training, validation, and test set split. Each iteration involved a new experimental run with different training-test set splits. Features were selected based on the training set using Shannon and Rényi min-entropy methods, with each iteration adding one selected feature at a time for a total of 50 steps. The models were trained and tested on the test set after hyper-parameter tuning with 10-fold cross-validation, demonstrating the effectiveness of Rényi min-entropy in feature selection, particularly with the BASEHOCK dataset.

This explains that after conducting multiple iterations of feature selection using Rényi min-entropy and Shannon entropy methods, the average performance was computed and presented in Figs. 3, 4 and 5 (c.f., [16]). The results consistently showed that the feature selection approach based on Rényi min-entropy generally outperformed the Shannon entropy method, particularly when analyzing the BASEHOCK dataset. The results show that the feature selection method based on Rényi min-entropy generally yielded better performance compared to Shannon entropy, particularly with the BASEHOCK dataset, indicating the effectiveness of Rényi min-entropy in improving classification accuracy in machine learning tasks.

In the context of the conducted research in [20], the text refers to using a grid-based environment called Maze2D to showcase the effectiveness of Rényi state entropy-driven exploration. Maze2D is a simple example used to demonstrate how this exploration method works in navigating through a maze with the goal of finding the shortest path from the start point to the endpoint, emphasizing the efficiency of the approach in this scenario, as depicted by Fig. 8 (c.f., [21]).

In the maze game scenario described, the agent can move in four directions- left, right, up, and down- aiming to find the shortest path from the start point to the endpoint. Additionally, the agent can teleport from one portal to another identical mark within the maze. The experimental setting involves using the Q-learning algorithm to navigate through mazes of varying sizes, updating the Q-table at each step to enhance training efficiency and achieve optimal exploration performance.

As showcased by Fig. 9 (c.f., [19]), Rényi State Entropy (RISE) outperformed other methods in terms of average return across six Atari games, with RISE achieving the highest performance consistently. Additionally, RISE demonstrated better training efficiency in terms of Frames Per Second (FPS) compared to other methods like RE3 and MaxRényi, showcasing its advantage in policy performance and learning efficiency.

Using distinct sets of circumstances to optimize the Rényi entropy function, we revealed fresh results in this study. The outcomes produce generalized t-distributions for the whole family of Lévy stable distributions. It is discovered that the Lévy distribution generalizes the Tsallisian case, not the other way around. The work of [22], where Shannon and Havrda-Charvat entropy functional were utilized, is strongly generalized in this work.

The current paper flags several sophisticated open problems:

I. Is it possible to replace the undertaken mathematical mechanism by Ismail's Entropy (IE) (c.f., [23]). If so, what will be the generated distributions?

II. It is expected that using IE would result in higher accuracies in feature selection in comparison to both Shannonian and Rényian cases. The mechanism could be quite sophisticated, but it is doable and would be highly promising as a new frontier to enhance machine learning.

Future research pathways include delving into unlocking the proposed open problems, as well as the exploration of novel information-theoretic applications to interdisciplinary fields of human knowledge.

No specific funding was received for this research.