Rank-Density Scaling for Transformer-Based Numerical Representation

School of Software Engineering, East China Normal University, Shanghai, China

^* This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

Data preprocessing is often seen as a simple preparation step, but for numerical data, the scaling method can change how clearly a model receives the information. This paper compares traditional Min-Max scaling with a Rank-Density Scaling method for transformer-based numerical representation. Min-Max scaling keeps the original numerical distance, which is useful in many cases, but when the data are clustered or long-tailed, most values may be squeezed into a small part of the target interval. Rank-Density Scaling takes a different approach. It maps each value according to its rank in the dataset, so the transformed values become more evenly spread. The experiment uses the Wine Quality dataset, with the residual sugar feature as the main object. After scaling, the values are converted into 64 discrete tokens and tested with a small transformer model through a next-token prediction task. The results show that Rank-Density Scaling uses the token space much more fully and reaches almost maximal token entropy. It also helps the transformer reduce training loss faster than Min-Max scaling. However, its validation loss is higher, which suggests weaker generalization. Overall, this study shows that Rank-Density Scaling can improve representation balance, but it may also weaken the original numerical geometry. The main finding is a tradeoff between token-space entropy and geometric fidelity.