Arts and Humanities and the others: Why can’t we measure arts and humanities

The use of numbers (publications and citations) to evaluate research/er performances are widespread since ease of use. However, disciplinary differences must be considered to evaluate research/ers accurately without misjudgments in tenures and incentives. The most different filed from others in terms of publications and citation patterns is Arts & Humanities. The main aim of this study is to reveal the main differences between Arts & Humanities and the other fields by considering publications, citations, and collaboration. For this aim, the main statistics for 59,728,700 papers published between 1980-2018 are gathered from InCites in terms of the 251 Web of Science subject categories. The data confirmed that Arts & Humanities is considerably different from other fields. We showed the degree of these differences using statistical measures. The huge difference found out that underline the indispensability for evaluating Arts & Humanities separately from the others.


Introduction
For the last couple of decades, policy-makers and managers try to find solutions for measuring research performance. After the foundation of the Institute for Scientific Information (ISI) and citation indexes in 1960, policy-makers and managers have preferred to use the numbers (publications and citations) to evaluate research/er performances since ease of use. However, disciplinary differences must be considered to evaluate research/ers accurately. Evaluations without considering these differences might end up with misjudgments in tenures and incentives.
The most different field from others is the arts and humanities. The percent of articles is 30% in A&HCI, while it is 58% for SSCI and 70% for SCIE. On the other hand, 44% of publications are book reviews for A&HCI (19% for SSCI and 33% for SCIE). The field differs significantly in terms of publications and citation patterns, especially compared to positive sciences. Many studies in the literature reveal the characteristics of the field [1][2][3][4][5]. All these works assert that while co-authorship is common in positive sciences, researchers in the arts and humanities prefer to work alone. The preferred document type for publication is not articles; they produce books, notes, etc. The papers do not get citations as many as the other fields. The subjects are mostly local and regional, so the dynamics of international publishing is different from the others. For ex., English papers indexed in A&HCI is 72% while it is 94-95% for SSCI and SCIE. It is possible to extend the list by adding more items because the Arts & Humanities do not resemble any other filed we know. Therefore, this study aims to reveal the main differences between Arts & Humanities and the other fields by considering the number of publications and citations, percent of documents cited, percent of highly cited papers, percent of collaboration with industry and internationally, The Kruskal-Wallis test is used to determine whether the four fields differ by the number of papers and citations; citation per publication; percent of documents cited, highly cited papers, industry, and international collaboration and open access publications. In addition to p values, 2 formula ( 2 = ( − + 1)/( − ) where H is the test value, k is the number of groups, and n is the total number of observations) is used to calculate the effect size to evaluate the test results more accurately 2 . Calculating η 2 ≥0.14 means a large effect 3 . Mann-Whitney test is implemented comparing the four fields and to understand the source of difference between fields by the number of papers and citations. The effect size of the Mann-Whitney test is calculated with Glass (1965) formula, as recommended by Cohen [6]. The formula is = 2( ̅ − ̅ )/ , where ̅ and ̅ are average ranks for two groups and is the total size. A value above 0.5 may be thought of large effect size.
A boxplot is created for the variable number of citations per paper using R Commander and KMggplot2 plug-in, which makes it possible to compare the median, minimum and maximum values, range, and quartile range of citations per papers for the fields. It is also possible to examine the distribution of the categories included in each of the four fields through citations per paper. In addition to Kruskal-Wallis test results, median, minimum and maximum values, range, and interquartile range is presented for variables % of documents cited, percent of documents cited, highly cited papers, industry, and international collaboration and open access publications to show the reasons for differences. Table 1 Fig. 1). This finding is essential in terms of showing the citation potential of the Arts & Humanities. Similarly, the percentage of cited publications in the Arts and Humanities is considerably low. Similar differences are observed for the percent of publications produced with collaboration. The rate of international collaborations is 1% for Arts and Humanities. On the other side, collaboration with industry is low for all fields, but it is scarcely any for Arts & Humanities (one in 10,000). Surprisingly, the rate of open access for Arts and Humanities is also lower than other fields. When the number of citations per paper is examined more carefully (see Fig 1), disciplinary differences can be seen more clearly. There is a statistically significant difference between four fields predominantly because of Arts & Humanities as can be easily estimated  Figure 1 represents a Web of Science subject category. When the graphic is scrutinized, in addition to need for evaluating Arts & Humanities separately as a field, the scatter of subject categories refer to the need for evaluating each subject category seperately to understand the patterns more accurately.  Table 2.

Findings
According to Table 2, there is no statistically significant difference between Health & Life Sciences and Pure Sciences & Engineering in terms of the number of publications and citations. On the other hand, there is no difference between Arts & Humanities and Social Sciences in terms of the number of publications, but they differ by the number of citations. There are statistically significant differences between all other fields with large effect sizes.
The Kruskal-Wallis test is also performed for the other four variables in Table 1; percent of documents cited, highly cited papers, industry collaboration, international collaboration, and open access. It was founded that there is a statistically significant difference between fields by all these variables. Table 3 summarizes the results of the tests, and Table 4

Results and discussion
All the findings of the study support other studies in the literature. Statistically, significant differences are found between Arts & Humanities and others within the scope of the study. These differences underline the need to evaluate Arts and Humanities separately from the others.
In the world, there are some positive developments in evaluating Arts and Humanities. ENRESSH (European Network for Research Evaluation in the Social Sciences and the Humanities) project (https://enressh.eu/about/), which started in 2016 and will continue until April 2020, works to create a new generation and improved research evaluation systems for social sciences and humanities. In addition to ENRESSH, Leiden Manifesto, which has a vital role in research evaluations, emphasizes the local and national structure of arts, humanities, and social sciences [7]. Thanks to all these studies, the current practices on research evaluations can be enhanced for the scholars working for arts and humanities literature.