Assessing Malaysian Pre-University Students’ Visual Literacy Skills Using Rubric Based On Objective Questions Constructed

. This paper discusses the mathematical visual literacy skills among pre-university students in Malaysia. Students’ were tested on a set of mathematical visual literacy skills based on Avgerinou’s Visual Literacy Index. Before conducting the main study, researchers designed a proper scoring rubric to guide scoring students’ performance. An analytical scoring rubric was used and its suitability was defined throughout theprocess. It was found that students performed poorly in mathematical visual literacy skills when the set instrument has given was in structure format but showed some improvements when the questions were changed to objective format.


Introduction
Assessment is a process used to determine an individual's mastery of complex activities generally through performance observation [1].Before conducting an assessment, the learning goals and objectives should be identified.Next, the understanding of how assessment is implemented in the curriculum is also identified, followed by data collection and evaluation of the results for educational programs improvement [2].Assessment in mathematics is nothing new in the education field irrespective of educational level, and this also includes the pre-university level.The purpose of such assessment is to observe the performance of the students so that changes and improvements can be made to the existing teaching methods.Past studies suggest that students' knowledge and skills in mathematics at the university level is a reflection of their basic knowledge at the pre-university level [3].Meanwhile, the field of visual literacy is gaining interest from various disciplines including arts education, engineering, science, mathematics and language.
The term of visual literacy was first introduced by [4].According to [4] visual literacy alludes to a gathering of vision capabilities an individual can create by observing and in the meantime having and coordinating other sensory experiences.The development of these skills is essential to ordinary human learning because it enables a visually literate person to discriminate and interpret the visible actions, objects, symbols, natural or man-made, that he experiences in his environment.Moreover, visual literacy is described as the capacity to both interpret and create visuals for the purpose of expressing ideas and concepts [5].Visual literacy also can be defined as the learned knowledge and skills needed to precisely comprehend, interpret, analyze and create visual messages [6] Visual literacy has been widely acknowledged and shows increasing recognition making it to be necessary to be coordinated into educational programs [7] The emphasis on visual literacy in teaching is important to help students learn more efficiently and it also enables them to understand the variety of information received in everyday life regardless of the field of knowledge better [8].Hence, a student is said to be visually literate when the student can apply and relate knowledge learned to real-world [9].In the context of language learning, for example, visual literacy is important because visuals are viewed as universal languages because they can carry meaning to verbal expressions that are not well understood (Burnmark, 2002) and solve problems related to language barriers [10].However, research concerning assessing visual literacy skills is still quite limited [11].Realizing the importance of visual literacy in education, researchers feel that it is necessary to look at the extent to which visual literacy skills are being applied to pre-university students learning in Malaysia, especially in mathematics.The visual literacy skills involved in this study are a set of visual literacy skills developed by [12] in which are classified into three main skills: Visual Information, Intellectual Skills and Cognitive Strategy.Intellectual Skills contains five distinct skills namely Discrimination, Concrete Concept, Defined Concept, Rule and High Order Rule.These seven skills form a framework known as the [12] Visual Literacy Index.
To study the mathematical visual literacy skills among Malaysian pre-university students, the researchers took the first step by investigating the emphasis on mathematical visual literacy skills in existing examinations.Some of the pre-university programmes offered in Malaysia include foundation studies, diploma courses, Malaysian High School Certificate (STPM) programme and Malaysian Matriculation certificate programme.Next, researchers analyzed 27 sets of mathematics examination questions from these pre-university programmes.Table 1 shows that 84.5% of the questions did not test any of the visual literacy skills, meanwhile, 15.5% of the questions tested the visual literacy skills.The percentages for each visual literacy skill are shown in Table 1.

Table 1. Personal Characteristics of Teachers
From Table 1, it is found that Cognitive Strategy and Discrimination skills were not tested at all in any of the questions.Visual Information skill (5.6%) was the dominant skill tested followed by Rule (4.1%) and Concrete Concept (2.9%) skills.The rest of the percentages were recorded for Defined Concept (1.7%) and Higher Order Rule (1.2%) skills.Due to lack of mathematical visual literacy skills being emphasized in these questions, there is a need to seek mathematical visual literacy skills mastery among pre-university students by conducting an assessment whereby a set of questions containing all the visual literacy skills will be tested.Prior to conducting any study, an appropriate measurement and scoring system for an assessment should be provided.Measurement for performance, work product, or learning skills can be challenging without appropriate benchmarks.In assessment methods such as case studies, essays, projects, and portfolios where students' feedback cannot be evaluated with complete objectivity, rubrics can be considered as an effective approach to achieve a reliable and valid professional judgment for students' performance [13].According to [14], rubric works differently compared to a checklist in performance assessment because rubrics functions as rating scale.It is also a descriptive scoring scheme developed by teachers or evaluators as a guideline in analyzing students' products or processes of their efforts [15][16].By examining students' answers in a test, often, the scores given to the answers represent students' achievement and mastery in a given field or domain [17].Scoring rubric can be divided into two types, namely holistic and analytical scoring [16].In holistic scoring, a student's work is evaluated as a complete unit in which student's feedback is read by the evaluator.[18] stated that teachers do not judge the component parts separately using this type of rubric but they are required to set scores for the entire process or product.Next, the score will be given either as a percentage, letter grade, or rating number representing achievement level [19].Reported rubric score is based on the overall quality, proficiency, or understanding of specific skills and content [14].Meanwhile, evaluating students' work through analytical scoring is done by scoring each element in a given assessment separately [19], [20], [21].The scoring process involves assessing students' feedback, scoring each element or attribute evaluated, and then summing, averaging, or proportionally weighs the scores in various dimensions to get the overall score.In other words, the assessor assigns a score to each element and combines these scores to form an overall score [15], [18], [22].Analytical scoring benefits more in giving more detailed diagnostic information on students' strengths and weaknesses in different areas of expertise [19], [20].
Before designing a rubric, the evaluator must determine whether the performance or product to be evaluated is using analytical scoring or holistic scoring.In this study, the researchers wanted to score and test each skill separately, thus, scoring was based on an analytical rubric.Some steps need to be considered before designing a scoring rubric.Diagram 1 shows the procedure in designing an analytical scoring rubric according to [14].
Step 1: Re-examine the learning objectives to be addressed by the task.
Step 2: Identify specific observable attributes that you want to see (as well as those you don't want to see) your students demonstrate in their product, process, or performance.
Step 3: Brainstorm characteristics that describe each attribute.
Step 4: Write thorough narrative descriptions for excellent work and poor work for each attribute.
Step 5: Complete the rubric by describing other levels on the continuum that ranges from excellent to poor work for each attribute.
Step 6: Collect samples of student work that exemplify each level.
Step 7: Revise the rubric, as necessary.In this study, the main objectives are to identify how the analytical scoring rubric is constructed for an assessment and to test if the rubric built can be used for the analysis of the study conducted.

Research Design
Researchers used the analytical rubric design proposed by [14] as shown in Diagram 1 to design rubric scoring for mathematical visual literacy skills items in this study.In Step 1 of rubric designation, the researchers need to identify the purpose of the study which is to investigate the mastery of mathematical visual literacy skills among pre-university students in Malaysia.Step 2 involves identifying attributes to be measured in the study.There are two important elements to be measured which are students' skills in solving visual literacy problems and their ability in giving explanations on the problems solved.Next, in Step 3, researchers are required to brainstorm the characteristics of each attribute.In every question, two parts need to be taken into consideration: the answers given and the explanations for the answers based on the visuals involved.Scores to each question later will be based on both attributes.
Meanwhile, Step 4 involves writing thorough narrative descriptions for excellent work and poor work for each attribute.As previously discussed, rubrics are classified based on the answers and explanations given.From the lowest to the highest score, the correct answer is prioritized over the explanation.The rubric scores are treated the same way as ordinal scales using a scale of 0 to 5. Next, completing the rubric will be discussed in detail in Step 5.In this study, students' responses were initially evaluated using a rubric adapted from [23] (Table 2).Their study intended to measure the levels of mathematical thinking skills among primary school students from different types of schools in Malaysia by administering a written test.The variety of students' feedbacks received were scored according to the rubric to determine the mastery of students' mathematical thinking skills at the end of the study.Similarly, this study used their rubric as shown in Table 2.

Rubric Scores Description 0
Omitted 1 Incorrect answer and no explanation 2 Correct answer and no explanation 3 Incorrect answer and reasonable explanation 4 Correct answer and unreasonable explanation 5 Correct answer and reasonable explanation

Participants
This study of mathematical visual literacy skills was conducted involving 132 pre-university students attending diploma, matriculation, foundation and STPM programmes, majoring in science in Malaysia.

Data Collection Tools
To carry out the study on mathematical visual literacy skills, firstly, an instrument consisting of 7 mathematics questions in structure format, representing each visual literacy skill were developed.Next, the instrument was validated and the reliability of the instrument was determined.Validity refers to the extent to which the research instrument can measure the constructs being studied accurately and measure what it should be measured [24], [25].Meanwhile, the reliability of an instrument refers to the degree of consistency of the instrument in measuring what is to be measured [26].The instrument was validated by 2 mathematics lecturers.They were asked to give feedback on the questions based on the appearance of the questions and instrument as a whole in terms of consistency of style and formatting, legibility, feasibility and the clarity of the language.Other than that, the validity of the contents of the questions was determined by calculating the degree of agreement between experts using the Cohen Kappa formula.Cohen's value obtained was 0.71 and considered as "Good" [27], with no questions being removed.Questions were then improvised based on experts' suggestions.Next, the test was administered to 83 preuniversity students taking a diploma course in mathematics from an institute to seek the reliability of the instrument.After the students' responses were analyzed, it was found that the reliability value obtained was 0.82 and categorized as "Good" [28].The validity and reliability values of the instrument enable researchers to proceed with the real study.The details of the construction of the questions based on topics, visual literacy skills and type of visuals are as shown in Table 3.All the questions were developed based on the definitions for each visual literacy skill.According to the definition, Visual Information refers to the ability to know visual vocabularies, knowledge of visual conventions and the knowledge of definitions of the mathematical signs and symbols [12].The students are expected to be able to recognize the main feature of visual representation of mathematical terminology, able to demonstrate their knowledge and apply their understanding of/on the meaning of mathematical notations, signs and symbols through agreement with the given definitions.Figure 2 shows problem-solving for the Visual Information skill question to be tested (Question 1).
Step 1: Identifying points  = 1 and  = 3 on the curve.mathematical signs or symbols and capabilities in critical viewing [12].A person with this skill should have the ability to use mathematical definitions to classify the examples of signs and symbols (or both) given.The procedures in solving problems in this question are shown in Figure 3.
Step 1: Identifying inequalities representing lines AB and CD.
Step 2: Shading region as a result of intersection between line AB and CD.This question digs students' knowledge and understanding of the meaning of signs and symbols for inequalities by interpreting them in the graph.This skill is an abstract mathematical concept, where mathematical formulas are usually involved.In this question, students are asked to state the region involved as the result of the intersection of two values of inequalities; 3 + 4 < 24 and 8 + 5 < 40.Students must first be able to determine the correct match for straight lines AB and CD with given inequalities of 3 + 4 < 24 and 8 + 5 < 40.Next, they also need to master how "less than" symbol functions.Based on the region marked in the graph, students will be able to give the explanation in determining which coordinate lies in the region.
Next, the Rule skill is tested in Question 3. It is defined as the ability to mentally trace constituting elements of a visual representation and decide on their associations, to interpret visual representations of data as being correct representations of the given information, and to identify, interpret and compare the visual information displayed [12].Figure 3 shows proper steps in solving the Rule skill question (Question 3).
Step 2: Identifying area representing the number of students taking three subjects.Question 3 requires students to gather information from the Venn diagram that illustrates the relationships of the number of students taking Mathematics (M), Physics (P) and Accounting (A).Students are asked to shade the area representing the number of students taking at least two subjects.This statement means that the area to be shaded must include the area of those taking two subjects and three subjects.Next, the area shaded in the Venn diagram helps students to determine the total number of students taking at least two subjects and ensure the accuracy of their answers.
On the other hand, Discrimination skill focuses on the abilities to perform visual discriminations and reconstruction, visual reasoning, to know visual conventions and associations [12].Students are expected to be able to recognize differences in shapes between/among two-dimensional visual mathematical representations and be able to read, discriminate and group, and interpret the information displayed in the visual mathematical representations.This question is tested in Question 4 of the instrument and the process of problem-solving is shown in Figure 5. Step 2: Stating the limit of degree of elevation after reaching the building.Question 4 requires students to identify changes that occur at the angle of elevation as they move toward the building shown in the diagram.This question stimulated the students to imagine themselves being in the given situation.Students should be familiar with the signs and symbols that engage with angle of the elevation and be able to relate them into the visualized situation.When given the picture, students will be able to explain that as they approach closer to the building, the angle of elevation will increase.Next, students will be able to state that 90° is the limit of the angle of elevation after reaching the building.
The next skill to be discussed is Higher-Order Rule.This skill contains the abilities to do visual reasoning, visualization and constructing the meaning of abstract mathematical concepts [12].Students are expected to be able to visualize, reason and reconstruct and produce a concrete-abstract continuum of given visual representation and to be able to determine the reconstruction of the elements of an object on the evidence of elements on other visible surfaces of the same object.The details of the problem solving for Question 5, testing on Higher-Order Rule skill is as shown in Figure 6.
Step 1: Drawing points A, B, C, D and E as described in the question.
Step 2: Finding width of the road, CD using trigonometry formula.This question involves students' visualization skills using the statements given in the question.Based on the given information for points A, B, C, D and E as well as mathematical concepts such as parallel and perpendicular lines, students should be able to transform the information into a drawing or sketch.Initial sketch for positions of lines ABC and EBD indirectly functions as explanations for students to solve the next problem.By fully understanding the positions of the lines and points from the statements and using mathematical knowledge correctly, students will be able to find the width of the road which is CD, where the width of CD is 3m.
Meanwhile, the Concrete Concept skill was discussed in Question 6 of the instrument.The concrete concept refers to the ability in understanding and applying the knowledge of design principle and their use and besides having the critical viewing [12].The process of solving this question is shown in Figure 7.
Step 1: Identifying the shapes making up the cylinder.
Step 2: Finding the shortest distance between A and B. Question 6 tests students' ability in understanding the text given and applying knowledge of the geometrical shapes learned in critical ways.This skill focuses more on the concrete situation, which is the cylinder itself.Based on the cylinder shown in the figure, students make explanations by relating how a 3-dimensional shape is formed from a 2-dimensional net.This cylindrical shape is composed of a rectangle of 20  10 and two circles with a radius of 10 each.Referring to the nets, students should be able to find the closest distance from A to B, which is √100 + 100 2 .
Lastly, the Cognitive Strategy skill was tested in Question 7. By definition, Cognitive Strategy refers to the ability to perform observation, visualization and visual thinking [12].The students are expected to be able to mentally combine or remove, one at a time, components of mathematical visual representations to expose the abstract concepts and to be able to visualize and perform certain procedures following textual instructions.Figure 7 illustrates the process of solving Question 8.
Step 1: Observing the pattern of number of beams relating to the number of sections.
Step 2: Making mathematical reasoning and generate a general formula.Cognitive Strategy question (Question 7) needs students to observe how the design of the truss of a bridge with 1 section and 3 beams will change in the number of beams as the number of sections increases.Students will make an explanation by gathering this information mentally and visualize the formation of the truss of a bridge consisting of 3 sections and the number of beams it has.When asked to relate the total number of sections, m, with the number of beams, n, students generate a formula replacing visualization process involving the large number of sections and beams of a truss of the bridge.

Data Collection
The researchers collected the data in this study face-to-face, where the researchers went to the centers involved and met with the students.First, the researcher explained to the students the purpose of this visual literacy study conducted.The purpose of this study is to investigate the mastery of mathematical visual literacy skills among pre-university students in Malaysia.After that, the students have to answer the questions given by the researcher within the given period of 1 hour.Finally, researchers retrieved questions along with students' answers after 1 hour.

Data Analysis
The data obtained were analyzed by the researcher by marking the students' answers according to the answer scheme provided and giving marks based on the scoring rubric and description that had been provided.

Findings
The process of this data collection also represents Step 6 in rubric designation.Using the analytical rubric shown in Figure 2, students' responses to the 7 items of mathematical visual literacy skills were scored and the results of the analysis are shown in Table 4. Based on Table 4, it was found that no student scored 1 (incorrect answer and no explanation) and 2 (correct answer and no explanation).This led the researchers to conclude that the existing rubric cannot satisfy the students' responses scoring.However, there exists a situation whereby many students gave incorrect answers with unreasonable explanations.Given the high percentage recorded, the researchers took into account this situation.As mentioned by [14] in Step 6 of rubric designation, there is a need to revise the rubric for certain conditions.Hence, researchers modified the existing scoring rubric to a new rubric as shown in Table 5.
Table 5.New Scoring Rubric and Descriptions.

Rubric Scores Description 1
Omitted 2 Incorrect answer and unreasonable explanation 3 Incorrect answer and reasonable explanation 4 Correct answer and unreasonable explanation 5 Correct answer and reasonable explanation Based on the modified rubric, the researchers re-coded the students' responses and presented the analysis using percentages for each score (Table 6).Table 6 shows that the percentages of students who scored 1 and 2 were quite high for each of the skills tested.One of the factors identified that might influence students' inability to provide correct answers and explanations were because they find it difficult to answer questions in a structural format.This is supported by studies that found the probability of structural questions not being answered are high [29].Thus, researchers made some improvements to the items by changing the format of the questions from structural format to objective format.An example of modification to the format can be found in Table 7.After modifying the format of the questions, the instrument was validated again by the experts.Cohen's Kappa value obtained was 1.00 and within the range of "Very Good" [27].Comments and suggestions from the experts were considered to polish the quality of the questions.Next, pilot testing was done on 85 matriculation students from an institute.The data collected were then used to find the reliability of the instrument.The reliability recorded was 0.88 and considered "Good" [28].The real study has then proceeded whereby all the 7 items of mathematical visual literacy skills were tested on 168 pre-university students taking diploma, matriculation, foundation and STPM programmes, majoring in science.The results for these objective questions were then compared with the results of the structured questions (Table 8).

Discussion
Based on Table 8, the percentages of students who scored 1 and 2 decreased for Visual Information skill question after structured format questions were changed to objective format.Meanwhile, the percentages of students who scored 3, 4 and 5 showed an increment in objective format questions compared to structured format questions where no percentage was previously recorded.For the Discrimination skill question, the percentages decreased significantly for students who scored 1, 2 and 3 in the objective format question while the percentages of students who scored 4 and 5 increased.Changing the format of the Concrete Concept skill question caused a decrement in percentage for students who scored 1 but the percentage for those who scored 2 slightly increased for objective format questions.The percentages for those who scored 3 and 5 showed an increment in objective format questions but no percentage was recorded for score 4.
For the Defined Concept skill question, the percentages for those who scored 1, 2 and 3 slightly decreased and slightly increased for scores 4 and 5 after the questions' format conversion.Percentages of those who scored 1 for objective format in Rule skill question showed a slight decrement and decreased drastically for score 2. Meanwhile, the percentage of score 3 slightly increased but no record was found for those who scored 4 for the objective format question.The drastic increment was shown for students who scored 5.After changing the question format, Higher-Order Rule skill question recorded a decrement in percentage for those with score 1 and score 2. On the other hand, score 5 showed a low percentage but this showed a positive change compared to the result in structured questions where no record was found.In the case of the Cognitive Strategy question, the conversion of the question's format showed obvious decrement for scores 1, 2 and 3, while scores 4 and 5 positively showed an increment in percentages.
Overall findings shown in Table 8 proves that conversion of questions' format led to the significant decrement in percentages of students who scored 1 and 2, meaning that the percentages of students who ignore these questions can be reduced.However, the percentages of those who scored 3 and 4 varied for all the questions with both types of formats.Researchers suggest that further study needs to be conducted to explore more on odd cases.For example, for score 4 in the Concrete Concept skill question (Question 6), percentages recorded maintained at 0%, showing that students were able to give correct answers but failed to provide a correct explanation based on the visuals.A study by [30] on pre-university students found that students had trouble justifying mathematical solutions involving visuals.Meanwhile, the percentages for those with a score 5 increased for all the mathematical visual literacy skills after changing the format of the questions.
Objective format items such as multiple-choice items allow extensive measurement of skills, knowledge and competencies across different types of disciplines and fields including the ability to understand concepts and principles, decision making, complete statements, interpret data, draw conclusions, reason and apply information [31].Hence, this format is widely used as an assessment in education.Objective format items consist of two main parts, namely stems and options [17].Stems contain contexts, content, and/or questions that need to be answered, while options are a set of alternative answers that contain one correct option and one or more wrong or distracting options [31].With the options available, students have the option to use their knowledge and problem-solving skills to identify the relationship between the content in the stem with the correct option.

Conclusion
Assessing students' performance is a difficult process [32].However, the use of an appropriate rubric makes it easy for scoring students' responses in an assessment.The study found that there is a possibility that a rubric cannot be customized in different assessment scoring, causing modification need to be done if necessary.In the context of students' achievement in mathematical visual literacy skills, it is clear that structured questions less encourage students to try answering the questions better.

Suggestion
Researchers suggest that the questions need to be improved such as changing the format of the questions.To further strengthen this study of mathematical visual literacy, researchers propose a more in-depth study by increasing the number of mathematical visual literacy questions to be tested.

Limitation
The limitation of this study is that the respondents in this study did not involve students from IPG programs, polytechnics, and vocational colleges.It is because polytechnics and vocational colleges offer skills training to students.

Funding
This research has been carried out under the Fundamental Research Grant Scheme (FRGS), 2017-0071-107-02, provided by Ministry of Higher Education of Malaysia.The authors would like to extend their gratitude to Universiti Pendidikan Sultan Idris (UPSI) that helped managed the grant.

Step 2 : 1 .Fig. 2 . 1 . 6 ITM
Fig. 2. Problem Solving for Question 2.The question intends to test students' ability to interpret the meaning of function(3)−(1)3−1 by drawing the line it represents in the given graph.Before interpreting this information, students must first understand and be able to explain that the line in the question refers to the slope of the curve between  = 1 and  = 3. Explanation based on visual is done when the slope is drawn by gathering and processing information from the given graph.A student is said to have this Visual Information skill when the student grasps the meaning of the gradient of the curve and further understands how the term is represented in the form of sign and symbol.The sign refers to the line drawn in the graph, meanwhile, the symbol is denoted by (3)−(1) 3−1 .Question 2 of the mathematical visual literacy skills test explores students' skills in Defined Concept Skills.The skill highlights knowledge and understanding of the meaning of

Table 3 .
Division for Mathematical Visual Literacy Skills Questions Construction.

Table 4 .
Analysis of Responses for Mathematical Visual Literacy Skills Items (Structured Questions).

Table 6 .
Analysis of Responses Using New Rubric.

Table 7 .
Modification of the format for Question 5

Table 8 .
Comparison of Scores' Percentages for Structured and Objective Questions.