Augmented and Virtual Reality for Public Space Art

Public space art, usually placed outside and accessible to all, is a proper target for the exploitation of transformative and immersive technologies such as augmented and virtual reality. In the recent years, artists and technology developers have collaborated to make physical art more expressive in the digital world, taking advantage of the democratisation of mixed reality devices and software. In this paper, we report on such development of the ArtTM application, an Android app for the public space art in Timisoara, Romania. We show the current state of the application and the innovative features that are currently in development. We present the envisioned roadmap to an augmented and virtual reality-enhanced experience and some initial thoughts on the actual implementation of them.


Introduction
Augmented Reality (AR) and Virtual Reality (VR) are some of today's stars in technology. Together with the more newer (term) Mixed Reality (MR), they are sometimes shortly called XR technologies.
While augmented reality adds virtual elements to the real surroundings of the users, the virtual reality immerses them in a completely virtual world [1]. Both have the potential of enhancing the experiences of the users in ways that have not been possible until now.
A contribution to the expansion of such technologies was made by the democratisation of mobile AR and VR devices (powerful smartphones, augmented reality glasses, virtual reality handhelds) and the adoption of many AR and VR games and apps in the mainstream [2]. AR and VR have been deployed successfully in many fields, including but not limited to entertainment, advertising, tourism, education, medicine.
The focus of this paper is the arts and culture domains, which are proven testbeds for innovative AR & VR applications. A particular category, the public space art, has seen numerous examples of enhancing through intelligent and creative use of such technologies. Public space art is defined as works of art in the public realm [3]. They are usually placed outdoors and that can be visited by anyone, freely.
As such, they can easily become the target of artists and technology developers who wish to digitally enhance them, as happened with the ArtTM application, the case study of this paper.

Related work
AR/VR in public spaces is an extension of the practice popularized earlier in the 2010' of placing QR codes on statues, artworks, monuments and other pieces, which users scan to learn more about the related pieces, as it happened, for example, in the University of Manitoba Public Art Project [4].
Since then, several applications of AR and/or VR applied in the fields of public art and heritage have emerged through tight collaboration between artists and technology experts.
The StreetMuseum application in London is one of the first to showcase great use of augmented reality in culture and history. The application overlays historical snapshots of London's life and big events on top of nowadays surroundings [5]. The application belongs to the Museum of London and uses old photos from its collections.
Lapse is a newer AR application that reveals hidden art scattered through the city of Miami [6]. This public art project combines the work of several artists, which contributed with animations, three-dimensional texts and sounds that can be experienced by tourists wondering through the city. The application is available for both Android and iOS.
The New Museum in New York, together with other partners, launched a virtual reality exhibition called "First Look: Artists' VR" [7]. It takes the form of a mobile app, both for iOS and Android, with which users can experiment various approaches to digital animation in virtual reality. From a memorial to victims of police violence to a journey in an industrial nightclub, the users can observe all sorts of situations, best experienced using a Google Cardboard-style headset.
SketchFab is an online platform that allows users -creators of 3D content -to upload such content and share it freely or for a price. These models can be viewed and controlled in augmented and virtual reality modes (both 3D and cardboard-style view), either directly in the browser or by using the Android and iOS mobile apps. The platform has generous categories for arts and cultural heritage content [8].
In the academia, there have been several attempts to design a framework for developing such mobile applications, followed by practical case studies.
The authors in [9] propose a methodological framework for designing and developing applications for cultural heritage. The reference frame of the methodology is represented through a scheme with 3 axes: technology (visualization, AR, VR, serious games), application category (fixed/mobile indoor, fixed/mobile outdoor) and type of content (text, pictures, audio, video, animations, 3D models). The methodology is also composed of a reference matrix, which shows the various combinations that can be performed between technologies and content resources, and of a use guide, which contains the phases and activities related to putting the methodology in practice.
Somewhat similarly, the authors in [10] propose a framework for mixed reality application development in the fields of urban city or archaeological site explorations. This framework focuses on the software development part of the process of creating such an application and emphasizes its performance on mobile devices.

Application context
The Timisoara City Art mobile app -shortly called ArtTM -is currently a mobile application that helps citizens and tourists to discover public space art in Timisoara. Users can identify these monuments on a map, can find the distance and the public transport means to reach them, but also various information about who did the artwork, when and why.
The initial prototype of the application was done during the first Open Cultural Data Hackathon in Romania, which took place in 18-19 April 2015 in Timisoara. The hackathon was the first event in Romania to facilitate the reuse of open cultural data. Developers, graphic designers, artists and representatives of national cultural institutions were invited to develop applications and design platforms that can reinvent the relationship between the public and the cultural works.
For this application, the open cultural data was provided by the Triade Foundation, a well-known organization acting in the cultural landscape of Timisoara, and consisted of 15 monuments or groups of monuments that belong to Timisoara public street art.
The prototype included a basic augmented reality section, to help the users to discover monuments around them. The geo-based AR feature, built on the Metaio framework, displayed big pinpoints, overlayed in space, on top of the monuments, together with the name and picture of the monument. The feature was discontinued after the Apple bought Metaio in 2015 [11].

Application facilities
In the current version, the app has information on 36 works of art that are distributed in three itineraries (13 artworks have information translated in English). More works of art will be added soon and an iOS version is in development.
The information is saved in a local database on the smartphone, so the application is available for tourists without a mobile data plan in Romania.
The current app features are detailed in the subchapters that follow.

Start page
The start page features the logo of the application and a short description of its objectives (Fig. 1). The logo was designed in such a way that it generically describes an artwork in the public space. The user can also see a random selection of monuments, as an invitation for them to quickly discover what the app offers.

The map
The second screen of the application shows the user all the monuments overlaid on a map (Fig. 1). The map is by default centred where the user is physically situated, so they can quickly discover what monuments lay around them. The works of art are displayed with different colours, representing the itinerary that they belong to.

The itineraries
The third screen of the application displays the four itineraries that are available in the app: Contemporary Art, Revolution Memorial, The Historical Timisoara and Baroque Timisoara (Fig. 2).
Each itinerary contains the corresponding list of monuments, represented through the name of the monument and a representative photo. On taping a certain monument, the user can view various pieces of information, such as: one or more photos, from different angles/perspectives; details about the author (name, biography); details about the artwork (year of creation, the material from which it was created, the physical dimensions, the context in which it was created, a short description of the artwork); details about its location (the address where the artwork is exposed, type of entrance, information on the public means of transport, information on accessibility); and a link to further information on the web.
All this data is available under a Creative Commons Attribution No Derivatives license. In addition, from this screen, the user can mark a work of art as favourite, so they can remember to visit it physically.

Favourites
The fourth screen of the application shows the users all the artworks that they already marked as favourite (Fig. 3). They can use this list to make a short-list of items that they wish to visit physically.

Settings
From the settings menu, the users can switch the language between Romanian and English and they can read the "about" page of the app (Fig. 3).

Research in AR & VR features
The next version of the application is in research and development phase and is focused on implementing augmented and virtual reality features designed for users to experience a more immersive interaction with the works of art. Fig. 4 shows the current and future state of the application, according to the methodological framework described in [9] and already-mentioned in the "related work" chapter. The current features of the app are highlighted in blue: the app allows users to visualize text and pictures about the artworks, which are, according to the above-mentioned framework, fixed outdoor (individual monuments placed outdoors) as well as mobile outdoor (set of monuments placed outdoors, related between them, such as those belonging to a thematic park).
The targeted future state of the application is highlighted in green. The proposed developments will integrate current text and pictures in augmented and virtual reality experiences, as well as add 3D models to these experiences. Other content such as audio, video and animations are quite interesting for the long-term development of the application.

Fig. 3. Screenshots of the ArtTM appfavourites screen and settings button
The AR & VR features would be available when the users are viewing the details of a single work of art. They would be able to start the AR/VR experience using two representative icons placed at the top of the screen (Fig. 5). 6 The proposed solution for exploring the artwork in AR mode, from the comfort of the users' home or office, is for them to be able to see the artwork in front of them, through the display of the smartphone, by activating a target image. For this target image to be easily reproducible by any user, we would inform the user to write the name of the app, i.e. ARTTM, on a sheet of paper, and place it on a horizontal surface. The experience will be similar to that in Fig. 6 (simulated with ROAR † ).
We are currently testing this feature in the Android app, using the new ARCore library. From a usability point of view, showing the artwork at real scale in AR is more realistic. The downside is that the target image tends to go outside the view of the smartphonewhen the user starts moving around and exploring the artwork -which leads to interruptions in the experience as the artwork abruptly disappears from the screen.
In addition to this type of experience, we plan to add an AR geolocation feature, which was available in the first prototype of the app, with which the users can get a visual indication of the artworks that are in their close proximity, when being actually outdoors.
For the VR experience, the proposed solution is for the user to get a 3D model of the artwork, which they can rotate and zoom in and out to explore the work (Fig. 7). The testing of this feature is currently done using a software module based on android-3D-modelviewer open source code ‡ . We are experimenting with .OBJ files that contain the 3D model of some of the artworks in the app. Some initial remarks are related to the need of optimizing the size (and, thus, quality of the files), since the original files, which are in the range of tens of megabytes, are slow to manipulate in 3D on a smartphone and almost impossible to store offline. † https://theroar.io/ ‡ https://github.com/andresoviedo/android-3D-model-viewer The next envisioned step is to add a cardboard-style view in VR. Fig. 8 shows a similar approach for an artwork in SketchFab [8]. To experience the artwork in this way, the user must wear specialized glasses, albeit inexpensive ones, such as Google Cardboard.
After implementing the above-mentioned AR and VR features, the authors plan to do user testing and usability evaluation of the upgraded ArtTM application.

Conclusions
The combination of augmented and virtual reality technologies has witnessed beneficial exploitation in the field of public space art and related domains. A wide range of such applications have emerged in the recent years, riding the wave of the democratisation of AR and VR devices and the adoption of the technologies in the mainstream. In this paper, we presented the case study of the ArtTM application, an Android app for public space art in Timisoara, based on open cultural data. We showed how we envision the app to evolve from simple visualization of artworks to AR and VR experiences that build on their story. We also mentioned some preliminary thoughts on the implementation of AR and VR features and the challenges encounters, i.e. uninterrupted AR experiences and optimized 3D