How we built an offline-first augmented reality heritage guide using ARCore, Kotlin, sensor fusion, and adaptive AI voice interaction that became a finalist at the Samsung PRISM Metaverse Hackathon.
Introduction
Most campus tours are forgettable.
You walk past buildings, glance at a plaque, maybe hear a short explanation, and leave without understanding why the place matters. The history exists, but the experience feels disconnected from the environment itself.
We wanted to change that.
So we built an offline-first augmented reality heritage guide that transforms physical spaces into interactive storytelling experiences using geospatial AR, sensor fusion, voice interaction, and adaptive intelligence.
What started as a hackathon idea eventually became a finalist project at the Samsung PRISM Metaverse Hackathon.
This article breaks down the engineering decisions, architecture, technical challenges, and lessons behind building the system.
The Core Idea
The original concept was simple:
What if history was something you experienced spatially instead of reading from a board?
Instead of relying on:
- static signboards
- guided tours
- brochures
- internet-dependent applications
we designed a system where:
- users walk near a monument
- the phone understands spatial context
- information appears directly in the real world through AR overlays
- voice interaction works even without internet access
The objective was not just augmented reality.
The objective was reliable augmented reality under real-world constraints.
The Real Problem We Focused On
Most AR demos work well inside controlled environments.
Real campuses and heritage locations are not controlled environments.
You deal with:
- unstable internet
- inconsistent lighting
- GPS drift
- device limitations
- users who do not want complicated onboarding
Many AR applications fail because they assume ideal conditions.
We designed for the opposite.
The engineering question became:
Can an AR system remain usable, meaningful, and responsive even when the network disappears?
That decision shaped the entire architecture.
What We Built
The final system was a native Android application built using:
- Kotlin
- Jetpack Compose
- Google ARCore
- Room Database
- Android Sensor APIs
- Speech Recognition + Text-to-Speech
- Hybrid offline/online intelligence
The application combines:
- geospatial AR
- sensor fusion
- offline-first storage
- adaptive AI
- voice interaction
into a seamless exploration workflow.
How the Experience Works
Step 1: Open the App
Users immediately see nearby monuments and heritage locations sorted by proximity.
No login.
No setup friction.
No dependency on cloud synchronization.
Step 2: Walk Around
As the user moves:
- location updates dynamically
- nearby heritage points are recalculated
- spatial awareness updates in real time
The system continuously combines GPS and sensor data to estimate orientation and movement.
Step 3: Enter AR Mode
When users point their camera toward a monument:
- contextual overlays appear
- information remains spatially anchored
- historical content aligns with the physical environment
The objective was immersion through context rather than visual overload.
Step 4: Ask Questions
Users can ask questions like:
- “Who built this?”
- “Why is this place important?”
- “When was this constructed?”
The system responds using either:
- cloud AI when online
- local knowledge processing when offline
This hybrid execution model became one of the most important architectural decisions in the project.
Why Offline-First Was Non-Negotiable
Most AR applications degrade into unusable experiences without connectivity.
We intentionally engineered the opposite model:
The application should function fully offline and improve further when online.
This dramatically changes deployment viability for:
- campuses
- museums
- tourism zones
- archaeological locations
- rural heritage sites
Offline capability was not treated as a feature.
It became the foundation of the system.
System Architecture
1. Geo-Spatial AR Using ARCore
We used the Geospatial capabilities of Google ARCore to:
- anchor overlays to real-world coordinates
- stabilize content while moving
- maintain persistent positioning
This allowed the application to associate digital information with actual physical landmarks.
The biggest challenge was maintaining stability outdoors where tracking conditions constantly change.
2. Sensor Fusion for Spatial Accuracy
GPS alone is not reliable enough for immersive AR experiences.
We combined:
- accelerometer
- magnetometer
- GPS
to compute:
- heading
- orientation
- proximity
- movement direction
Distance calculations used the Haversine formula while filtering reduced jitter and instability.
This hybrid positioning model significantly improved outdoor AR usability.
3. Offline-First Data Layer
Instead of relying on external APIs for core functionality:
- monument metadata was stored locally
- JSON datasets seeded a Room database
- content retrieval occurred fully on-device
Benefits included:
- zero network dependency
- predictable performance
- lower latency
- faster startup
- better reliability
This architecture also reduced operational complexity.
4. Hybrid AI + Voice Interaction System
The application used a dual execution model.
Online Mode
When connectivity exists:
- queries are processed through cloud AI models
- richer contextual responses are generated
- conversational interaction becomes more dynamic
Offline Mode
When offline:
- local keyword-based intent detection activates
- responses are fetched from an embedded knowledge base
- text-to-speech remains functional
Pipeline:
Speech → Intent Detection → Knowledge Retrieval → Response → TTS
The important part was graceful degradation instead of complete feature failure.
5. Adaptive Intelligence Layer
One of the most practical engineering decisions was making the system network-aware.
Condition
System Behavior
WiFi / Strong Network
Full AR + AI features
Weak Connectivity
Reduced media + lightweight responses
Offline
Local knowledge + sensor-driven AR
This prevented the application from becoming unusable under poor network conditions.
Instead of binary online/offline behavior, the system adapted dynamically.
Engineering Challenges We Solved
AR Stability Outdoors
Outdoor AR is significantly harder than indoor demos.
Problems included:
- lighting variation
- tracking instability
- sensor noise
- GPS inaccuracies
We introduced:
- fallback positioning systems
- filtering and smoothing
- sensor-assisted orientation correction
to stabilize overlays.
Context Without Internet
Most intelligent systems fail entirely when disconnected.
We solved this through:
- structured local knowledge storage
- lightweight intent matching
- embedded offline responses
This preserved usability without requiring network access.
UX Without Friction
Hackathon projects often overload users with features.
We intentionally simplified interaction.
The UI focused on:
- minimal onboarding
- clean navigation
- low cognitive load
- fast transitions
The goal was reducing interface complexity so the environment itself remained the focus.
Impact Beyond a Demo
This project was more than an AR prototype.
It represented a new way of thinking about contextual computing.
Cultural Impact
The system enables:
- preservation of local history
- interactive storytelling
- stronger engagement with heritage spaces
Instead of static information, locations become narrative environments.
Economic Impact
Potential applications include:
- tourism enhancement
- museum modernization
- reduced dependency on physical guides
- higher visitor retention
Offline capability also reduces infrastructure costs.
Social Impact
The project improves accessibility through:
- voice interaction
- intuitive navigation
- educational exploration
- multilingual expansion possibilities
Technology becomes more inclusive when it works under constrained conditions.
Why Samsung PRISM Recognized It
The Samsung PRISM Metaverse Hackathon emphasized practical innovation rather than visual novelty.
The project stood out because it focused on deployment realism.
1. Real-World Usability
The system worked outside controlled environments.
That matters far more than polished demos.
2. Mobile-First Engineering
Everything was optimized around:
- battery constraints
- mobile compute limitations
- unreliable connectivity
- practical user behavior
3. Scalable Architecture
The architecture can extend beyond campuses into:
- museums
- smart cities
- tourism systems
- heritage preservation networks
4. Future Metaverse Relevance
The project explored a practical version of spatial computing:
- physical environments
- contextual digital overlays
- AI-assisted interaction
without requiring expensive hardware ecosystems.
What Makes This Different
Most AR systems focus only on visuals.
This project focused on systems engineering.
The differentiation came from:
- offline-first architecture
- adaptive intelligence
- voice integration
- sensor fusion
- deployment realism
The innovation was not merely adding AR.
It was making AR reliable.
Key Insight
AR itself is not the innovation.
Reliability, accessibility, and contextual relevance are.
That became the biggest takeaway from the project.
What We Learned
Building the system taught us several important lessons:
- real-world constraints matter more than feature count
- offline capability is a massive engineering advantage
- UX simplicity beats technical complexity
- adaptive systems outperform rigid architectures
- systems thinking wins hackathons
The strongest projects are not the most visually impressive.
They are the ones that survive reality.
Future Roadmap
There are several directions we want to explore further:
- 3D interactive historical reconstructions
- occlusion-aware rendering
- multilingual conversational support
- larger geospatial deployments
- smarter offline semantic retrieval
- collaborative AR exploration
The long-term vision is a scalable contextual computing platform for physical spaces.
Conclusion
Campus AR Heritage Guide started as a hackathon experiment.
But the deeper realization was this:
Technology becomes powerful when users stop noticing the technology itself.
When users are not thinking about:
- AR frameworks
- sensors
- AI models
- networking layers
and instead experience:
- history
- context
- place
- storytelling
the system succeeds.
Final Thought
This project was never really about augmented reality.
It was about augmenting understanding.