Digital Twin FT UGM

Overview

Digital Twin FT UGM is an interactive, browser-based digital twin and virtual tour of the Fakultas Teknik Universitas Gadjah Mada (FT UGM) campus. Visitors explore a real-time 3D model of the campus, click buildings to zoom in and open details, browse floor plans with clickable room markers, view 360-degree panoramas, and read FAQ/About content — from desktop or mobile, and installable as a Progressive Web App.

The problem it solves is remote access: prospective students, parents, and the general public cannot always visit the physical campus, and a flat map or photo gallery cannot convey the spatial layout. The platform replaces that gap with an immersive, photogrammetry-derived environment. The underlying 3D asset was produced from drone aerial photos processed in Agisoft Metashape, edited and texture-baked in Blender, and Draco-compressed for the web; a usability study reported 84.6% satisfaction.

The application is deliberately client-side only — no backend, database, or CMS. All content (landmarks, tours, FAQs) is authored as static TypeScript modules that are bundled at build time, and only images are fetched from a CDN. Target users are prospective students and campus visitors exploring FT UGM remotely.

Tech Stack & Rationale

Architecture

The app is a single-page, client-side 3D experience built on the Next.js Pages Router. index.tsx mounts a StoryBoard landing overlay on top of the R3F <Canvas>; interaction flows through Redux, which drives the 3D camera, content panels, and the map/panorama/gallery views.

The flow runs: index.tsx (StoryBoard + Canvas) → Scene (R3F Canvas) (camera, lights, PerformanceMonitor) → Model + Grass (useGLTF('object/map-min.glb')) → Redux (navigation) (dispatch location/content); Control (OrbitControls) (camera zoom to landmark) → Static data modules (Landmarks.ts · Tour.ts · Faq.ts) → Cloudinary (panorama + denah); [content].tsx panels (Map · Panorama · Gallery · Faq).

Data flow walkthrough:

1. pages/index.tsx renders the R3F <Scene> inside a full-screen canvas, with the StoryBoard landing overlay gating entry.
2. Scene (in src/components/canvas/Scene.tsx) configures the Canvas (camera, near/far, frameloop='demand'), lights, and a PerformanceMonitor that adapts the device pixel ratio.
3. Model loads the Draco-compressed campus model via useGLTF('object/map-min.glb'), maps each landmark to a tooltipLocation, and renders hover tooltips with Drei's Html; Grass adds displaced terrain geometry.
4. Clicking a building dispatches toggleLocation/toggleContent to the Redux navigation slice and performs a shallow router push (/landmark?location=…).
5. Control reads the selected landmark and animates the OrbitControls camera to its zoomCamera target.
6. [content].tsx routes to the active content panel — Tour, Landmark, Faq, or About — which pull their data from the static Landmarks.ts / Tour.ts / Faq.ts modules.
7. The Map (React Konva + react-map-interaction), Panorama (Pannellum), and Gallery (lightbox) components render floor plans, 360 views, and photos whose image URLs point to Cloudinary.
8. A localStorageMiddleware in the Redux store persists theme, music, and firstTutorial across sessions; there is no server, database, or CMS at runtime.

Folder Structure

digital-twin-ft-ugm-2023/
├── public/
│   ├── object/                 # map-min.glb, baked-min.glb (3D models)
│   ├── img/                    # logos, icons, static images
│   ├── audio.mp3               # background music
│   └── manifest.json           # PWA manifest
├── redux/                      # Redux Toolkit store
│   ├── store.ts                # configureStore + localStorage middleware
│   ├── navigation.ts           # navigation slice (theme/location/content/…)
│   └── hooks.ts                # typed useAppDispatch/useAppSelector
├── src/
│   ├── components/
│   │   ├── canvas/             # WebGL scene: Scene, Model, Control, Grass
│   │   ├── content/            # Content, Tour, Landmark, Faq, About, Navbar
│   │   ├── Map/Map.tsx         # floor-plan map (Konva + map-interaction)
│   │   ├── panorama/Panorama.tsx  # Pannellum 360 viewer
│   │   ├── gallery/Gallery.tsx # photo grid + lightbox
│   │   ├── navigation/         # bottom bar, icons, weather
│   │   ├── Tutorial/           # joyride onboarding
│   │   ├── data/               # Landmarks.ts, Tour.ts, Faq.ts (content)
│   │   ├── StoryBoard.tsx      # landing overlay
│   │   ├── Background.tsx      # gradient/canvas background
│   │   ├── Loading.tsx         # drei useProgress loader
│   │   └── ErrorBoundary.tsx
│   ├── hooks/                  # useBackgroundAudio, useAccordion
│   ├── utils/sanitize.ts       # DOMPurify HTML sanitizer
│   ├── types/                  # data.ts, components.ts, redux.ts, *.d.ts
│   ├── pages/                  # _app, index, [content], 404, 500
│   ├── styles/index.css        # Tailwind directives + global CSS
│   └── config.tsx              # SEO/meta header
├── next.config.js              # PWA, i18n, webpack (audio/glsl), headers
├── tailwind.config.js
├── tsconfig.json
└── package.json

• redux/ holds the entire client state — a single navigation slice persisted via middleware.
• src/components/canvas/ is the WebGL layer (Scene, Model, Control, Grass) rendered by React Three Fiber.
• src/components/content/ holds the route-aware content panels and the top navbar.
• src/components/data/ is the content source of truth: large static TypeScript modules bundled at build time.
• src/pages/ is the Next.js Pages Router entry (index, dynamic [content], error pages).

Impact & Results

• A usability study of the virtual tour reported 84.6% satisfaction among users.
• Gives prospective students and the public remote, immersive access to the FT UGM campus on desktop and mobile.

My Contributions

My main work on this project:

• Migrated the codebase from JavaScript to TypeScript and pinned the TypeScript version with tuned tsconfig settings.
• Built the 3D scene and camera controls, including OrbitControls with animated zoom-to-landmark behavior.
• Implemented the Grass terrain component (custom displaced geometry) in the WebGL scene.
• Maintained the Model GLB loader and hover tooltip mapping for campus buildings.
• Built responsive/mobile layouts for the navigation and control elements.
• Optimized the web bundle and runtime performance of the 3D experience.

References

[1] three.js, "JavaScript 3D library," threejs.org. [Online]. Available: https://threejs.org/ [2] pmndrs, "React Three Fiber," GitHub. [Online]. Available: https://github.com/pmndrs/react-three-fiber [3] pmndrs, "Drei — useful helpers for react-three-fiber," GitHub. [Online]. Available: https://github.com/pmndrs/drei [4] Konva, "React Konva — 2D canvas for React," konvajs.org. [Online]. Available: https://konvajs.org/docs/react/ [5] Pannellum, "A lightweight 360 panorama viewer," pannellum.org. [Online]. Available: https://pannellum.org/ [6] Redux Toolkit, "The official, opinionated, batteries-included toolset for Redux," redux-toolkit.js.org. [Online]. Available: https://redux-toolkit.js.org/ [7] next-pwa, "PWA plugin for Next.js," GitHub. [Online]. Available: https://github.com/shadowwalker/next-pwa [8] Agisoft, "Metashape — photogrammetric processing," agisoft.com. [Online]. Available: https://www.agisoft.com/ [9] Google, "Draco 3D compression," GitHub. [Online]. Available: https://github.com/google/draco