Brick Block in AR

Years Active: 2022-2023

Brick Block in AR is an experimental project that explores procedural building in augmented reality using mixed-initiative techniques. Drawing on concepts from Townscaper and the Wave Function Collapse (WFC) algorithm (reference, project), this prototype investigates the challenges and opportunities of 3D block-based interaction through mid-air hand gestures on Microsoft HoloLens 2.

• Project Type: Prototyping, Interaction Design, AR, Procedural Content Generation
• Tech Stack: Unity3D, Microsoft HoloLens 2, Mixed Reality Toolkit (MRTK), Hand Gesture Interface, Wave Function Collapse
• Demo, code, or feedback? Check out the GitHub Repo

Demo Video

Technical Challenges & Lessons Learned

• Gesture Design: Mapping place/select/delete/undo to gestures is challenging with the limited HoloLens hand tracking capabilities. I iterated through a user dashboard for switching these functions.

• Block-Based Interaction in 3D: Picking and placing blocks with just hands is ambiguous. Where are you pointing? What do you mean by select? I explored raycasting, dynamic snapping, and contextual cues to help.

• AR UX for Creation Tools: Direct manipulation and AI suggestions were called for a creative UI, featuring floating menus, block previews, and an “auto-generate” function. AR interaction revealed lots of hidden friction!

Research & Future Potential

Brick Block in AR is an ongoing technical playground for:

• Comparing mid-air gestures, controllers, tangible media, and 2D tablets for complex 3D design tasks
• Studying cognitive workload and UX in mixed-initiative, AI-assisted tools
• Prototyping rapid architectural/engineering workflows where human and machine co-create structures in real-time