By Marshall S. Bennett

Introduction

Augmented Reality, or AR, is a form of virtual reality where the real world is retained as part of the experience, and computer generated 3-D content is overlaid on top of the environment and can be interacted with as though it was an actual object placed in the real world. Augmented Reality can range from many different categories, such as constructive AR, where the information and content added is additive to the real world, or destructive AR, where the real world is less of a factor and is more covered up. As such, there are thousands of applications for Augmented Reality, and this article will do its best to explain the existing technologies, and how they can be used in the future.

Background

Augmented Reality, while a relatively new technology, was first envisioned to a wider audience back in 1901 by the famous author, L. Frank Baum. In his short story “The Master Key”, Baum mentions a “Demon of Electricity” who produces a device called a Character Marker. These spectacles would project a letter onto everyone they meet, marking their true character. The glasses idea would prove to be precognizant, as one of the first mainstream technologies that brought AR to the wider world would be the Google Glass. The Google Glass was one of the first modern examples of Wearable AR, where the technology was directly displayed via a headset shaped and weighing around the same as a pair of eyeglasses (in fact, prescription frames could be made for those that required glasses). Many people at the time predicted that we would do away with phones altogether, and we would instead have wearable headsets that would completely replace all other computers and electronic devices. However, the technology wasn’t ready and was often too clunky, restrictive, and expensive. However, as smartphones began to gain popularity and improve in technology, AR became a part of technologies that we use daily, even without thinking about it, included in our smartphones and apps that are a part of everyday life.

Modern Augmented Reality first works by creating a map of the environment around the device. Technologies like LIDAR and Cameras on the device create a 3D model of everything in the field of view, and algorithms use technologies such as corner detection and interest points to detect features. These are normally used to then create a canvas from which to affix the AR experience that will be projected onto that environment. By keeping track of those points of interest, as well as using external sensors such as gyroscopes, accelerometers, and GPS data, the device will detect movement, and will be able to adjust the project of the object accordingly, whether it is making it smaller if it is “moved away from”, or by showing it from another angle as the device is moved around the point where it has “placed” this virtual object. Techniques such as SLAM (Simultaneous Localization and Mapping) are also used in the event that some of the environment is unknown and the perspective changes. Through these processes, devices can use Augmented Reality to project realistic images that can appear to exist in, and even interact with. the real world.

Types and Uses of Augmented Reality

Wearable Augmented Reality

As mentioned before, the public’s first experience with Augmented Reality came at the hands of the Google Glass. However, that wasn’t the first piece of wearable technology to be created. Like many technologies, the first applications for augmented reality were developed for military forces. The Battlefield Augmented Reality System (or BARS) was developed by the US Naval Research Laboratory in the late 1990s and early 2000s as a way to assist soldiers going into low visibility environments. The system had a way of highlighting the walls of rooms, important waypoints, other humans, and even routes of entrance and escape. It required a full backpack that contained a GPS unit, a large network receiver, and even a GPS antenna. These would all be connected to a heads-up display worn like a pair of goggles. While this technology was revolutionary, it was ruled too clunky to be used in a battlefield, and was never widely used until it was made much smaller.

1. The hardware for the prototype BARS system. Julier

The Google Glass was the next major foray into Augmented Reality. As mentioned before, the Google Glass was developed as a way of wearing around your computer or smartphone. The Google Glass had a single camera, a touchpad for controlling the computer (much like the one on a laptop), and a small heads up display. While far from immersive, it allowed a small window into the world of augmented reality. For example, when at the airport, you would immediately see details regarding your specific flight. Once again, the Google Glass was considered clunky, niche, and extremely expensive, boasting a price tag of $1,500 when it was released in 2013.

2. A User Wears the Google Glass. Ariel Zambelich for Wired

Wearable technology isn’t entirely a thing of the past, however. One of the major features of the brand new F-35 planes being rolled out to militaries around the globe by Lockheed Martin is its Helmet Mounted Display System (or HMDS). This system provides pilots with up to the second data such as air speed, temperature, position, and even error data in the event of a malfunction. However, the coolest features come from its AR capabilities in improving vision. For ages, pilots would be required to wear separate night vision goggles to fly night missions, increasing cost and operational complexity. With the HMDS, the night vision is implemented directly into the helmet, and can take advantage of the thousands of sensors the F-35 has to get an even better picture of the world around them. That capability extends to removing blind spots as well. The biggest blindspot for a pilot is below the plane. With the new HMDS, pilots can simply look down, and the fuselage will simply be removed as sensors below the plane send data to the heads up display, where a complete picture is projected onto the screen. With these innovations in Augmented Reality, pilots can improve their own safety, as well as their mission success rate.

3. The Heads-Up Display of the F-35 Helmet. K2 Communications for National Defense

Marker Based Augmented Reality

Marker Based Augmented Reality requires a designated Marker or Symbol to begin and place the Augmented Reality Experience. These are often used as part of a museum or other educational experience, or as part of a product promotion. By scanning a QR code (that is presumably a set size, at an easy to detect orientation, and usually is one of only a couple of options), Users can see a set 3D model that, while they cannot fully interact with it, can be looked around, and expanded upon via a few preset options. In the example below, this technology is showing a miniature model of Big Ben being projected onto a set QR code. The model is colorized so as to highlight its different components and main features. There are even options to expand informational boxes that elaborate on the history and architecture of the structure. This technology was, and is common in educational attractions, such as Museums and Zoos, in order to augment their existing physical attractions, and engage younger tourists and attendees. It is a fun, cheap, and interesting way to ensure that people fully get to experience exhibits, and learn about intricacies that could be harder to promote solely in a traditional exhibit. This technology was once groundbreaking, however, it has since been eclipsed by Markerless Augmented reality.

4. A phone scans a QR code and displays an annotated model of Big Ben on top of it. ZealAR.

Markerless Augmented Reality

Markerless Augmented Reality comes in two major categories.

The first are filters, where the augmented reality must still be put on a specific object, but that specific object isn’t standardized like the QR codes or Symbols present in Marker Based Augmented Reality. An example of this are the Snapchat AR face filters. These filters still require that a face is in the frame to place the filter on. However, the filter can be used on any and all faces, and will work, regardless of the differences between them. Using a data set based on thousands of pictures of different faces from throughout the world, Snapchat marked out where a person's eyes, ears, mouth, and nose are on the pictures, and the algorithm learned to recognize them on other images as well. When you open your camera within Snapchat, it immediately makes a 3D map of your face, and tracks where specific points on your face, such as your eyes and mouth, are. It then takes the specific filter, and adjusts it to fit your face before matching it with the points on your face it was designed to fit next to. For example, a beard filter might take into account where your ears and base of your chin are, or a glasses filter might take into account where your eyes and nose are. These points can even be adjusted to fit filters that change when you move your face in specific ways, such as a filter that makes you vomit rainbows when you open your mouth. By measuring the distance between points on your upper and lower lip, and detecting when your teeth and the inside of your mouth are visible, the filter can then activate, placing the experience in the right place on your face.

5. The Author of this paper uses a “Minion Filter” on Snapchat, showcasing how the filter can find his skin and turn it yellow, as well as surrounding his eyes with goggles.

Another type of Markerless Augmented Reality is the kind that can be placed anywhere, and isn’t reliant on a consistent structure, such as a face. The applications of this are vast, and all it requires is a wide open empty flat space. Possibly the most famous example of this would be the viral mobile game, Pokémon GO. Pokémon GO is by far the most popular game from game developer Niantic, who combines location and camera data to turn the real world into game boards. While their games range in subjects from Transformers, to Harry Potter, to even the NBA, Pokémon GO remains their most popular and influential game. It also is a great example of how Augmented Reality technology has evolved over time. When the game first came out, the Pokémon on the screen were little more than stickers. Occasionally, they could be moved around, but they felt isolated from the world, and could not be significantly interacted with. However, as technology evolved more over time, the Pokémon could be much better placed in the real world, even circled around, and petted. This culminated in the “reality blending” feature, where, once a Pokémon was placed in the real world, should an object pass in front of it, such as a passing pedestrian, the device would recognize it, and stop generating parts of the model, much as though the model was actually in the real world. This feature was revolutionary, and the fact that it is being rolled out to the masses is a testament to how much the technology has advanced over time.

6. Two AR snapshots from Pokemon Go showing how an AR based object can be walked “around” as perspective changes

Markerless Augmented Reality technology can have other uses, however. One that can save lives are medical applications. Oftentimes, it can be hard for surgeons to find a specific point in the body, and getting a full, unobstructed picture of the problem directly can be difficult. Augmented reality is being used to overlay medical images, techniques, and labels onto patients in the Operating Room, assisting doctors in treating and performing surgery on patients. For patients with a more specific problem, CT scans can even be overlaid, allowing doctors a template to properly ensure they are more precise than ever before.

7. Doctors use AR to get a better picture of a patient’s brain before surgery. Sosna.

In addition, Augmented Reality can be used in products and sales. It can be hard to get proper measurements for a space for a major piece of equipment or furniture, and oftentimes, a client would like to see how an object looks in a space before fully committing to purchasing it. However, offering to drag objects and set them fully up in a space without a guaranteed payoff or purchase is risky and expensive to sellers. However, AR allows a customer to place a piece of furniture in their own homes and fully see how it looks without having to risk bringing it in or out of their residence. Retailers like IKEA and Amazon pioneered this feature, and now it is an expected feature of every store that sells bigger purchases, such as furniture and appliances.

8. The IKEA app placing an AR version of the ADDE chair (left) in the author’s living room next to the real thing (right).

Finally, AR can be used to help us expand our dominion as a species and traverse the stars. NASA’s use of Augmented Reality (AR) aboard the International Space Station (ISS) has already proven to be a game-changer, and its potential for future missions is even more exciting. AR is helping astronauts operate more independently, providing hands-free, step-by-step guidance for repairs and experiments through devices like HoloLens. This capability is critical in space, where traditional support from Earth can be delayed by communication lags. In the future, NASA plans to expand AR’s role in training, allowing astronauts to prepare for complex tasks in simulated environments that mirror the challenges of space. AR could also revolutionize scientific research, overlaying real-time data directly onto equipment or specimens, enabling faster and more accurate analyses. As NASA sets its sights on deep-space missions to the Moon and Mars, AR will be indispensable for navigation, habitat construction, and managing life-support systems in harsh and unfamiliar environments. These tools will empower astronauts to troubleshoot and innovate on the fly, ensuring mission success even when Earth is millions of miles away. Through these advancements, AR is not just enhancing daily life on the ISS but also shaping the future of human space exploration.

9. NASA Astronaut, Scott Kelly uses the HoloLens device to increase efficiency aboard the ISS. NASA

Technical Frameworks and Challenges in Augmented Reality

While the applications of Augmented Reality are wide-ranging and exciting, it is equally important to explore the technical underpinnings that make these applications possible. AR's ability to seamlessly integrate virtual content into real-world environments relies on a complex interplay of hardware, software frameworks, and algorithms. This section delves into the key technical aspects of AR, shedding light on the tools and challenges developers face when building AR experiences.

Hardware Foundations of AR

The effectiveness of any AR application begins with the hardware it runs on. Modern AR systems leverage a combination of sensors, cameras, and processors to create an immersive experience. For instance: Cameras and Depth Sensors are crucial for capturing and mapping the real-world environment. Advanced AR devices, such as Apple’s iPhone with LiDAR or the Microsoft HoloLens, use depth sensors to create highly accurate 3D models of their surroundings. IMUs (Inertial Measurement Units), such as accelerometers, gyroscopes, and magnetometers, work together to track the device’s orientation and movement in space. For example, when a user tilts their phone to view an AR object from a different angle, the IMU ensures the virtual object adjusts accordingly. High-performance processors, often paired with GPUs (Graphics Processing Units), handle the computationally intensive tasks of rendering 3D graphics, analyzing sensor data, and maintaining real-time interactions. One significant challenge for AR hardware is balancing power consumption with performance. Devices must be lightweight and portable, yet capable of performing complex computations without draining battery life—a problem that has driven innovations like energy-efficient chipsets and cloud-based rendering.

Software Development Frameworks

Creating AR applications requires specialized software frameworks that provide developers with the tools to build, test, and deploy their ideas. Three major frameworks dominate the AR development landscape:

1. ARKit (Apple): ARKit is a framework designed for iOS devices, leveraging Apple's advanced hardware like the A-series and M-series processors. Features such as motion tracking, face tracking, and scene reconstruction are built into the platform, making it easy for developers to create sophisticated AR experiences. ARKit’s "RealityKit" framework further simplifies the development process by providing high-level APIs for physics-based interactions and photorealistic rendering.

2. ARCore (Google): ARCore serves as the equivalent framework for Android devices. It offers similar capabilities to ARKit, including environmental understanding, motion tracking, and light estimation. ARCore also supports features like Augmented Images and Augmented Faces, allowing developers to overlay content onto specific surfaces or facial landmarks. Google’s commitment to cross-platform functionality means ARCore applications can often run on iOS devices as well.

3. Unity and Unreal Engine: These are versatile game engines that have become staples for AR development. Unity, for instance, offers an AR Foundation package that supports both ARKit and ARCore, enabling developers to create cross-platform applications. Unreal Engine’s AR capabilities, powered by its "AR Developer Framework," are known for their ability to produce photorealistic graphics, making it a favorite for high-end AR experiences.

Despite these robust tools, one of the challenges developers face is maintaining compatibility across devices. Variations in hardware capabilities, operating systems, and sensors require extensive testing and optimization to ensure consistent user experiences.

Core Algorithms Behind AR

The seamless blending of virtual objects into the real world hinges on a suite of sophisticated algorithms:

SLAM (Simultaneous Localization and Mapping): SLAM allows AR devices to map their surroundings while keeping track of their own location within that map. This algorithm is particularly important for markerless AR, enabling objects to remain anchored even as users move around.

Image Recognition and Tracking: Marker-based AR depends on algorithms that detect and recognize specific patterns, such as QR codes or images, and align 3D content with them. Advances in computer vision, including neural networks, have significantly improved the accuracy and speed of these processes.

Occlusion Handling: For AR objects to appear truly embedded in the real world, they must interact realistically with physical objects. Occlusion algorithms ensure that parts of a virtual object are hidden when obstructed by real-world elements, enhancing the illusion of depth and presence.

Each of these algorithms comes with computational challenges. For example, SLAM requires a delicate balance between speed and accuracy, as real-time mapping must be computationally efficient without sacrificing the quality of the virtual overlay.

Technical Challenges in AR Development

Developers encounter a range of challenges when creating AR applications:

Environmental Limitations: Poor lighting conditions, reflective surfaces, or lack of distinct features can hinder AR tracking and object placement. While advancements like LiDAR have mitigated some of these issues, they remain significant obstacles for consumer-grade devices.

User Interaction: Designing intuitive user interfaces for AR is complex. Developers must consider how users interact with both physical and virtual objects, ensuring that gestures and controls feel natural.

Latency: Real-time interactions are critical for AR. Any delay in rendering or responding to user input can break immersion. Optimizing performance to minimize latency is a persistent challenge, particularly on mobile devices with limited processing power.

Scaling for Multi-User Experiences: Collaborative AR applications, such as multiplayer games or shared workspaces, require precise synchronization across devices. Achieving this level of accuracy involves overcoming hurdles in network latency and data consistency.

The Future of AR Technology

As AR continues to evolve, emerging technologies like 5G and edge computing promise to address some of these challenges. Faster network speeds and reduced latency will enable cloud-based processing, offloading intensive tasks from devices and improving battery life. Additionally, advances in machine learning will likely enhance the accuracy of algorithms, making AR experiences even more realistic and accessible. By understanding the technical frameworks and challenges of AR, we not only gain a deeper appreciation for the technology but also glimpse the immense potential it holds for reshaping our interaction with the world around us. AR is not just a tool for enhancing reality, it is a frontier that continues to push the boundaries of what is possible.

The Intersection of AR and Neuralink

The combination of augmented reality (AR) and Neuralink’s brain-computer interface (BCI) represents an area for incredible growth and future development. By integrating direct neural interaction with AR experiences, this fusion could redefine how we interact with digital content, process information, and even communicate. The intersection of AR and BCIs has an incredible potential, mostly because of the speed at which users would be able to get work done and the potential for free-form hands-free work.

Hands-Free AR

One of the most immediate changes Neuralink could bring to AR is the elimination of traditional input devices. Current AR interfaces rely on hand gestures, eye tracking, and voice commands, all of which have inherent limitations. Neuralink’s brain-chip technology, however, could allow users to navigate AR interfaces purely through thought.

Users would be able to summon virtual screens, manipulate 3D objects, or compose messages in midair near instantly just by thinking about them. This would provide an unprecedented level of convenience, particularly in high-intensity fields such as medicine, where surgeons could interact with patient data or anatomical overlays in real time without breaking focus. Similarly, architects and designers could visualize, modify, and rotate 3D models instantaneously without the need for physical controllers. Realistically, any field of work that requires visualization and large amounts of data to be viewed would benefit from this potential technology.

AR as a Direct Brain Extension

Neuralink’s potential to read and interpret brain activity could lead to AR becoming an extension of human conciousness and thought. Instead of requiring external screens or glasses to display information, AR content could be streamed directly into the brain’s visual cortex, making digital overlays feel indistinguishable from real-world perception.

While this is obviously a stretch from today's technology, this could be the most revolutionary technology to ever exist. For instance, at it's most simple level a student learning a new language could see real-time translations of text and speech in their field of vision, seamlessly integrating knowledge into their daily experiences. Professionals could retrieve crucial data at a moment’s notice—an engineer repairing complex machinery could see step-by-step instructions appear in their mind’s eye without ever glancing at a manual. To make it clear just how groundbreaking this invention could be consider that this would be a technology that sits between human conciousness and human perception. While this would only, in the simplest cases, insert some data or pull some thoughts from the users head, the potential exists for a technology that can completely change human perception and life itself.

Enhanced Communication and Telepathy

An example of a more groundbreaking possibility of combining AR and Neuralink is redefining how humans communicate. Current digital communication relies on text messages, voice calls, and video chats, but BCIs could enable a new form of direct brain-to-brain interaction. Instead of typing out messages, users could “think” words, images, or even emotions into existence and transmit them instantly.

This could create an entirely new form of AR-enhanced telepathy. Imagine engaging in a virtual meeting where participants not only hear words but also “see” thoughts as interactive AR objects. Two designers working on a new concept could share mental blueprints and collaborate on visualized prototypes without speaking a word. This level of instantaneous, high-fidelity interaction is like something out of a sci-fi movie, yet the technology to do it is in (early) development right now.

Medical and Accessibility Breakthroughs

For individuals with physical disabilities, the synergy between AR and Neuralink could provide incredible independence. Those with paralysis could use AR interfaces to control smart home devices, navigate virtual spaces, or even drive robotic exoskeletons—restoring mobility and agency through neural commands.

Beyond mobility, AR combined with BCIs could help those with visual or auditory impairments by enhancing sensory input. AR overlays could provide real-time descriptions of surroundings for the visually impaired, while those with hearing loss could receive visual subtitles that are directly interpreted by their neural interface.

Conclusion

Technology for Augmented Reality (AR) has not only evolved remarkably over time but continues to display incredible diversity in its applications today. This diversity is not a limitation but rather an opportunity, as it enables AR to be tailored to countless uses that enhance and enrich our daily lives. Through the integration of AR into various facets of modern living, the way we work, shop, learn, and play is being fundamentally transformed. Jobs can be performed with greater precision and efficiency, consumers are empowered with better access to information, and entertainment has become more interactive, engaging, and immersive. The journey of AR from a futuristic concept to an integral part of our lives is a testament to its transformative potential. For instance, early visions of AR, like L. Frank Baum’s speculative “Character Marker” in 1901, were fantastical yet oddly prescient, foretelling devices like Google Glass. Google Glass, while not perfect, introduced wearable AR to the mainstream and demonstrated the promise of having information accessible at a glance. Though its initial form was bulky and expensive, its legacy laid the groundwork for wearable AR systems like the advanced Helmet Mounted Display Systems used in modern F-35 fighter jets, which not only improve pilot safety but also enhance mission effectiveness. The leap from a $1,500 experimental headset to a military-grade tool underscores how AR has evolved to meet diverse needs, from convenience to critical functionality. On a more accessible level, AR has seamlessly integrated into our daily routines through smartphones and apps. Marker-based AR, for example, allows for interactive educational experiences in museums or augmented promotional campaigns by projecting 3D models onto predefined symbols. Meanwhile, markerless AR, as seen in Snapchat filters and Pokémon GO, has brought the technology to our fingertips in fun and engaging ways. These innovations have grown from simple overlays to dynamic, context-aware projections, with advanced techniques like SLAM enabling realistic and interactive virtual objects to coexist with real-world environments. Such advancements highlight AR’s progression from novelty to a reliable and versatile tool. The practical benefits of AR are equally impressive. In healthcare, AR assists surgeons with overlays of critical medical data during procedures, improving precision and outcomes. Retailers have embraced AR to help customers visualize furniture or appliances in their own homes, removing barriers to purchase decisions and enhancing customer confidence. These applications showcase AR’s ability to address real-world challenges, whether it’s saving lives in the operating room or making home renovations easier. In entertainment, AR continues to redefine how we engage with content. Pokémon GO, for instance, began as a simple game but evolved into a platform that blends digital creatures seamlessly into our environment, utilizing advanced "reality blending" techniques. By allowing virtual elements to interact with real-world objects, AR has bridged the gap between digital and physical experiences, creating a sense of presence and interactivity that was previously unimaginable. As AR technology continues to evolve, the possibilities are boundless. It’s reshaping industries, revolutionizing everyday tasks, and opening up new avenues for creativity and productivity. The innovations we see today are just the tip of the iceberg, promising a future where AR isn’t just an enhancement to our reality but a fundamental part of it. Through AR, we are not only augmenting our environments but also enriching our lives in ways that were once confined to the realm of science fiction.

References

1. https://www.sap.com/products/scm/industry-4-0/what-is-augmented-reality.html#:~:text=Augmented%20reality%20definition,real%2Dlife%20environments%20and%20objects.
2. https://computer.howstuffworks.com/augmented-reality.htm
3. https://www.onirix.com/learn-about-ar/types-of-augmented-reality/
4. https://www.reydar.com/how-does-augmented-reality-work/
5. https://www.jasoren.com/what-augmented-reality-is-and-how-it-works-the-ultimate-tutorial/
6. https://www.aircards.co/blog/markerless-vs-marker-based-ar-with-examples
7. Julier, Simon & Baillot, Yohan & Lanzagorta, Marco & Brown, Dennis & Rosenblum, Lawrence. (2004). BARS: Battlefield Augmented Reality System.
8. https://www.banuba.com/blog/snapchat-filter-technology-whats-behind-the-curtain
9. https://www.collinsaerospace.com/what-we-do/industries/military-and-defense/displays-and-controls/airborne/helmet-mounted-displays/f-35-gen-iii-helmet-mounted-display-system
10. https://www.bannerflow.com/blog/pokemon-go-augmented-reality-changing-marketing
11. https://pokemongolive.com/post/realityblending-announcement/?hl=en
12. https://nianticlabs.com/news/realityblending-announcement
13. https://www.wired.com/2013/12/glasshole/
14. https://www.nasa.gov/missions/station/nine-ways-we-use-ar-and-vr-on-the-international-space-station/
15. https://developers.google.com/ar/develop/unity-arf/getting-started-ar-foundation
16. https://developer.apple.com/augmented-reality/arkit/
17. https://unity.com/solutions/xr/ar
18. https://www.unrealengine.com/en-US/xr
19. https://www.electronicspecifier.com/products/vr-ar/imu-improves-virtual-and-augmented-reality-applications
20. https://www.andreasjakl.com/basics-of-ar-slam-simultaneous-localization-and-mapping/

Image Citations

1. Julier, Simon & Baillot, Yohan & Lanzagorta, Marco & Brown, Dennis & Rosenblum, Lawrence. (2004). BARS: Battlefield Augmented Reality System.
2. https://www.wired.com/2013/12/glasshole/ Ariel Zambelich for Wired
3. https://www.nationaldefensemagazine.org/articles/2018/10/31/navys-f-35-helmet-problem-fixed-with-tv-technology K2 Communications
4. https://zealar.com.au/explore-how-marker-based-ar-works-for-your-business/
5. Self Created Image
6. Self Created Images
7. https://medcitynews.com/2019/09/the-benefits-of-ar-in-healthcare/
8. Self Created Image
9. https://www.nasa.gov/missions/station/nine-ways-we-use-ar-and-vr-on-the-international-space- station/ NASA