7. Camera based sensor technology - RichiWolf/HHZ_Projekt_Measurement-of-shopping-time-in-supermarkets GitHub Wiki
Indoor navigation enables quick orientation in confusing, large shopping centres with an extensive range of products. In addition, the added value for the customer can be further increased by linking to location-based services, which enable customers to receive, for example, an app, discount coupons, product information or complementary product offers (Mietzner, Vandenhouten, Schultz, p. 58).
In the following, three different approaches of the camera-based technology are presented, which enable indoor navigation. These are Stereo Cameras, Landmarks and Visible Light Communication (Li-Fi).
One projects, which concerns with stereo cameras, is the project Easyliving by Microsoft. This requires Triclops stereo cameras, each connected to its own PC. After installing the two stereo cameras, calibration is required to compare the data supplied. Each part of the room is captured by at least one camera. The camera captures the colours and depths of the room. The PCs run the "Stereo Module" program, which processes the registered colour and depth images and generates reports on people's locations. The system measures the position at about 10cm above the ground and maintains the identity when people move on color histograms. In addition, the system allows the creation and deletion of personal representations. It automatically creates new instances of people as they cross a specific region in space. This is the entrance, for example. The cameras are mounted in the room on the walls at a height of about 2.3 meters. The Person Tracker calculates the current speed and a possible position from the person's previous positions. (Kistenich, Retscher, p. 32 f.)
Required Hard- and Software (Brumitt, Harris, Krumm et al., p. 3)
Advantage:
- User gets along without any adaptations. He can move completely free and is not restricted in his latitude
Disadvantages:
- Each room must be equipped with two stereo cameras and three computers (Brumitt, Harris, Krumm et al., p. 4)
- Leads to space and cost problems (Brumitt, Meyers, Krumm et al., p. 3)
- Tracking only for up to three people at a time. More than three people create poor clustering in the stereo module (Brumitt, Meyers, Krumm et al., p. 7)
Provider of stereo cameras:
- Stereolabs (450 USD)
- Robotshop (86,11€)
With the indoor navigation approach presented in this section, a robust solution with comparatively low investment requirements is presented, which enables different applications in the trade and creates a greatly improved shopping experience.
As shown in the figure "Systemstucture" below, the area in which the location is to take place will initially be equipped with landmarks (1). The landmarks can be two-dimensional (e.g. printed on a wall) or three-dimensional (e.g. sphere, cube, cuboid, hexagonal shapes). It is advantageous for the process if the landmarks are optically striking (e.g. cubes with coloured edges) so that they can be easily recognised. Each landmark should be positioned so that it is clearly visible from as many positions as possible in the area. For the identification of individual landmarks, they are visually clearly marked (e.g. by a two-dimensional barcode). If a landmark has several outer surfaces pointing in different directions of the area, it is advantageous if each of these surfaces has its own marking so that they can be distinguished and the viewing direction can be determined more reliably. The size, the number and the spatial density of the landmarks to be installed depend on the area of the area and the desired spatial resolution of the location. To locate the mobile device, the environment is recorded with its camera (2). With the aid of image processing algorithms, the landmarks in the digital image are detected and identified by means of the unique identification. The relative position and orientation of the mobile device in relation to the landmarks is then determined on the basis of the visible size of the landmarks, their position in the image and their perspective distortion. If the mobile device has acceleration and/or position sensors, these can also be used for a more precise determination of the relative position. The position and orientation on a map can be visualized on the display of the terminal device (3). The absolute positions of the landmarks and their orientation are stored either in the mobile device itself or on a server to which the mobile device connects wirelessly (e.g. via mobile radio or WLAN) and queries the desired data (4). In addition, location-related information (e.g. advertising) can be inserted into the camera image using augmented reality. The content can also be stored either on the device itself (e.g. in an app) or on one to which the device connects wirelessly. (Mietzner, Vandenhouten, Schultz, p. 60 f.)
Systemstructure (Mietzner, Vandenhouten, Schultz, p. 60)
Advantages:
- Cost-effective landmarks (Mietzner, Vandenhouten, Schultz, p. 61)
- No additional components or devices needed (smartphone only) (Mietzner, Vandenhouten, Schultz, p. 61)
- Low investment costs (Mietzner, Vandenhouten, Schultz, p. 61)
- The absence of a radio-based navigation solution contributes to a comparatively lower error rate of the navigation result (Mietzner, Vandenhouten, Schultz, p. 66)
- The method of optical basis provides stable, good navigation results (unlike WLAN or Bluetooth) (Mietzner, Vandenhouten, Schultz, p. 59)
Disadvantages:
- Smartphone battery is heavily loaded
- Smartphone must be constantly held in the hand/holder
- No tracking if camera does not recognize the landmarker
(Mietzner, Vandenhouten, Schultz, p. 61)
The end customer can navigate within a store with the help of his smartphone and thus find the shortest route to the product he is looking for. Such an application, which can be made available in the form of an app, improves the shopping experience and increases customer satisfaction. The retailer can provide the indoor navigation via an appropriate app to collect extensive data about customer behavior, assumed the customer allows this data collection. In a first step, the customer needs, e.g. regarding the availability of products, can be analyzed. In a second step, location-based services such as discount coupons or location and person-specific advertising can be used to increase customer satisfaction and revenue per purchase and customer. (Mietzner, Vandenhouten, Schultz, p. 58 ff.)
Another approach that has so far been implemented in practice are LED luminaires with VLC technology (Visible Light Communication) and a specific mobile application. VLC can transmit data using light for indoor navigation. VLC can transmit data using light. The light emanates from an LED or fluorescent lamp and must be received and converted by a photodetector, such as a smartphone camera or a single photodetector attached to the shopping basket.The luminaire "flickers" imperceptibly for people and thus transmits a code that can be used to determine the position. Large supermarkets are a possible application area. If the customer has the appropriate app on his device, he can be directed directly to the products on his shopping list or receive location-based offers. This technology is being researched among others by Philips and Fraunhofer. Each luminaire functions as transmitter and receiver. They radiate a circular surface onto the floor. The neighbouring luminaires receive the reflected light, as light reflects in all directions. The received brightness decreases with increasing distance. (Knobloch, Slomczyk, p. 1 ff.)
Functionality:
- Use of commercially available LED modules for data transmission
- Through a conventional data transmission interface (e.g. Ethernet), desired contents are sent to the LED lamp
- With the help of a modulator, the LED lamp is quickly switched on and off and the data is sent to the receiver (smartphone)
- Smartphone receives the light, decodes the information and converts it into electrical pulses
(Saini, p. 13)
Functionality of VLC
LBT (Listen before Talk) is a technique to check whether the transmission channel is free or not. For this purpose, the station willing to transmit listens to the transmission medium. If the medium remains free for a certain time, the station starts the transmission. If an active data transmission is detected, the station waits and then checks again whether the transmission medium is free. All stations in the network should be able to send their data. In larger networks, several stations can communicate simultaneously if they do not influence each other. In networks that operate according to the LBT method, the hidden station problem is known. To explain the hidden station problem, we assume three stations A, B, and C. One station can only communicate with another station if it is within its own radius. Stations A and C can't receive each other. Station B can receive both stations. Station A is currently sending something to B. If C also wants to send something to B, it checks the medium. Since station C does not see A, however, it assumes that the medium is free and starts sending. Now a collision occurs. However, this problem can be prevented using the Ready-to-Send (RTS) and Clear-to-Send (CTS) procedures. Request to Send, the request is to start sending. Clear to Send is the channel release. (Knobloch, Slomczyk, p. 4 f.)
Further facts about VLC:
- Transmission speeds up to 10 Gbit/s (Sharma, Raunak, Sanganal, p. 152)
- No interference with radio systems (Sharma, Raunak, Sanganal, p. 150)
- Data transmission with standard high-power LED lamps (Sharma, Raunak, Sanganal, p. 152)
- Combination of lighting and data communication (Omanakuttan et al., p. 2340)
- Cost-effective conversion possible (Sharma, Raunak, Sanganal, p. 153)
- Parallel operation of several VLC systems possible (Sharma, Raunak, Sanganal, p. 151)
- Easy data protection due to opaque surfaces (Sharma, Raunak, Sanganal, p. 153)
- Transmission standard IEEE 802.15.7 (Sharma, Raunak, Sanganal, p. 153)
- Robust data transmission without transmitter and receiver alignment (Sharma, Raunak, Sanganal, p. 153)
- Worldwide unregulated, freely available transmission frequencies (Sharma, Raunak, Sanganal, p. 153)
Advantages:
- No additional hardware required
- Flickering of luminaires not perceptible to the human eye
- No radiation exposure
- High data security (data is only transmitted in the respective light cone)
- Lower interference potential
- No radio technology
- No electrosmog
- Beacons can complement the position determination
- Modern LEDs save electricity
- Power supply not dependent on batteries
- VLC is accurate (under one meter) and has a long range (up to eight meters)
- Works cross-platform
(Saini, p. 14)
Disadvantages:
- As soon as something gets between lamp and photodetector (e.g. a hand) the transmission is affected
- Battery of the smartphone is loaded
- Less flexibility in installation than Beacons
- High costs with existing lighting infrastructure - retrofitting does not always make sense
- Return channel and tracking requires a additional App
- Relatively small light source on the smartphone must always be in direct visual contact with the central light source
(Saini, p. 14)
Provider:
- Philips
- Fraunhofer
Costs:
LED prices depend mainly on efficiency (watts/lumen), lifetime and brightness (lumens). LED prices are falling by 30 to 50 percent every 2 years due to ever more efficient production. The LED prices can be roughly divided into 4 categories:
Brightness | Cost |
---|---|
Up to 250 lumen | Between ~ 9€ - ~ 26€ |
Up to 470 lumen | Between ~ 13€ - ~ 55€ |
Up to 800 lumen | Between ~ 35€ - ~ 65€ |
Until 955 lumen | Between ~ 50€ - ~ 85€ |
Costs of LED's
In view of the use case developed in cooperation with a retailer, extensive research into all three approaches has shown that the VLC technology is the most suitable technology. The main reason against the use of stereo cameras are the costs. Since many such cameras are required for a supermarket, the time-consuming installation is not worthwhile. Moreover, as no more than three people can be tracked with stereo cameras, this is not suitable for a supermarket with many customers. The introduction of landmarks seems to be a viable alternative to VLC technology. However, since the VLC technology outweighs the advantages, this technology was found out to be the most suitable.
The free space communication with LED light offers applications that are easily compatible with the primary function of LEDs, namely signalling and lighting. Furthermore, these can be combined interference-free with other methods of free-space communication (infrared light and radio waves). There is a wide range of application examples for this technology, with exemplary advantages such as eavesdropping security, interference-free operation and the possibility of communicating at high speed under water. Such applications could lead to synergetic unique selling points for LED lighting and signalling and thus provide new arguments for a lighting market characterised by very high price pressure. Despite its technical maturity, VLC still has new things to discover and explore (Langer, Vučić, Walewski, p. 2).
Due to the latest developments in LED lighting technology, VLC can be used for simultaneous lighting and data transmission indoors and outdoors. VLC is a useful and promising addition to radio systems for future short-range applications. Although there are many challenges, such as overcoming annoying noise, VLC is expected to be one of the most promising technologies of the future. The system is a challenging but interesting field of research. (Langer, Vučić, Walewski, p. 2 ff.)
Due to this result, the further chapters will only refer to the VLC technology.
The VLC technology is already in use in several countries. There are already projects in Germany, France and the Netherlands, among others. The Fraunhofer Institute and Philips in particular are working on interesting projects. A few projects are listed below:
- Completed project "VLC Mainau":
On May 20, 2015, Europe's first practical project with optical data transmission, Visible Light Communication (VLC), started on the island of Mainau in Lake Constance. The aim of the project was to equip a conference room on Mainau Island with VLC technology. This can replace the existing radio-based WLAN solution. The Fraunhofer Institute successfully completed this project. The upgrade of the conference room was carried out in several steps and was intended to provide insights for practice-oriented parameter optimization. This VLC installation is the first in Germany that was developed outside a laboratory environment.
The communication system was installed in the first project phase. In the second project phase, the technology was expanded. Due to that several users have flexible access to the Internet.
The project is important because it is intended to demonstrate that the alternative - optical instead of radio-based data transmission - not only exists as a research idea, but can also function in real environments.
- Current project "VLC Classroom":
Since the beginning of 2017, Fraunhofer HHI has been working together with the Hegel-Gymnasium in Stuttgart on equipping a classroom with Visible Light Communication technology. In Okober 2017 the room was inaugurated with an optical WLAN environment of. The project will be completed by 31th of March 2019.
VLC Classroom in the Hegel-Gymnasium Stuttgart
- All current projects of the Fraunhofer Institute:
Next to the above introduced projects, the Frauhofer Institut is currently working on many other projects.
- Carrefour France
The french retail giant Carrefour installed a Philips networked indoor lighting system with LED-based positioning when the hypermarket in the Euralille shopping centre in Lille was renovated in 2015. It is one of the top ten shopping centres in Europe. The system enables customers to find special offers in the store using a dedicated smartphone app.
The LED-based system is perfectly appropriate for Carrefour because its energy efficiency supports the company's sustainability efforts. The company has announced that it will reduce total energy consumption by 30% and CO2 emissions by 40% by 2020. The Philips system helps to achieve this goal by reducing energy consumption by 50% compared to the previous system.
- Media Markt Netherlands
The Media Markt flagship store in the center of Eindhoven will introduce the "Store Guide" app from Philips Lighting, the world market leader for lighting. This offers customers of the flagship store the opportunity to navigate through the sales areas via smartphone in order to easily reach the desired products without detours. It is based on an innovative LED lighting system from Philips Lighting, which uses coded light and an app to serve as an indoor navigation system (indoor positioning).
The Store Guide app uses the high-precision Philips indoor navigation system and interior mapping from Aisle411, a development partner of Philips Lighting for digital retail space architecture. Data protection is guaranteed because no personal data is collected. The app runs on iOS and Android. The redesigned Media Markt store in Eindhoven was reopened on 26th of April 2017. The Media Markt Store Guide app can be downloaded from the Apple App Store and Google Play Store and used at Media Markt Eindhoven Centrum and Arnhem.
Philips Lighting is a world leader in indoor navigation architecture and its VLC technology helps customers to locate products with an accuracy of up to 30 centimeters.
A comparative analysis and study on Li-Fi and Wi-Fi Technology from the King Faisal University in the Kingdom of Saudi Arabia came to the conclusion, that the Li-Fi technology (VLC) is now developing into a ubiquitous system technology with innovative networking capabilities for universal application to provide a variety of device platforms for high-speed internet communications. Although Li-Fi technology does not use any radio frequencies, but it provides safer, greener and cheaper technology. This Li-Fi technology provides users better security measures, capacity and availability in comparison to Wi-Fi. The future implications of Li-Fi include its application in different fields such as industries, medicine and education. (Zahrani, p. 1)
Another study on Li-Fi Technology from the Saranathan College of Engineering in India found out, that Li-Fi will make our lives more technology driven in the near future. Every bulb can be used like a Wi-Fi hotspot to transmit wireless data (Sanam, Srikanth, Sathis, p. 4388). The main difference between Wifi and Lifi is that Wifi uses radio waves in order to transmit data at a slower data rate whereas Lifi uses visible light in order to transmit data at a much faster rate (Sanam, Srikanth, Sathis, p. 4387).
Lifi is a groundbreaking technology which has been introduced recently in wireless communication. Through this technology, the bulbs at homes, offices and streets will not only be able to light and illuminate the environment but will also be able to transmit data wirelessly at high speed. As there is growth in population and their devices access wireless internet, the air waves are becoming increasingly clogged, making it more and more difficult to get a reliable, high-speed signal. So Li-Fi has a bright future (Sanam, Srikanth, Sathis p. 4388).
Criterion | Description |
---|---|
Range | The technology was demonstrated at the 2012 Consumer Electronics Show in Las Vegas using a pair of Casio smart phones to exchange data using light of varying intensity given off from their screens, detectable at a distance of up to 10 metres (Sharma, Raunak, Sanganal, p. 152), (Omanakuttan et al., p. 2341). A single room can be served by multiple optical attocells, with each covering an area of 1–10m² and distances of about 3m (Haas, p. 2). |
Accuracy | The location of the customer's smartphone can usually be determined with an accuracy of up to 30 centimetres (Fan, Li, Luo, p. 5), (Signify; Location-based Services). |
Power consumption |
LED's:
They use at least 75% less energy last 25 times longer than incandescent lighting. Most of the energy of LEDs is used for illumination. Therefore the VLC system is energy efficient. The efficiency of LEDs is 20lm/W greater than the incandescent lamps efficiency (Fan, Li, Luo, p. 3). With the advantage of lower power consumption and longer lifetime, LED’S are expected to be the next generation illumination device. Smartphone: Power consumption of smartphone is higher because of app usage (Fan, Li, Luo, p. 2). |
Life span | 20.000 – 50.000 h (2 - 6 years) (Fan, Li, Luo, p. 1) |
Bit rate | A speed up to 10Gbits/sis obtained using Li-Fi (Saini, p. 13), (Bhut, Parmar, Mehta, p. 27). |
Disturbing factor | Thick fog or smoke can blur visible light links and decrease system performance. Smartphone needs direct intervisibility to LED‘s (Arnon, p. 189 f.). The VLC is vulnerable to interference from other sources such as sunlight and other illumination (Saini, p. 14). |
Embedding (no additional Hardware required) | - Smartphone / App required Using an iOS or Android software development kit and a cloud service, retailers can embed the tracking capabilities in their own mobile applications (Günther). |