In a H2020-project MIREGAS, Technical Research Centre of Finland VTT Ltd and Tampere University of Technology teams, together with European partners, developed novel components for miniaturized gas sensors exploiting the principle of Mid-IR absorption spectroscopy. In particular, the main advances concerned the development of novel superluminescent LEDs for 2.65 µm wavelength, Photonic Integrated Circuits (PICs) with 1nm bandwidth for spectral filtering at Mid-IR, hot-embossed Mid-IR lenses for beam forming and photodetectors for 2 to 3µm wavelengths. Silicon photonics PIC technology that was originally developed for optical communication applications allows for the miniaturization of the sensor. The components entail important benefits in terms of cost, volume production and reliability.

The market impact is expected to be disruptive, since the devices currently on the market are typically complicated, expensive and heavy instruments. The components developed by the consortium enable miniaturized integrated sensors with important benefits in terms of cost, volume production and reliability, which are instrumental features for the wide penetration of gas sensing applications.

The IR spectroscopy is a powerful tool for multigas analysis. Conventional sensors are based on the use of filters, spectrometers or tuneable lasers. MIREGAS project, has introduced breakthrough components, which enable multigas analysis using integrated solution. Moreover, using the new technology, the wavelengths of light can be filtered more precisely, and interfering gas components can be excluded.

H2020 European consortium, MIREGAS, brought together world-leading European institutes and multinational companies. On the technology side, VTT Technical Research Centre of Finland Ltd coordinated the programme and provided Si PICs as well as photonics packaging and integration technologies. The Optoelectronics Research Centre at the Tampere University of Technology in Finland was responsible for developing innovative superluminescent LEDs, ITME (PL) for mouldable Mid-IR lenses and VIGO (PL) for Mid-IR detectors. Industrial partners Vaisala (FI), AirOptic (PL) and GasSecure (NO) brought their competences in the areas of gas sensing and Mid-IR sensor fabrication, and at the same time, validated the technologies developed by the consortium.

"The project will enable us to extend our gas sensing technology into new market areas. The MIREGAS technology offers a unique mix of sensitivity, selectivity and competitive pricing that will be a complement to our present high-performance, laser-based sensing technology. Airoptic plans to a release a first product for hydrocarbon detection based on the technology developed within the MIREGAS project in 2019", says CEO Pawel Kluczynski from Airoptic.

"Bearing in mind the potential market for devices developed in the MIREGAS project, VIGO System SA has started to implement a development plan that assumes investments in increasing technological and production capacities - which will allow for the future, high volume production of sensors based on SLED sources. The main actions of the plan are the purchase of a high-performance epitaxial machine, the expansion of the assembly capabilities and the acquisition of specialists in the field of epitaxial materials for the production of IR sources," stated Chief Sales Officer Przemysław Kalinowski from VIGO.

"Within this project, ITME was able to establish technology for low-cost mid IR optics development using moulding optics approach. We have developed several novel glasses transparent from visible up to Mid-IR range. We establish moulding processing that allow us to develop free-form optical components with optical surface quality for Mid-IR without further polishing. It allows to reduce cost of production first, but also to develop lenses in environment friendly matter since number of glass waste is dramatically reduces. Also grinding and polishing process that requires use of large amount of water and polishing powders is totally redundant. We found also new materials for mould development that are low cost and can be processed with standard CNC machines. This way our technology becomes cost effective also for prototyping and short series of components. This breakthrough give access for custom made free-form glass optics even for SME companies," says Prof. Ryszard Buczynski, head of Department of Glass, ITME. "Now we verify market potential of our technology and offer access to this technology as a custom service through ITME. After market validation, we plan to transfer this technology to other existing companies or establish spin-off company. Some venture funds and world-recognized companies already expresses their interest in our innovations."

"At the start of the project, Mid-IR SLED technology targeting high-brightness and broadband operation was simply not existing. MIREGAS enabled important scientific breakthrough; in fact, the project results represent the state-of-the-art both in terms of power and wavelength coverage. Yet, what it is maybe the most important, contributed to creating a new European ecosystem based on combined expertise in Mid-IR optoelectronics and Si-photonics. We are just at the start of many other applications we will target with this powerful combination of technologies", adds Prof. Mircea Guina, the head of ORC team at Tampere University.

"VTT's Silicon-on-Insulator (SOI) based Photonic Integrated Circuit (PIC) technology owns two unique features: Firstly, in addition to the conventional optical communications wavelengths at 1550 nm, it is applicable for Mid-IR wavelengths; Secondly, it allows for the integration of active devices, such as, laser diodes or photodetectors, directly on the PIC chip. Therefore, the SOI PIC technology is very attractive for gas sensing and sensor integration, in general. In the MIREGAS project, we were able to take this offering to the next level together with the beneficiaries", stated Pentti Karioja, VTT, project coordinator. 

A new sensor fusion technique based on X-ray and 3D imaging promises improvements to the 3D modelling of mineral resources and more efficient sorting of precious metals. VTT Technical Research Centre of Finland Ltd (VTT) is coordinating the sensor development process through an EU project called X-Mine in collaboration with businesses, international research institutions and mining lobbies.

The European Commission has granted EUR 9.3 million to a three-year H2020 project coordinated by VTT called X-Mine, which develops new sensor technologies for mining companies' drill core analyses and for efficient sorting of precious minerals and metals in ore. The X-Mine project also promotes more efficient 3D modelling of mining companies' mineral resources, and thus improved recovery of precious minerals. The project combines products developed by various sensor manufacturers with commercial ore dressing equipment and aims to achieve sensor fusion that allows rock with low levels of minerals to be separated from the ore. The project reached its midway point at the turn of the year and is progressing gradually to the piloting of both drill core analysers and ore dressing equipment at mines during the spring of 2019. Two drill core analysers were adopted at mines in Greece and Sweden towards the end of 2018, and members of the project consortium are currently testing new sensors for ore dressing equipment and designing ore dressing algorithms. The project consortium consists of 15 research partners and businesses from around the 

 Figure 1. a) Project consortium, b) Members of the consortium on a visit to Kęty in Poland to learn about the development of ore dressing equipment at a demonstration event hosted by Comex

The project gives the participating research institutions and businesses an opportunity to develop sensor-based solutions for the mining industry together with experts specialising in the exploitation of ore resources, mineral processing and geological mapping. The consortium's aim is to develop solutions for efficient ore extraction and for reducing the amount of waste generated by the ore extraction process. All in all, the project is expected to reduce the harmful environmental impacts of mining by reducing the need for ore processing and chemical processing in mineral recovery. The project consortium is also looking to lower mining companies' production costs.

The X-Mine project involves developing and piloting two prototypes to meet mining companies' needs. Mining companies extract material from the bedrock in order to analyse the location of the ore and the volume of minerals and to estimate their mineral resources. The project consortium is hoping to increase the efficiency of ore exploration by developing equipment that can be used to scan drill core samples on site using new, highly sensitive layered imaging technology based on X-ray fluorescence as well as composition analyses. Analysing and scanning drill core samples on site speeds up the evaluation of ore resources.

Figure 2. a) A drill core analyser developed by Orexplore b) Prototype of ore dressing equipment to be built in Comex's research environment

The project also expects to improve the operation of automated mineral selectivity systems at the extraction stage by establishing a sensor fusion technique that combines X-ray transmission scanning along a line of highly sensitive sensors developed in the course of the project, X-ray fluorescence technology and 3D vision technology with rapid analyses with the help of efficient algorithms. More efficient ore dressing increases resource efficiency and mining companies' profitability while reducing the harmful environmental impacts of mining.

The project was granted funding through the Horizon2020 instrument – the EU's research and development programme that has EUR 80 billion to award to European research initiatives over a seven-year period (between 2014 and 2020). The EU's H2020 funding programme promises more breakthroughs, discoveries and world-firsts by taking great ideas from the lab to the market. The total budget for the three-year project is EUR 11.9 million, of which EUR 9.3 million comes from the European Union.

The X-Mine project is based on international cooperation between research institutions from Finland (VTT), Sweden (Uppsala University, Geological Survey of Sweden) and Romania (Geological Institute of Romania) as well as sensor and equipment manufacturers from Finland (Advacam Oy), Poland (Antmicro Sp. z o. o. and Comex Polska Sp. z o. o), the Czech Republic (Advacam s.r.o.) and Sweden (Orexplore AB). End users involved in the project include mining companies in Bulgaria (Assarel Medet Jsc.), Greece (Hellas Gold S.A.), Cyprus (Hellenic Copper Mines Ltd) and Sweden (Lovisagruvan AB) as well as mining lobbies in Sweden (Bergskraft) and Australia (Swick Mining Services Ltd).

Sensirion is a manufacturer of environmental as well as gas and liquid flow sensors. The cost-effective LD20 single-use liquid flow sensor, which has been used for this concept study, is compact and measures flow rates in the micro and milliliter per hour range with outstanding precision and reliability thanks to the patented CMOSens Technology. Additionally, it features sensitive failure detection mechanisms to help counteract, for example, occlusions and air-in-line. Quantex is an innovative leader in single-use disposable pump technology. The pump’s design is based on a rotary, fixed displacement principle thus is much less sensitive to variables such as line pressure, fluid viscosity and flow rate.

The new wearable drug delivery IoT platform “Quantex 4C” demonstrates an innovative approach to infusion therapies. It integrates connectivity with a modular drug delivery platform and enables continuous monitoring of the therapy with the help of a single use liquid flow sensor for safe ambulatory treatments. “Real time bidirectional flow verification at the low flow rates that are typical for the targeted application and the sensor’s compact form factor made the Sensirion LD20 single-use liquid flow sensor the ideal choice”, said Paul Pankhurst, Founder and CEO of Quantex. The objective of the presented study is to demonstrate the possibilities of a comprehensive drug delivery system to the medical technology industry. The compactness, low power consumption and cost-effectiveness of both, sensor and pump, allow the design of a wearable device which controls and monitors the drug delivery therapy simultaneously. Such a connected drug delivery platform opens up entirely new possibilities for all involved stakeholders. While patients and clinicians benefit from an increased ease of use and confidence in the therapy, low maintenance and less training efforts; it allows healthcare and pharma companies to capture vital adherence data and simplifies patient compliance.

Reliable failure detection is a central requirement for next generation drug delivery devices. Ambulatory treatments and wearable applications will only be successful if patients as well as clinicians trust the reliability of those next generation devices.

Redsense Medical announces that the company has started the development of a prototype with fully functional optical measuring based on the innovative, smart wound technology that was presented in November 2018. The prototype is expected to be ready in Q2 2019.

The company's smart wound care technology makes it possible to develop thin sensor layers for optical measuring of several physiological and biological parameters such as blood and exudate. The layer of sensors can be used separately or integrated directly into smart bandages or adhesive plasters. As the technology enables very cost and resource effective individual wound care, it has potential to become as revolutionary for wound care.

"We have now decided to go from TRL 2 to TRL 3 on NASA's readiness level indicator for development projects, and it will be very exciting to present a working prototype before the start of the summer," says Redsense Medical's CEO Patrik Byhmer.

As announced earlier, Redsense has already initiated discussions on potential collaborations or out-licensing of the technology with large, global wound care companies. The discussions are expected to intensify and become more concrete when a working prototype can be presented and evaluated.

Redsense will present ongoing project updates during the coming months as substantial progress is expected. Based on this outlook, the smart wound care technology has the potential to build substantial value for Redsense Medical´s shareholders.

Strong outlook for the wound care market

The wound care market was valued at approximately USD 18 billion in 2016, and it is expected to grow at a CAGR of 5.3 % until 2023. The primary drivers of market growth are an ageing global population, a larger number of diabetics and increasing investments in research and development, which is also expected to lead to new research discoveries in this area. Chronic wounds are expected to constitute most of the wound care market up until 2022.

The Internet of Things (IoT) is creating many new exciting application opportunities to create smart environments where sensors monitor for changes so that the appropriate actions can be taken. The fastest growing examples of this are HVAC (Heating Ventilation and Air Conditioning), IAQ (Indoor Air Quality), smart homes and smart offices where a network of sensors monitors temperature and carbon dioxide (CO2) levels to ensure the optimal conditionals are maintained with the minimum of energy expenditure. A challenge for such systems in that the CO2 sensors need mains power to operate incurring costs for cabling and, in the case of installing in existing buildings, redecoration.

Gas Sensing Solutions (GSS) has solved this problem with its low power, LED-based sensor technology. The sensor’s power requirements are so low that wireless monitors can be built that measure CO2 levels as well as temperature and humidity with a battery life of over ten years. Being wireless means that they can be placed wherever they are required with no need for cabling or disruption and simply relocated as building usages changes

CO2 Detector with Lime

To make the design of these monitors even easier, GSS has added an I2C interface to its very low power CO2 sensor, the CozIR-LP. Having the widely used I2C interface now makes the integration of the sensor into a design very easy. The CozIR-LP is the lowest power CO2 sensor available requiring only 3mW that is up to 50 times lower than typical NDIR CO2 sensors. The GSS patented LED technology also means that the solid state sensor is very robust.  This keeps maintenance costs to a minimum as the expected lifetime is greater than 15 years making them the perfect choice for fit and forget applications that measure low (ambient) levels of CO2 from 0-1%.

“Although HVAC and IAQ are major application areas,” explained Calum MacGregor, CEO at GSS, “the lightweight, miniature size of the CozIR-LP also opens up other new possibilities for CO2 monitoring such as portable and wearable devices. The power requirements are so low that energy harvesting designs, such as solar, are now easily achievable. Here again, the new feature of an I2C interface will simplify the design process of integrating the sensor with other sensors and devices all on the I2C bus.”

Samsung introduces its smallest high-resolution image sensor, the ISOCELL Slim 3T2, said to be the industry’s most compact 20MP image sensor at 1/3.4-inches. The 0.8μm-pixel ISOCELL Slim 3T2 is aimed to both front and back cameras in mid-range smartphones. The 1/3.4-inch 3T2 fits into a tiny module making more space in ‘hole-in or notch display’ designs.

“The ISOCELL Slim 3T2 is our smallest and most versatile 20Mp image sensor that helps mobile device manufacturers bring differentiated consumer value not only in camera performance but also in features including hardware design,” said Jinhyun Kwon, VP of System LSI sensor marketing at Samsung Electronics. “As the demand for advanced imaging capabilities in mobile devices continue to grow, we will keep pushing the limits in image sensor technologies for richer user experiences.”

When applied in rear-facing multi-camera settings for telephoto solutions, the 3T2 adopts an RGB color filter array instead of Tetracell CFA. The small size of the image sensor also reduces the height of the tele-camera module by around 7% when compared to Samsung’s 1/3-inch 20MP sensor. Compared to a 13MP sensor with the same module height, the 20Mp 3T2 retains 60% higher effective resolution at 10x digital zoom.

The Samsung ISOCELL Slim 3T2 is expected to be in mass production in Q1 2019.

The incident at London Gatwick airport in the UK caused major travel disruption for more multiple days after drones were spotted flying over this sensitive area. This incident highlighted the need for anti-drone technologies to address this evolving threat and secure the safety of flight. Following the episode, the US Federal Aviation Administration was instructed to develop a strategy to allow wider use of counterdrone technologies across airports. Detecting drones, and any UAV threat is a real challenge for many reasons. HGH Infrared Systems with its family of renowned SPYNEL thermal sensors offers a unique set of solutions to address this evolving threat and ensure true, real-time airport security.

In these times of heightened UAV threats, the SPYNEL IR imaging camera provides an innovative solution which guarantees the ability to detect, track and classify any types of drones.

Whereas the drone technology is constantly evolving, bringing on the market many different types of drones including fixed wing, multi rotor drones, drones with GPS, autopilot and camera, autonomous drones emitting low or no electromagnetic signature, the SPYNEL thermal imaging technology, makes it impossible for a UAV to go unnoticed: any object, hot or cold will be detected by the 360° thermal sensor, day and night. Driven by the CYCLOPE intrusion detection software, the panoramic thermal imaging system tracks an unlimited number of targets to ensure that no event is missed over a long-range, wide area surrounding. SPYNEL is thus fully adapted to multi-target airborne threats like UAV swarming.  SPYNEL is a versatile, multi-function sensor with a large field of view enabling real-time surveillance of both airborne and terrestrial threats at the same time.

The CYCLOPE automatic detection software provides advanced features to monitor and analyse the 360° high resolution images captured by SPYNEL sensors. The ADS-B plugin enables aerial target identification and the aircraft ADS-B data can be fused with thermal tracks to differentiate an airplane from a drone. With the forensics analysis offering a timeline, sequence storage and playback possibilities, it is also possible to go back in time to analyse the behaviour of the threat since its first apparition on the CYCLOPE interface. Moreover, the latest CYCLOPE feature makes 3D passive detection by triangulation available when using several SPYNEL sensors at the same time. The feature consists in analyzing the distance and the altitude of multiple targets, creating a kind of "protective bubble" around the airport.

Edouard Campana, Sales Director at HGH Infrared Systems, said: "Spynel 360° panoramic thermal camera and its Cyclope software are frequently used against drones to ensure the security of national and international events, critical infrastructures, airport and more. The real-time visualization and detection of multiple targets makes it a unique sensor for ultimate situational awareness. This solution is rapidly deployable and offers HD playback capabilities, very useful for events clarification.”

A key advantage of the SPYNEL detection system for airport applications is that it is a fully passive technology, meaning it will not be a source of disturbance in the electromagnetic environment of the airport, unlike radars. Indeed, a concern often raised by air-safety regulators is that anti-drone systems designed to jam radio communications could interfere with legitimate airport equipment.

Part of the complete surveillance equipment of an airport, the SPYNEL thermal imaging sensor is the must have security equipment for such a high-risk infrastructure, operating with complementary detection sensors. Military facilities, correctional institutions, stadiums and other critical infrastructures have already chosen to integrate the SPYNEL sensor with their other security and facility systems, such as radars, PTZ cameras, Video Management System and more. SPYNEL can also be rapidly deployed as a standalone solution for temporary surveillance, to face urgent cases. With its 24/7 and panoramic area surveillance capabilities, the SPYNEL thermal camera provides an early warning and an opportunity for rapid and accurate detection over large areas, to support proactive decisions.

Single points used for realtime retail analysis on the spot

EdgeNPD offers innovative SaaS (Software as a Service) solutions with state-of-the-art tools to help retailers and manufacturers increase sales and ROI from retail space with minimum cost and risk. theStore2 Retail Lab software is a VR enabled 3d application for optimising planograms, POSM and space management, allowing for prototyping and testing future trade, marketing or in-store communication strategies. The solution facilitates faster and better decisions driven by advanced data analytics. The full-stack model allows clients to outsource the entire process necessary to manage the category efficiently - from market research, through analytics and decision making, with implementation and training of sales teams.

Edge NPD’s underStand is an Internet of Things hardware and software solution for both retailers and manufacturers. It is a weight sensor-based, cloud connected device which collects and processes data from POS in real time. UnderStand reports and analyses rotation and out of stock data from your promotional activities allowing to separate regular and promotional sales and gain full control over your secondary placement. UnderStand can be used for pre-testing future- or monitoring current promotions allowing to optimize replenishments and sales rep visits.

This goal of Edge NPD solutions matches very well with Zemic’s vision. As we strongly believe that our focus on creating value for our customers will help them to differentiate themselves in their market. Zemic Europe's slogan is therefore "We believe we make you stronger!".

How Zemic consulted EDGE NPD

For the concept "Edge IoT underStand" which EDGE NPD developed, monitoring sales is the most important feature. Not only to detect when a product is bought by a consumer; but more importantly to monitor the product & turnover flow in order to optimize the retail store. All monitoring features of the "Edge IoT underStand" concept are meant to create the most optimum and efficient retail store. This is how space is converted into sales.

To have an accurate and reliable result EDGE NPD searched for a reliable partner which could supply a large number of single point loadcells in a relatively short time. Zemic Europe offered, with their european warehouse with 40.000 products in stock, the right combination of fast delivery together with the right quality/price/service ratio. This enabled EDGE NPD to act quickly and professionally, assisting their customers' needs within the fast moving consumer goods branch.

About underStand from EDGE NPD

EDGE NPD is the laureate of the 2017 edition of the New Europe 100 list. This is a listing of outstanding innovators in Central and Eastern Europe.

Edge underStand has the following benefits for retailers :

1. underStand requires no involvement from the store – it can be installed on a variety of stands and can provide both stand-based or shelf-based monitoring

2. underStand makes it easy to organize and monitor promotions in real time so you can estimate future profits instantly

3. the unique architecture of underStand allows you to monitor parameters such as the effectiveness of shelf product distribution

4. underStand is integrated in all Edge systems: theStore, Geo, ProPOSe, VideoAnalysis, and Trade Planner

Zemic Europe L6E single point loadcell for retail concept.

Single Point Load Cells are often installed in most small to medium sized platform scales. The L6E aluminium load cell is used for the "underStand" concept of Edge NPD. This is an aluminium alloy IP65 single point loadcell suitable for platforms up to 400 x 400 mm. The L6E family is available in OIML C3 , C4 and C5 accuracies in capacities from 50kg up to 300kg.

For more information: 

Artificial Intelligence (AI) is not a new topic at all. In fact, the idea and the name AI, already appeared in the 1950s. People started to develop computer programs to play simple games like Checkers. One milestone known to many people was the launch of the computer program ELIZA, built in 1966 by Joseph Weizenbaum. The program was able to run a dialog in written English. It created questions, the user provided an answer and ELIZA continued with another question related to the response provided by the user.

Figure 1: Example screenshot of the ELIZA terminal screen

The main AI technique are neural networks. Neural networks have first been used in 1957 when Frank Rosenblatt invented the perceptron. Today’s neurons in neural networks are still using a very similar principle. However, the capabilities of single neurons were rather limited. It took until the early 70s before scientist realized that multiple layers of such perceptrons could overcome these limitations. The final breakthrough on multi-layer perceptron was the application of the backpropagation algorithm to learn the weights for multi-layer networks. An article in Nature in 1986 by Rumelhart et. al.[Rum] made backpropagation mark the breakthrough of neural networks. From this moment, many scientists and engineers were drawn into the neural network hype.

In the 1990 and early 2000, the method was applied to almost any kind of problem. The number of research publications around AI, and particularly neural networks, significantly increased. Nevertheless, once all the magic behind neural networks has been understood, it became just one of many other classification techniques. Due to their very demanding training efforts, neural networks in the second half of the 2000 decade faced significantly reduced interest.

Reinvestigating neural networks with respect to their operating principles caused the second hype, currently still ongoing. By having more computational power at hand and the involvement of a large number of people, Google demonstrated to beat the best Go players with a trained neural network.

Types of AI's

Over the last decades, different AI techniques have emerged. In fact, it is not black and white whether a certain technique belongs to AI. Many simple classification techniques like principle components analysis (PCA) also use training data. They are not classified as AI however. Four very prominent techniques are outlined in the subsequent sections. Each of them has many variants and the given overview in this article does not claim to be complete.

Fuzzy Logic

Fuzzy Logic extends classic logic based on false/true or 0/1 by introducing states that are in between true and false, like “a little bit” or “mostly”. Based on such fuzzy attributes it is possible to define rules for problems. For example, one of the rules to control a heater could be the following: “If it is a little bit cold, increase the water temperature a little bit”. Such rules seem to express the way humans think very well. That is why Fuzzy Logic often is considered an AI technique.

In the real application, large sets of such fuzzy rules are applied to control problems for instance. Fuzzy Logic provides algorithms for all the classical operators like AND, OR and NOT to work on fuzzy attributes. With such algorithms, it is possible to infer a decision out of a set of fuzzy rules.

A set of fuzzy rules has the advantage of being easily read, interpreted and maintained by humans.

Figure 2: Example for fuzzy states of a heater control. The arrows denote the state values at the indicated temperature

Figure 2 illustrates the fuzzy states of a heater control. The states are “Cold”, “Warm”, and “Hot”. As can be seen in the figure, the three states do have some overlap and some temperatures belong to two states at the same time. In fact, each temperature belongs to a certain state with a defined probability.

Genetic Algorithm

Genetic Algorithms apply the basic principles of biological evolution to optimization problems in engineering. The principles of combination, mutation and selection are applied to find an optimum set of parameters for a high dimensional problem.

Assuming a large (more than 20) set of parameters with a given fitness function, it is mathematically not possible to determine an optimum set of parameters to maximize the fitness function.

Genetic Algorithms tackle this problem in the following way. First, a population of random parameter sets is generated. For each set in this population, the fitness function is calculated. Then the next generation is derived from the previous generation by applying the following principles:

• Two selected sets of the previous populations (parents) combined form all sets of the next generation (children). The selection is random but sets with a higher score on the fitness function have a higher probability of being chosen as parents
• Based on a mutation rate, random parameters of random sets of the new generation are modified by a small percentage

Each set of the next generation is then evaluated using the fitness functions. In case one set appears to be good enough, the genetic algorithm stops, otherwise a new generation is created as described above.

Figure 3: Description of generating algorithm cycle

It has been shown that for many high-dimensional optimization problems a Genetic Algorithm is able to find a global optimum, whereas conventional optimization algorithms fail, because they were stuck in a local optimum.

Genetic Programming

Genetic Programming takes Genetic Algorithms a step further by applying the same principles to actual source code of programs. The sets are replaced by sequences of program code and the fitness function is the result of executing the actual code.

Very often, the generated program code does not execute at all. It has been demonstrated however that such a procedure can indeed generate working source code for problems like finding an exit in a maze.

Neural Networks

Neural networks mimic the behavior of human brains by implementing neurons. They take input from many other neurons, and then perform a weighted sum. Finally, the output is limited to a defined range. The impact of a specific input depends on the weight associated to this input. These weights resemble the functions of synapses in the human brain to a certain extent.

The weights of the connections are determined by applying inputs and desired outputs. This again is very similar to the way humans teach their kids how to determine the difference between a dog and a cat.

Figure 4: Example of a neural network architecture

The main components of a neural network architecture are input nodes where the input data is applied. The second set of components are hidden layers who process the inputs through application of weights to their inputs. The weighted inputs are then transferred to the inputs of the next layer. Finally, the outputs assign a certain weight to the classification of the input set as a result.

Local Intelligence

IoT sensor solutions today are mostly only responsible for data acquisition. The raw-data needs to be extracted from the sensor and transmitted to another, more computationally capable device within the network. Depending on the use-case, this device could be either an embedded system, or a server within the cloud. The receiving end collects the raw-data and performs pre-processing in order to present relevant results. Frequently the raw-data of the IoT device needs to be processed using artificial intelligence, as in speech recognition for example. The amount of IoT devices and especially the demand for artificial intelligence is expected to increase dramatically over the next years since sensor solutions become more complex.

However, the growing amount of connected IoT devices, which rely on cloud solutions to compute meaningful results, leads to various problems in several areas. The first issue is the latency between acquiring the raw-data and the response with the evaluated information. It is not possible to build a real time system, since the data needs to be sent over the network, processed on a server and then again interpreted by the local device. This leads to the second problem, which is the increasing network traffic and therefore is reducing reliability of network connections. Servers need to handle more and more requests of IoT devices and thus could be overwhelmed in the future.

A major advantage of neural networks is their ability to extract and store the essential knowledge of a large set of data in a fixed, typically much smaller set of weights. The amount of data, which is used to train a neural network, can be vast. In particular, for high dimensional problems the data set needs to scale exponentially to maintain a certain case coverage. The training algorithm extracts the features out of the data, which efficiently will classify unseen input data. As the number of weights is fixed, the amount of storage does not correlate to the size of the training data set. For sure, if the network is too small it will not deliver good accuracy, but once a proper size has been found the amount of training data does not affect the size of the network anymore. Nor does it affect the execution speed of the network. This is another reason why in IoT applications a local network can outperform a cloud solution. The cloud solution may of course store vast amounts of reference data, but then the response time of the cloud degrades quickly with the number of reference data stored in the cloud.

By definition, IoT nodes are connected to a network, and very likely to the Internet. However, it can be very desirable to have a local intelligence. Then, processing of raw data can happen on the sensor or in the IoT node instead of requiring communication with the network. The most important reason for such a strategy is the reduction of energy consumption of network communication traffic.

Major companies as embedded microprocessor manufacturers already realized, that cloud-based services have to be adopted. One of the consequences is the introduction of new embedded microprocessor cores capable of machine learning tasks. In the future, the trend of processing data within the cloud will be further shifted back to local on-device processing. This allows more complex sensor solutions, which involve sensor fusion or pattern recognition. For these applications, local intelligence of the IoT device is needed. Sensor solutions will become truly smart, as they already deliver finalized meaningful data.

Figure 5 represents this paradigm shift from cloud-based solutions, to local intelligence.

However, computing elaborate AI solutions within an IoT device, requires new solutions which meet power, speed and size constraints. In order to archive this, the trend is shifting to integrated circuits optimized for machine learning. This type of processing is commonly referred to as edge AI.

Constraints

In sensors for IoT applications, which are very frequently mobile or at least service free, the most prominent constraint is power consumption.

This leads to a system design, which minimizes the amount of data to be transferred via a communication channel as sending and receiving data. In particular, in wireless mode, transmission is always very expensive in terms of power budget. Thus, the goal is to process all raw data locally and only transmit meaningful data to the network.

Local processing neural networks are a great option as their power consumption can be well controlled. First, the right architecture (recurrent versus non-recurrent) and the right topology (number of layers and neurons per layer) must be chosen. This is far from trivial and requires experience in the field. Second, the bit-resolution of the weights get important. Whether a standard float type is used or whether someone can find an optimized solution using just 4 bits per weight contributes significantly to memory size and therefore to power consumption.

Gas Sensor Physics

The sensor system, used as a test case for AI in sensors, is a Metal-oxide (MOX) based gas sensor. The sensor works based on the principle of a chemiresistor[Kor]. Under a certain possible set of reducing (e.g. CO, H2, CH4) and/or oxidizing (e.g. O3, NOx, Cl2) gases the detector layer is changing its resistivity. This can in turn be detected via a metal electrode sitting underneath the detector layer. The main problem of such a configuration is the indiscriminate response to all sorts of gases. Therefore, the sensor is thermally cycled (through a microhotplate). This causes the sensor to react with a resistance change with a unique signature and thus significantly increases the selectivity of gas detection.

Figure 6: Structural concept of a chemiresistive gas sensor

Another approach is to combine different MOX sensor layers to discriminate further between the different gas types.

A closed physical model explaining the behavior of chemiresistors would depend on too many parameters. A non-exhaustive list of these parameters are thickness, grain size, porosity, grain faceting, agglomeration, film texture, surface geometry, sensor geometry, surface disordering, bulk stoichiometry, grain network, active surface area, size of necks of the sensor layer. Together with the thermal cycling profile, the model would be too complex and is currently simply not available.

Therefore, such systems form an ideal case to apply modern AI methods.

Supervised Learning

Gas sensing is an especially potent application for AI. The problem, which needs to be solved, is the prediction of the concentration of gasses with the resistance of multiple MOX pastes as the inputs.

To solve the task, the behavior of the MOX pastes when exposed to various gasses with different concentrations has been recorded. From this data, a dataset consisting of features (the temporal resistance trend of each paste) and labels (the gas that was present) has been created.

This kind of data is especially well suited for the supervised learning method. In supervised learning, the neural network is given many samples, each consisting of features and a label. The network then learns to associate features with labels in an iterative learning process. It is exposed to every sample multiple times. Its prediction is nudged in the direction of the correct label every iteration by adjusting its weights.

Architecture and Solution Approach

A neural network is defined by its architecture. The architecture is usually influenced by the dataset at hand. In our case, the dataset has a temporal structure, so a recurrent neural network is a good fit. Recurrent neural networks process the features in multiple steps and keep information about previous steps in an internal state.

The architecture also has to be adapted to the already mentioned IoT constraints. The neural network should be as small as possible to minimize power consumption. We use one hidden layer with 47 neurons. The weights are quantized to 4 bits, which further reduce power consumption. On top of this, the network is implemented in an analog circuitry to make it even more efficient.

The network was first tested in a pure software environment using TensorFlow (https://www.tensorflow.org/). This allowed rapid adjustment of the architecture to make sure it is able to solve the task properly before actually building it.

Results

It is not trivial to evaluate a machine learning classifier. There are multiple ways to measure performance. One of the most popular ones is the Receiver Operating Characteristic (ROC). The ROC is a line with false positive rate on one axis and true positive rate on the other. The area under the curve (AUC) of this line should be as high as possible. It measures how well the classifier can separate the positive and negative samples.

Figure 7: Correlation of false positives to good positives for the example of ROC classified gas data

Another interesting metric is the mean absolute error (MAE). The MAE averages the absolute error of the prediction over all samples. We have been looking at this metric over time to get a sense of how many time steps the network needs to achieve a good prediction.

Figure 8: Development of MAE metric over time of a neural network trained on the gas sensor data

Systems where AI could be relevant

This section gives an overview of possible future sensor solutions. In general it can be assumed, that reading raw-data from sensors will not be sufficient anymore, since the complexity and capability of sensors themselves is increasing.

The raw-data needs to be heavily computed and/or combined with the raw data of other sensors to deliver a meaningful result. This method is called sensor fusion. Often, all sensors required for the task, are already included in one solution including an embedded system. Using AI, sensor fusion can be done much easier and more accurately than with classical algorithms. Neural networks can cope with unknown situations much better. In addition, it can detect compensation techniques given the training data and potentially increases the value of the delivered result to the customer.

For our example, classical algorithms would require physics based models, which are not available.

Another example, where the usage of AI is the state-of-the-art solution is object recognition within a camera image. This area has huge potential within the sensor industry.

It allows for small wirelessly connected IoT devices, which are reporting detected objects immediately, anywhere throughout the building without the need of cloud processing.

This could spawn sophisticated surveillance solutions, presence detection or even enable things like smart fridges, which recognize the type of grocery inserted into it.

In addition, facial recognition in embedded devices and smartphones are a trending topic. Here, the facial recognition could be directly performed within the IoT device, creating a plug-and-play solution for customers.

Another area are sensor hubs, which are IoT devices collecting information of other sensors and cameras within the network. Using AI, these devices are capable of identifying patterns, routines and make pro-active steps in support of the customer.

This allows new applications within home automation, which are currently not possible.

Privacy and Data Protection

Typical IoT devices provide no protection against reverse engineering. Therefore, any data, which is stored as a standard firmware on a micro controller in a sensor node, must be regarded as public. Very often, the algorithm to infer meaningful information from sensor raw data is an important asset of sensor providers.

Neural networks do not store the data, which has been used for training. Instead, they extract the relevant features of the data and store these features in the set of weights. It is just impossible to reverse this data extraction and storage operation. This means even given all the weights of a network it is not impossible to derive even parts of the original data, which has been used for training.

This is an important property of neural networks. In particular, in the area of biometry where training data is always a large set of very personal data like face images or fingerprints. Neural network based systems can use such data without any problems of privacy and data protection after the training phase has been finalized.

Summary & Conclusions

This article has shown the increasing importance of AI within sensors in IoT systems. The history of AI and its types was introduced, which lead to the use cases within sensor solutions. It was shown, why AI has a strong potential for sensor solutions in the future and it was demonstrated using the example of gas sensing applications. A prototype implementation for this example including the results was presented. In the end, possible future applications were discussed.

The importance of privacy though protection of the data is very important and cannot be ignored in future systems. Shifting the intelligence back to the IoT device instead in the cloud allows for better selection of data, which need to be processed by third parties.

It is now well accepted that simulation will have to form part of the validation process for any autonomous vehicle.  This is based on the fact that it is simply not physically possible or practical to conduct all the required tests in the real world due to the sheer number of scenarios that need to be considered.

The RAND Corporation published a report in 2016 that looked into the problem of how many miles an autonomous vehicle would have to be driven to prove it was safer than human operated vehicles.  They produced the figure of 5 billion miles as the distance an autonomous vehicle would need to be driven in the real world to have 95% confidence that it was significantly safer than a human driver.  “To drive this distance is not feasible especially when you consider that this would have to be repeated every time a line of code or piece of hardware is changed,” says Mike Dempsey, Managing Director of Claytex.

The generally accepted approach, and the route that Claytex advocates, is the need to have a mix of field testing, proving ground testing and simulation to build up the validation process for an autonomous vehicle.  The field trials allow the vehicle to be tested in the real world with all the random and unpredictable features and behaviours that exist on the roads today.  “Proving ground testing allows us to control many of the scenarios and situations we expose the vehicle to so that we can start to objectively measure its behaviour.  However, in both field tests and proving ground tests, there is little chance that you can exactly repeat a scenario and you have no control of environmental factors such as temperature, rain, and more,” Dempsey explains.

“Therefore, it is necessary to use virtual testing in combination with physical tests to allow the rapid and repeatable testing of the autonomous vehicle throughout the development and validation phases.  However, in order to rely on virtual testing, it is important to be able to confirm that the simulation produces the same results as the real tests”.

“He continues: “The simulation tools used need to be able to accurately recreate real-world locations so that we can build exactly the same scenarios we experience in the field in simulation.  This means the simulation tool needs to be able to include traffic, pedestrians, cyclists and many other movable objects that might feature in the recorded scenarios.  We also need our simulation tool to support running in real-time with the unmodified vehicle control system so that we are testing the same controller that runs in the vehicle”.  

Driving in Montreal with complex scenarios including traffic, pedestrians and cyclists

“This is the challenge that we are working on at Claytex.  We want to ensure that virtual testing is truly representative, and that the AV will respond the same on the road as it did in simulation. Just as a driving simulator must immerse the driver in a convincing virtual reality, the sensor models used to test an AV must accurately reproduce the signals communicated by real sensors in real situations,” Dempsey comments.

Claytex is an established distributor and systems integrator of the 3D driving simulation software rFpro. The software provides high quality graphics and accurate 3D models of real-world locations.

Using rFpro as the core simulation environment enables Claytex to build full autonomous vehicle simulators that allow the vehicle control system to be immersed into the virtual test environment.  rFpro renders images using physical modelling techniques, the laws of physics, rather than the computationally efficient special effects developed for the gaming and film industries. This means that rFpro images don't just convince human viewers, they are also suitable for use with machine vision systems that must be fed sensor data that correlates closely with the real-world.

New sensor models help autonomous cars to ‘see’ the road ahead more clearly

Dempsey explains that one of the key challenges of building a virtual testing environment for autonomous vehicles is the need to replace the sensors that the vehicle control system relies on with sensor models.  “These sensor models need to generate the same output format messages as the real device and they must replicate how the real device perceives the environment.  For a camera, this is relatively easy because rFpro already generates very high-quality images and it is capable of doing that at resolutions and frame rates that exceed those being used in autonomous vehicles today.” 

He argues that it is the LiDAR, radar and numerous other sensors that need the most work.  “rFpro has developed solutions for a number of the technical limitations that have constrained sensor modelling until now, including new approaches to rendering, beam divergence, sensor motion and camera lens distortion.”

“In order to create a representative model of a LiDAR sensor a sensor model is needed that captures the behaviour of the real device,” adds Dempsey.  “So, if the real device is a scanning LiDAR and spins at 1200 rpm, then the sensor model needs to do the same and whilst it’s spinning it needs to move through the virtual world.  The model needs to measure the distance and intensity of the reflection to each point in the environment and pack this into the same UDP message format used by the real sensor”.  This is all possible using rFpro and the VeSyMA – Autonomous SDK developed by Claytex.

The VeSyMA – Autonomous SDK (Sensor Development Kit) has been created to make the creation of device specific sensor models easy by encapsulating a lot of the common features needed by sensor models.  Using the SDK, Claytex have created a suite of generic sensors (radar, LiDAR, camera, ultrasound and GPS) that can be used in the early stages of an autonomous vehicle development project to assess what kind of sensing is desirable. 

In addition, Claytex are building a library of device specific sensor models that support testing throughout the development process.  The first of these device specific models are already being used to support the virtual testing of an autonomous vehicle and the range of sensors available will be continuously added to.

Driving in Paris with a LiDAR sensor model hosted in rFpro providing data to sensor suppliers own data visualisation tool

With sensor models representative of the real devices in place, the focus then shifts to the virtual world and the definition of the test scenarios.  rFpro are able to provide a wide range of high-fidelity models of public roads, proving grounds and race tracks.  These models are all built from detailed surveys of the real locations and cover locations such as Paris, rural Warwickshire, Connecticut, Shanghai and many other world locations.

“Support for traffic, pedestrians and cyclists as well as other movable objects in the virtual world enable us to create a wide variety of test scenarios that we can then repeat under different weather and lighting conditions.  These capabilities allow us to start building complex scenarios that we can use to test the vehicle control system and measure its performance”, concludes Mike Dempsey, Managing Director of Claytex.

The drive towards autonomy is continuing to push the development of virtual test environments into new and challenging areas.  The challenge of producing physics based sensor models that can run in real-time is one being tackled by Claytex with the first full vehicle simulators now in operation.

The increasing variety of applications for the Internet of Things (IoT) and Industrial Internet of Things (IIoT) and their growing adoption, are driving predictions that the sales volume for sensors will reach 75 billion units by 2025. It would be fair to assume sensors will eventually become commodity components. As machine learning grows in importance, these increasing numbers of sensors will need to be more intelligent and capable. Use-cases unforeseen just five years ago are driving the development of sensors with unprecedented performance.

A growing number of applications have emerged in consumer markets. With imaging, for example, consumers are now looking for a level of photo quality in slim smartphones that had earlier only been possible with expensive and bulky DSLR cameras — and this while walking, if not running. To achieve such image quality, with the limited optics that can be accommodated in such a tight space, OEMs have deployed Optical Image Stabilization (OIS) and Electronic Image Stabilization (EIS) solutions. The performance of the OIS/EIS components, especially the motion sensor, has a major impact on overall performance. For example, the Google Pixel 3 device, which has a camera with both OIS and EIS, received a DXO Mark score of over 100 from DxOMark Image Labs, never given to a smartphone until 2018. The technology that led to the high performance is a motion tracking sensor that has extremely low gyroscope and accelerometer noise (<0.004 dps/√Hz and <100µg/√HZ respectively) and sensitivity error within 0.5%, while having very high temperature stability (within ±0.016%/◦C maximum) given the fast temperature change constraints.

It is not just cameras that require such precise motion sensor performance. Virtual reality (VR), augmented reality (AR), and mixed reality (XR) all need low noise and extremely good temperature stability.

Dead-reckoning systems are used for navigation in both consumer and commercial applications. They provide location data in the absence of GNSS/GPS signals by means of Inertial Measurement Units (IMU), which used linear acceleration information derived from accelerometers and angular rotation rate from gyroscopes to calculate altitude, angular rate. linear velocity, and position. The accuracy of these measurements is critical because errors are additive — they increase over time. For high-precision navigation, a gyroscope needs to have extremely low gyro offset over temperature (between 3 to 5mdps/^oC) and gyro noise density (below 4mdps/√Hz). 

A promising consumer application for dead-reckoning, is to sense one’s location inside a shopping mall. It is particularly challenging to sense the floor someone is on. It would allow users to know which shops are near them. More importantly, it could save time and lives by providing emergency services with the specific floor and location where the mobile caller requested help, rather than searching floor by floor. This can be achieved with high-precision, low-noise pressure sensors that can measure individual stairs. This is now achievable because pressure sensors are available that can go as low as one Pascal, or about 10 cm in precision.

In many cases, sensor performance is application-driven. For example, drones care more about bias, time, and temperature stability, while house-cleaning robots will care mostly about bias stability and noise. Virtual Reality (VR) stand-alone head mounted devices (HMD) will need exceptionally good temperature linearity and stability, as well low hysteresis and noise to provide the best possible user experience. For the very high precision required by VR and Augmented Reality (AR) controllers, sensors will need to have high resolution to be able to accurately handle the slow and precise movements of a hand.

For the Industrial Internet of Things (IIoT), including the manufacturing, automotive, and aerospace sectors, requirements are generally an order of magnitude more demanding than for consumer applications.  For example, inaccuracy for consumer applications can typically be 1000ppm (parts per million) and for automotive and IIoT, 100ppm.

For precise navigation, to take one example, vibration isolation for sensors is extremely important. The extreme vibration experienced by a driverless agricultural tractor can cause navigational errors because of its effects on the motion tracking sensor. To deal with this problem, navigational grade solutions require bias stability of below 2°/hr for gyroscopes and below 10µg on Accelerometer side, while an ARW (angle random walk) of 0.084 °/√hr and VRW (Velocity Random Walk) of 0.016 m/s/√hr. Beyond just sensor performance, it is also important to have very little misalignment errors and mounting errors, less than 0.05° for both.

The good news is that silicon MEMS allow lower cost structures than piezo, with batch processing, and this has opened new markets in low-cost sensing alternatives for these demanding applications.

“Industry 4.0” drives production towards constant and permanent monitoring of a large and increasing number of processing parameters. Sensors are integrated in new production equipment, but they can also be added to older equipment to allow for machine health monitoring. Online monitoring, instead of monthly manual read-out, brings clear benefits and ROI: reduced downtime by as much as 70%; extended motor lifetimes by up to 30%; reduced energy consumptions by up to 10%. Indeed, like a person, it is always better to catch the symptoms early than wait until they get too severe.

Take the example of a sensor’s potential to address more use-cases than ever before by ‘simply’ improving one of its performance metrics by an order of magnitude: the pressure sensor’s noise, measured as Pascal RMS. When you can achieve a pressure noise at the level of one pascal, you are able to detect stair steps of 18cm very easily – and this drives new use cases not possible before. New use cases could include activity monitoring and calories counting, improved flight control for drones, improved interior routes and travel time, rapid emergency response in the right floor of your building and more.

Microphones are also increasingly deployed in our homes, with smart speakers having multiple microphones; for example, the Amazon Echo has seven microphones. You can easily imagine a future where a family has more than 100 microphones inside their home. These devices require very high-performance microphones, with a signal-to-noise-ratio (SNR) of more than 70 dB, while having an acoustic overload point (AOP) of more than 120 dB. A good example is ICS-40730 that provides 74dB SNR and 123dB AOP. This allows the microphone to still listen to everything happening near the smart speaker even in the presence of loud noises, like a door closing too hard, or a TV playing an action movie too close to the smart speaker.

Another category of IoT and IIoT is machines that move, such as self-driving cars, service robots and industrial monitoring drones. For them to be able to “see” their surrounding precisely and avoid obstacles is increasingly important, in addition to cost and size. Range finding or time-of-flight (ToF) sensors are starting to address these applications better and better, with some using ultrasonic methods to measure distances up to five meters very precisely.

Ultrasonic technology also allows new use-cases in industrial or consumer contexts with fingerprint sensors that can work well even under metal or plastic, allowing for more challenging environments to provide secure access via fingerprint. For example, challenges such as security requirements, needing both false acceptance rate (FAR, or giving access to an unauthorized person) of at least one to 50,000 and false rejection rate (FRR, or not giving to the authorized person) of less than two percent, are still met with ultrasonic solutions, and are actually better than traditional solutions for use-cases like people using wet fingers after washing their hands or getting out of the shower.

The examples covered in this article are only a few of many use-cases that require increasing performance from many sensors, on various, and many times different, performance KPIs. Innovation is simply the way forward to address these demanding expectations, and it won’t be long before digital experiences are humanized by sensors across all industries. 

There are typically three options being considered for powering IoT sensors:

• Mains powered: historically, this has been the most straight-forward and reliable way of supplying energy to a device. However, for both economical and practical reasons, cabling is not a viable solution for the large number of devices involved.     
• Battery powered: this allows more freedom when deploying wireless devices in the field. However, there is significant environmental and practical concern of manually replacing or recharging these batteries when an empty state-of-charge is reached. Many of these individual devices will indeed be placed in remote locations that are difficult to identify or access. Importantly, the consequence of device failure due to a fully-discharged battery could be significant if the device is used for safety-related applications such as condition monitoring or in medical applications.
• Powered using energy harvesting: this has been widely considered the only logical and sustainable solution in the medium to long term. New energy harvesting and sensor technologies are now making a viable, practical option in the short term, too. It is also in-line with current trends of device miniaturisation and low-power consumption. There are various ways of harvesting ambient energy depending on what form of energy (thermal, kinetic, light) is available. Light being available almost everywhere and most of the time (either outdoors or indoors), photovoltaic energy harvesters (converting ambient light into electricity) are expected to hold the largest market size within the next 5 years.

On average, people typically spend 90% of their time indoors. Hence one can also imagine that a considerable fraction of current and future IoT devices are to be located in an indoor environment: homes, offices, supermarkets, factories, warehouses, hospitals, to name a few. However, most of existing light energy harvesting solutions, based on silicon active materials, have been designed principally for outdoor use and show a drastic drop of performance in low-light level indoor environments. Other alternatives based on dye-sensitised solar cells (DSSC) and organic photovoltaics (OPV) offer a small power density increase but are larger in size (tens of cm²) and frequently suffer from material degradation and limited lifetime (a few years only). As a consequence, this means none of the commercially available solutions can provide enough “juice” or power for a wide range of wireless applications (including wearables), where there is a clear limitation in the size of the final device or product. So is this the end of it?  

Lightricity Ltd¹ is a high-tech company which recently span-out from Sharp Laboratories of Europe, Sharp’s European R&D laboratory. After over 4 years of intense research to develop the perfect indoor solar cell, Lightricity has been set-up in 2017 to further develop and commercialise its unique technology based on an inorganic material. With efficiencies up to 35% achieved under standard artificial lighting conditions (200 lux white LED or Fluorescent/FL light), Lightricity currently offers the world’s most efficient photovoltaic (PV) energy harvesting components (ExCellLight) for IoT devices. This high conversion efficiency enables now up to six times more power to be delivered to a sensor or IoT device (Figure 1), avoiding the multiple replacements of coin cells, AAA-AA types of batteries. 

Figure 1: Comparison of indoor Photovoltaic (PV) Energy Harvesting technologies

This power boost can be used to provide more device functionality or to reduce the footprint of the complete system (to a few cm²). For example, Lightricity’s energy harvesting component can provide a perpetual source of energy for more power-hungry air-quality gas sensors, such as CO₂ sensors. The Lightricity team, in collaboration with Gas Sensing Solutions (CO₂ sensor manufacturer) has previously demonstrated an autonomous wireless CO₂, temperature, light, humidity sensor that can perpetually perform and transmit all environmental measurements every 2min 30s at 200 lux, only using a 10 cm² Lightricity PV cell². Over 10 years typical product lifetime, up to 150 battery replacements can be avoided, assuming 1000 lux average illumination (e.g. retail environment) - thereby saving €100’s in maintenance and battery costs to the end-users (Figure 2).

Figure 2: Lightricity battery equivalent under indoor conditions

In a project funded by Innovate UK, and in partnership with Ilika (solid-state battery storage) and E-peas (power management chipsets), Lightricity is currently developing a second generation of compact everlasting power supply that can connect to any IoT sensor or wireless tracking devices and beacons (Figure 3).

Figure 3: Energy harvesting power pack (Lightricity, Ilika)

Some environments can be particularly harsh: Lightricity’s energy harvesting component has been designed to withstand temperatures of up to 150 °C continuously, and up to 250 °C intermittently. The temperature coefficient of Lightricity’s device is twice better than that of silicon, translating into up to 8x performance improvement at elevated temperatures. This opens new applications in automotive and industrial environments, and also new use cases in health monitoring for worker safety, emergency responder safety and sports applications.

These developments demonstrate that powering IoT devices and sensors by energy harvesting is both practical and beneficial for lowering cost of ownership. This opens the way to large scale deployment of connected IoT devices in the short term.

Next-level 3D ultrasound sensor based on echolocation

Toposens provides echolocation, i.e. the ability to “see with sound” for the automotive industry and robotics. Up until today, ultrasound has only been used for one-dimensional applications, e.g. precise distance measurement for industrial applications. Toposens invented the first 3D ultrasound sensor system. It can detect multiple objects and people in real-time and generate a 3D point cloud of its near-field environment.

By using the principles of bionics i.e. engineering a reliable product based on biological methods found in nature, many of the sensor’s features are closely related to the echolocation system used by bats. Toposens sensors send out signals, evaluate the echoes that come back and precisely detect where in a given space both static as well as dynamic objects are located in real time.

Even though ultrasound has been studied by scientists for over 200 years, its true capabilities have not yet been unleashed. Ultrasound technology provides many interesting characteristics that have not been combined into any other 3D sensor yet. Toposens’ current sensor TS ALPHA is very robust and can see in the dark, under varying lighting conditions, through dust and dirt and does not have any problems with transparent or reflecting surfaces. It currently operates at the lower end of ultrasound with approximately 40kHz.

3D Ultrasound for the Near-Field: A Robotic Application  

To give potential customers the possibility to test the sensor in various applications and under multiple conditions, Toposens launched a Development Kit for testing and prototyping purposes. The latest Development Kit TS ALPHA covers applications in both the robotic as well as the automotive field. To simulate a real-world use case where the sensor evaluates the near field environment during relatively slow, Toposens recently tested the Development Kit in a specific robotics parcours.  

A pipe of convex shape, fluffy fabric surfaces and a door mat rounded off the parcours for the ultrasound sensor. While TS ALPHA felt at home in his natural habitat and calculated the 3D coordinates of all nearby objects in real time, those same objects would not have been detectable under the same conditions by conventional cameras, radars or LiDAR systems. Due to limited visibility and lighting but also due to the shape and characteristics of the objects like the transparency of the glass bottle or the fine mash of the bowl, the objects would be difficult to detect in the raw data of existing sensor technology. These issues of reflecting and transparent surfaces or bad lighting conditions appear frequently when sensors are being used in the automotive field. Darkness, fog, wind and sun do not interrupt ultrasound, whereas the weather heavily affects cameras and lasers.

3D Ultrasound’s role as far as ADAS are concerned

As outlined in the parcours experiment, 3D ultrasound technology has strong benefits when it comes to near-field sensing. It covers objects in the opening angle of up to 180° (horizontally and vertically), has a range of up to 5m and generates real time data in the form of clustered data points.  

Toposens is using a minimalistic approach with the TS ALPHA Sensor by limiting its size while keeping all of the functionalities at a peak level. Standard hardware components are put together in a special layout and combined with proprietary algorithms, machine vision and AI-software. The hardware components are off-the-shelf. They are very small, use little energy and weigh below 20g. Unique is the ability of the sensor to immediately evaluate the received data on a small built-in microcontroller, making an external control unit obsolete and keeping the generated data flow low. Only for further clustering and processing of the data, additional hardware is needed. The company’s main knowhow lies in the sophisticated algorithms, which turn the ultrasound signals into 3D coordinates.

Besides covering the long distance, a fundamental requirement for autonomous vehicles is to perceive their near-field surroundings as precisely as possible. Toposens sensors provide reliable, rich and three-dimensional data (point cloud) for the close-range environment around the vehicle. The sensors are therefore well suited for applications in the automotive field and are able to cover areas, that conventional radar and camera technologies are struggling with. 

Parking sensors 

For improved accuracy and safety in everyday parking situations, sensors have become an essential in every car. Up until today, they helped the driver steer more safely and guide him into a parking spot via sound and light signals. Although the currently installed sensors provide a good guideline to avoid collisions, while navigating the car into a tight parking spot, they still leave a lot of room for improvement. Conventional ultrasound sensors used for parking assistance only record one-dimensional data, namely the distance to the closest object. Azimuth and especially elevation angles of objects can hardly be calculated, and vertical opening angles must be limited strictly, so many objects – such as the curb – cannot be detected at all.

As every drive starts from a parking position and ends with a parking maneuver, close-range sensing is one of the crucial challenges in going from L3 to L4 automation. Thinking about autonomous parking and precise manoeuvring in tight parking spaces, distance measurement alone quickly becomes insufficient. In order to take things to the next level, more sophisticated sensors are needed around the car. By ultimately using 3D ultrasound sensors, the car’s surroundings can be evaluated accurately. Furthermore, blind spot monitors are an option that may include more than monitoring the sides of the vehicle. What mainly differentiates the technology from existing parking sensors are both the algorithms and the special layout which allow 3D sensing with commoditized, readily available hardware.

The sensor data is further useful for additional comfort features, e.g. gesture control to open doors and trunks, positioning the vehicle for automated charging (for EVs), collision avoidance for automatically opening doors and many more.

Passenger monitoring in the interior of the car

With further improvements in autonomous driving space, the behavior of the driver is also likely to change. While drivers today have to be completely focused on the road, ready to react every millisecond, this could change once cars can steer and break fully automatically. The drivers can lean back and relax, work on their computers, turn to the children in the back seat, or temporarily enjoy an expanded infotainment program.

In this vision of autonomous driving, drivers can use their travel time for activities besides driving. This puts new demands on assistance systems. Just like the numerous sensors available for analyzing a car's external environment, similar knowledge is needed for the interieur in order to realize a more secure and intuitive interaction experience. In this scope of application, the usage of 3D ultrasound provides interesting advantages. Data gained from the 3D ultrasound sensor is able to identify the number of people sitting in the car, their size and their posture. Based on the information regarding where people are sitting and how big they are, airbags could also be adjusted to individual body sizes to further improve safety. Toposens technology does not collect any personal data since ultrasound cannot evaluate visual material. Instead, anonymous point cloud data is recorded. This is an especially important consideration in terms of privacy and data protection. Last but not least, gesture simulation in the interior of the car can be used for information and entertainment purposes, e.g. controlling your car’s infotainment systems with simple gestures.

There is no doubt that future autonomous vehicles need more assistance to operate safely in our everyday lives. Whether you are living in a big city where people detection and accurate parking plays a major role or on the countryside where automated charging can assist you, sensors will have a big impact on your everyday life. Toposens contributes to the high quality of sensor output and keeps developing new products to accelerate the advancements in sensor technology.