The joint surging advance of sensors, communication, and integration technology allows the realization of more versatile and pervasive systems in nearly all domains of industry and daily life. Established and emerging application domains are, e.g., measurement, instrumentation, and automation, Industry 4.0, the Internet of Things, cyber-physical systems, and ambient intelligence/assisted living. Smart environments, in particular, in homes are a prominent example, where deeply embedded intelligent sensory systems add significant functionality unobtrusively merged into everyday life structures and devices. Miniaturization of such autonomous, potentially wireless, sensory systems for distributed measuring and observation can be found from sensate floors, over leading edge integrated data loggers to lifestyle and sportive gadgets, such as smart watches or activity trackers .

Further intriguing research work is done in the field of lab-on-chip devices; see, e.g., , , , and . cover a wide field from medical to food applications. Point-of-care diagnostics are one thrilling field of lab-on-chip application systems. Advanced microelectromechanical systems (MEMS) and packaging technology are employed for potentially disposable system solutions with a high level of sophistication and potential price tags. Though this class of systems has numerous features in common with the Lab-on-Spoon (LoS) research presented in this paper, and has inspired the naming of the project, subtle differences, e.g., in cost, embodiment, reuse, mobility issues, autonomous system implementation, and system level integration can be identified. In perspective, a convergence of lab-on-chip technology and the Internet of Things, cyber-physical production systems, and Industry 4.0 application fields, e.g., for in-line and portable measurement, can be expected. The information obtained by such sensory systems can serve to achieve improved or novel assistance functionality in various domains of daily life, e.g., in the domain of nutrition, to assist the user in estimating achieved calorie burning in sports and the calorie contents of food, or even in providing better performance to unskilled users or supporting impaired persons in restoring lost sensing capability. In the kitchen environment, numerous product and research activities can be found to improve device performance or to achieve assistance system functionality. Relevant commercial and research work has been summarized, e.g., in , indicating the potential of sensing and sensory context to achieve a new class of assistance system for this domain, denoted as culinary-assistance systems . A second, related field of application, or better, concern, has emerged in the last years. Sources of unintentional or intentional contamination of soil and sea, e.g., by radiation, chemical, or biological pollution, have increased substantially and, correspondingly, contaminated food can enter the food chain from various sources and reach the consumer unnoticed. For instance, the omnipresent problem of product fraud or falsification, e.g., for frying oil , has also become more noticeable in semi-processed and processed food supplies. The increasing need for food quality and safety monitoring adds further momentum to the outlined research to provide such support at a feasible cost in the consumer's home.

This challenge, in addition to the work presented here, triggered activities on assistive technologies in food safety and food analysis. One example is the portable Bio-Scout of SARAD GmbH , designed to discover radiation in food. Quite recently, e.g., the Vessyl , a metal cup claiming to be able to identify cup contents and give a reckoning of the calories, and the Thai e-tongue , as a particular representative of the research on artificial degustation systems or e-tongues, reviewed in , conceived to measure and assure the quality and authenticity of original Thai food, have emerged.

In our research, we aspire to contribute to advanced living-assistance systems for smart homes and, in particular, kitchen environments (see Fig. ), that receive information from new smart autonomous devices, which are inspired by the technologies summarized above and embodied as common items of daily life, e.g., bowls, cups, forks, or spoons. The focus is on sensing principles and packaging technology that allow the achievement of low-cost, low-power, high-volume, multi-sensory integrated intelligent sensory systems and devices for both cooking assistance and food safety.

To achieve this goal, pioneering work by MIT Media Lab on smart or intelligent spoons by is picked up and extended to wireless communication, advanced packaging, and low-cost multi-sensor capability, in particular impedance spectroscopy, which is a method of increasing impact and applicability, e.g., in , , and . However, in the majority of applications, powerful but expensive and bulky desktop equipment, e.g., an HP4195A network analyzer with an impedance measurement extension, Agilent 4294, LCZ meter model 4277A, or Xiton Hydra 4200, etc., are used. Applications are in the field of bio-impedance spectroscopy and electrochemical-impedance spectroscopy , and medical tasks like skin cancer or wound healing monitoring or fish, liver, or meat freshness determination , tea quality or general food monitoring in the food industry , as well as water monitoring and detergent concentration determination in, e.g., dishwashers. The size of common instrumentation equipment hampers the system realization beyond discrete proof-of-principle prototypes. This has motivated various dedicated embedded designs based on off-the-shelf components and PCB integration. However, the existing commercial solutions, such as the AD5933 chip, cover only a small part of the interesting impedance, frequency range, and measurement quality for the different application domains, which stimulates ongoing dedicated chip design activities.

In this work, the concept and the first prototype of our Lab-on-Spoon will be presented. Section  will describe the concept and architecture of LoS, Sect.  will give details of the first LoS prototype, and Sect.  will describe selected applications and conducted experiments, including applied computational intelligence methods and tools. Concluding, the motivation for a custom CMOS chip and a reconfigurable, potentially MEMS-switch-based LoS will be discussed, and a preview of ongoing activities for an improved LoS version will be given.

Figure  illustrates by a block diagram the concept of the proposed living-assistance system, and, in particular, the LoS as one possible autonomous sensory front end to it. The institute of integrated sensor systems (ISE) central smart kitchen host is designed to communicate with various smart devices for activity recognition and sensor context acquisition by wire, e.g., standard bus or power-line communication, or standard wireless communication, such as XBee. For instance, Sensitec current sensors are applied for activity recognition and loading assessment of electrical appliances, and the popular MS-Kinect sensor is applied for gesture control together with an emerging electronic or interactive cookbook (see Fig. ), which is inspired by professional tools like ChefTec (see ). As exemplified in Fig.  for a few extracted recipe lines from a simple Chinese dish, for each step of ingredient inclusion and preparation, the e-cookbook is conceived to call on the sensory context, which will be provided here by the LoS, and by further resources emerging in our current research and development. In this step, ingredients can be checked by LoS for agreement with the current preparatory step, freshness, fraud, or contamination. The LoS is equipped with a microcontroller that runs the measurement control, the sensor readings, and the communication software to collaborate and exchange data with the smart kitchen host indicated in Fig. . Power awareness of the autonomous measurement system equipped with a rechargeable accumulator is achieved, e.g, by employing power-saving devices and sleep modes, etc. Conceptually, sensors for integrated temperature, color, infrared, impedance, pH, viscosity, weight, as well as radiation measurement, are aspired to.

Wireless communication with the smart kitchen host is realized by an RF module, e.g., by the XBee standard. The sensory context for each step in the recipe can be acquired and processed by computational intelligence methods on the host for food assessment. Thus, wrong or inadequate or even dangerous ingredients potentially can be detected, indicated to the user, and excluded from further processing. In addition to ingredient monitoring, quantity determination for dosing, and the assessment of intermediate cooking step results are aspired to as part of the concept.

LoS data have to be processed by suitable means to allow the assessment according to common fuzzy textual statements in recipes on color, consistency, crispiness, etc., of the meal components. In addition to standard textual recipe information, LoS sensory contexts from successful (expert) meal preparations could be stored for each step to provide an even better basis for assessment of the current user activities.

It is anticipated that the spoon will be in sleep mode and get woken up by either a button press or, alternatively, a wake-up by radio from the host. The button is allocated in the spoon handle and serves also to synchronize the taking of the measurement. The system can visually, possibly complemented by standard speech output, prompt the user to enter the required ingredient into the spoon and confirm this by pressing the same button. That procedure helps to avoid the spurious analysis of LoS contents, e.g., of an empty spoon, of a spoon filled with residuals of prior activity, or of a spoon in cleaning. Measurement data will be acquired, sensory context data will be communicated to the host, and the LoS will go back to sleep again.

The potentially wide ranges of measurement quantities for different ingredient categories as well as tolerance and calibration issues require the on-the-fly reconfiguration capability for LoS. Some reconfiguration and self-x features are already included; more are under consideration for the implementation of the next LoS version. This will be discussed in more detail in Sect. .

The physical realization of the concept will exploit contemporary techniques of 3-D printing and the corresponding 3-D integration or packaging of sensors and electronics. 3-D printing allows the easy, low-cost, and rapid realization of arbitrary prototype shapes, including even sintered metals in the spectrum of materials. Suitable spoon shapes can be created in which sensors and electronics could be snapped into place. More advanced system-in-package approaches and technologies, e.g., the molded interconnect device or the active multi-layer technology , merge electronics and package design, and also seem very promising for LoS embodiment.

A subset of the outlined multi-sensor LoS concept and architecture outlined in the previous Sect.  has been realized in a hand-held, multi-sensor, and autonomous system implementation and embodied in spoon shape for the first time.

In Fig. , the block diagram of the LoS prototype, including three different sensors, is given. Their geometrical alignment in the spoon cavity is outlined in Fig.  according to the LoS picture given in Fig. . The spoon cavity has a maximum depth of 7 mm. A ceramic substrate pt10k temperature sensor of UST GmbH, which comes with a custom calibrated PCB along with a corresponding third-order calibration polynomial, is placed in the center front of the cavity. The MCS3AS true color sensor with its corresponding MTI04QS transimpedance four-channel amplifier chip from MAZeT GmbH is placed in the center of the cavity with the objective to be always completely submerged in the liquid to be analyzed. The impedance spectroscopy measurement unit consists of the AD5933 network analyzer chip and gold-plated electrodes, which are placed at the center back of the cavity, just below the active illumination LED. By this placement, they will be completely immersed even for a low degree of spoon cavity filling. The placement of the sensors is mainly subject to the constraint to avoid variations in measurement due to uncertainty in filling level. For the impedance spectroscopy, the basic two-wire measurement approach is applied together with an optional simple analog front end for materials of very low impedance, e.g., ingredients of high salinity.

This complementing standard circuit reduces the output voltage and prevents the AD5933 from overloading by excessive output current in low-impedance measurements below the 1 kΩ range. For substances of higher impedance, e.g., oils, the AD5933 will be just used straight, bypassing the front end. Of course, the amplification of the AD5933 input stage has to be reconfigured too with regard to the aspired-to impedance measuring range and the use or bypass of the front end. These setting requirements and calibration issues give rise to the reconfiguration concepts discussed in Sect. . Additional reconfiguration requirements come with the color sensor. Depending on the illumination intensity, the transimpedance of the sensor electronics can be digitally adapted or programmed in three stages in the MAZeT MTI04QS chip.

However, to provide reduced vulnerability to environmental illumination variations, LoS has been equipped with an active illumination of the spoon content by a white-light LED, which is activated during color measurement. Temperature and color values will be converted to digital by the 10 bit ADC of the microcontroller, while impedance values will be converted by the internal 12 bit AD5933 ADC.

In this context, a further feature for user interaction has been added, which gives a haptic feedback on the spoon contents' temperature. Inspired by previous activities of, e.g., MIT's smart sink , where tap water was illuminated by a color coding the temperature, from cold (blue) to very hot (red), to warn the user and avoid injury due to scalding, the LoS was extended. In wake-up state, without host request on data, the spoon in this mode continuously measures the temperature of the spoon contents and illuminates the spoon contents in a corresponding color by a full color LED. This is illustrated in Fig. . In addition to that warning function, thresholds and color assignments can be reconfigured by the host, e.g., in tasks where a liquid has to be in an arbitrary interval or even meet an exact value. The water temperature for yeast bacteria cultivation in bread making is one example, which should best be about 30 ∘C.

The microcontroller and system of choice for LoS, after considering and testing alternatives, in particular the low-power EnergyMicro Cortex M3 EFM32G890 F128 microcontroller, was the Arduino system family, due to the well-known properties of flexibility, easy and rapid prototyping, and a large portfolio of modules and accessories. The Arduino system family with wireless extensions has evolved to be very popular for Internet of Things realizations. For reasons of constrained space, the Arduino Pro Mini board, 5V supply version, equipped with the ATmega 328 microcontroller had been chosen, which provides time-multiplexed ADC inputs for temperature and color sensor reading, I2C bus support for the AD5933 communication, and general purpose digital I/O, including a PWM option for button reading, as well as white light and color LED control.

A USB module has been included in the system design for (re)programming of the microcontroller, as well as wired supply, host communication and data transfer for basic LoS operation in the development phase. Figure  shows an early prototype of our LoS system without accumulator/XBee extension and before finalization and package sealing. For autonomous operation, the system is completed by a lithium polymer accumulator and a standard XBee communication module. This option is pointed out in Fig.  by the only dashed box, labeled Accu/XBee(Arduino), and the corresponding physical realization is illustrated in the top right corner of Fig.  as an plug-on board extension to the spoon handle. The spoon package itself has been shaped in a first-cut design employing the Blender software and the Makerbot Replicator I 3-D printer. The currently employed thermoplastic spoon package serves only for the first step of evolution and has to be replaced by a material more robust to the full range of common cooking temperatures and food safety regulations.

The LoS embedded system software is developed in C in the standard Arduino development environment 1.5.6. On the host side, a Python-based interface communicates with the LoS via the serial interface and reads the data in for the application software. Depending on the connection, directly by USB during development or by wireless in the actual application, a second Arduino/XBee module is required as a gateway on the host side.

In the following, front-to-back application of the LoS from sensory registration to data analysis and recognition system design and employment will be investigated for selected liquid food samples. The proprietary ISE QuickCog tool () offers the required facilities to analyze, fuse, and recognize the acquired multi-sensor data, but the integration of LoS spoon control and a data acquisition interface seemed to be more promising on a multi-platform flexible and open system, which is provided by the Python-based Orange system . So, in this work, Orange was extended on both the Windows and Linux platforms by the LoS interface and a QuickCog interface to immediately exploit features and efficient methods from the field of computational intelligence still unavailable in Orange. Moving these features and methods to Orange and public availability is one of our research goals in the context of LoS research.

Plots for interactive data visualization and analysis can be generated by, e.g., the choice of two particularly relevant variables from the measurement, or by dimensionality reducing mapping techniques, e.g., by Sammon's nonlinear mapping and related fast techniques . The first option is very transparent and the axes of such a scatterplot have a clear physical notion. However, in particular, in multi-sensory systems, the required salient information rarely is provided by only two or three measurement inputs or variables. The latter option can deal with measurement data of arbitrary dimensionality and reduces the data dimensionality to a 2- or 3-D plot under the constraint of, e.g., preserving the distances of the data points as undistorted as possible. These distances represent the similarities of data points and their underlying physical measurement data. In such a plot, commonly denoted as a feature map or a feature space projection, the plot axes have no assignment to a particular physical notion.

In gas sensing, commonly linear discriminant analysis is also employed for the same visualization purpose. As this is a supervised method, which includes the labeling or class affiliation of the measurement in the dimensionality reducing mapping, we prefer the unsupervised Sammon nonlinear mapping and employ it for LoS data presentation and assessment in the following.

The maps thus generated can help in understanding and optimizing both data acquisition and processing for ingredient recognition/identification or grading in new applications (see Figs. –). The feature map or feature space projection can be complemented by labeling the data points with class information as well as temperature context information, which could stem from the pt10k sensor in the spoon volume or the AD5933 or other chips' internal temperature sensors. Furthermore, QuickCog provides a set of automated feature selection (AFS) options, which assist in efficient feature-level fusion from the different sensor channels to obtain well-discriminating but lean intelligent systems. For the generation of the following feature maps and the final results table, the qoi overlap measure with k neighbor parameter k=5 and the simple sequential-forward selection (SFS) scheme has been employed. AFS leads to more lean, in some cases better performing decision systems, but the particular advantage in impedance spectroscopy is that only a few case-specific spectral components have to be measured, and the measurement time can be reduced significantly. The full sweep will only be needed in the analysis of new tasks and related measurement data.

In the following, two different kinds of experiments were conducted. In the first group, the discrimination capability of the LoS for various common cooking ingredients will be investigated and demonstrated.

In the second group of experiments, the LoS grading capability with regard to loss of freshness or the presence of contamination will be tentatively investigated and demonstrated.

For all experiments, the following settings for sensor signal conditioning have been applied. The transimpedance setting of the LoS color sensor is set to 500 k. The frequency sweep range of the AD5933 chip is 10–100 kHz, with a frequency increment of about 175 Hz. The temperature sensor module is calibrated, has no setting options, and returns a temperature value in degrees Celsius. LoS gives, for each measurement, 1 temperature value, 3 values of RGB channels for color registration, and 512 complex impedance values, given as 1024 magnitude and phase values. This amounts to the data vector of 1028 entries exemplified in Fig. .

In addition, Fig.  illustrates the plot of the impedance magnitude and phase spectra from LoS measurement data for several examples of the contamination recognition and data given in the following Fig. . Each point in Fig.  corresponds to one measurement, i.e., one of the spectra given in Fig. .

For the measurement of substances with a low impedance or high impedance range, two different LoS prototypes with an employed or bypassed analog front end, as indicated in Fig. , were employed in the work described in the following.

In the following, 30 measurement repetitions have been adopted as the standard for each substance or ingredient.

The measurement currently proceeds in three phases measuring the temperature first, followed by accumulative color registration of 50 samples of each RGB channel with active white-LED illumination switched on (see Fig. ), and concluding with the impedance spectroscopy measurement in the given frequency sweep range. After the scheduled number of measurements, which were triggered by the push button and/or by wireless, the LoS goes back to sleep.

This paper presented the concept and the first embodiment of the Lab-on-Spoon autonomous, hand-held, multi-sensory system as a low-cost front end of an intelligent assisted-living system with a distributed sensor network for home-based smart kitchen applications . The LoS is designed to provide sensory context to the ingredient and preparatory step results monitoring of recipes in an interactive cookbook and, thus, support both unskilled or challenged persons by improving or partially restoring perceptive and assessment ability. Furthermore, LoS is conceived to detect decay and/or contamination in food, which might be imperceptible to humans. With regard to the severe and increasing pollution of soil and water worldwide, contaminations could reach the consumer undetected, and strongly advocate the creation of capable, yet affordable, local sensing capability at the consumer's end of the food chain.

The first LoS prototype was implemented in our work on the favorable Arduino platform with temperature, color and impedance sensing, and integrated into the Orange system for capable multi-sensor signal processing and recognition and life or online classification, e.g., on CeBIT 2014 . With regard to our goals, it showed encouraging capabilities and sensitivity for a challenging application spectrum from food classification to grading. Improvements in electronics, packaging, sensor palettes, and algorithms, as outlined in the discussion above, are on the way. Self-x features for dependable and self-sufficient operation, as needed for related cyber-physical systems, the Internet of Things, or Industry 4.0 domains, are under investigation for LoS. Further inspirations are expected from the thriving lab-on-chip research field.

In addition to advancement of the scientific LoS development and improvement, including the abstraction to alternative devices, e.g., a lab-on-fork, lab-in-bowl, etc., to probe non-liquid food and materials, such as meat or cheese, etc., commercialization with industrial partners is aspired to, with a potential mass market in mind. Recent competitive approaches underpin this view, e.g., and . From the results on frying oil, and preliminary experiments on combustion engine oil, an extension of the project to simple and low-cost motor and gear oil sensors seems to be promising and straightforward.