Harnessing the Power of Artificial Intelligence (AI) and Internet Of Things (IoT) for Elevating Safety

12 October 2018

Prepared by Acada Consulting Inc. with Technical Safety BC

Table of Contents

Wai Ho Choy, Abhijit Pandhari, Ido Refaeli, and Courtney van Ballegooie prepared this report as part of the 2018 consulting project contracted to Acada Consulting in partnership with Technical Safety BC.

For more information about this project, please contact:

Project Lead
Wai Ho Choy - Consultant
Email: choywaiho1@gmail.com

Project Team/Authors
Wai Ho Choy - Consultant
Email: choywaiho1@gmail.com

Abhijit Pandhari - Consultant
Email: abhijitpandhari@gmail.com

Ido Refaeli - Consultant
Email: idorefaeli@gmail.com

Courtney van Ballegooie - Consultant
Email: cballegooie@bccrc.ca

DISCLAIMER

The information and views set out in this publication are those of the authors and may not reflect the official opinion of Technical Safety BC. The authors do not guarantee the accuracy of the data included in this study. Neither Technical Safety BC, the authors, nor any person acting on their behalf may be held responsible or liable for the use which may be made of the information and recommendations contained herein.

© Acada Consulting Inc., 2018. All rights reserved. Reproduction is authorized provided the source is acknowledged.

Acknowledgments

We want to take this opportunity to thank the many change-leaders at Technical Safety BC who have wholeheartedly contributed their time, enthusiasm, and encouragement into this project. Thank you, Catherine Roome, Abraham van Poortvliet and Soyean Kim for your continued support of this project, your belief in us gave us and your organization the confidence to venture into new but important technical safety challenges.

We also wanted to thank all the people who we have met and worked with along the way, Doris Li, Zack Zhu, Ravi Bisla, Yuyi Zhou, Phat Tran, Frank Do, Bo Feng, Tom Ng, Nav Chahal, TJ Sanchez, Cam Surine and Carson Lew at Technical Safety BC. Thank for being supportive of our work and for treating us as one of your own. We are very grateful to have such open-minded and approachable mentors that are supportive of the young talent here in BC. Special thanks as well to Phat Tran for providing the early inspiration for this project from a Research Day project at Technical Safety BC.

We also wanted to extend our thanks to Dr. Richard Peters, Dr. Behraad Bahreyni, Dr. Soheil Azimi, Dr. Hamid Omid, Dr. Ali Narimani, Dr. Rocky So and Floris Hoogenboom who provided valuable advice as experts in the field of IoT and AI. As we are students in this field, your guiding advice helped us refine our thinking and approach. We also wanted to thank the various municipal and building asset owners that have contributed their time, effort and interest in making British Columbia a much safer province.

Research Collaborators

Technical Safety BC (formerly BC Safety Authority) is an independent, self-funded organization that oversees the safe installation and operation of technical systems and equipment across the province. In addition to issuing permits, licences and certificates, we work with industry to reduce safety risks through assessment, education and outreach, enforcement, and research.

Acada Consulting Inc. is a graduate student-founded business that began in 2018. Acada Consulting originated from the Graduate Consulting Program (GCP) through the Graduate Management and Consulting Association (GMCA) at the University of British Columbia (UBC). GCP provided a unique opportunity for graduate students and postdoctoral fellows to apply their critical thinking and communication skills to solve real-life business problems for companies, government agencies, and nonprofit organizations in BC. After the successful completion of the original 3-month GCP project in 2017, which ended in a successful client report and presentation, the four GCP members founded Acada Consulting in order to continue their pursuit of providing graduate students the opportunity to apply their skills in a business-oriented environment towards socially-meaningful causes.

Intelligent Sensing Laboratory (ISL) is the leading Canadian research laboratory with focus on development of materials, devices, systems, and methods for various sensing applications. We conduct experimental research on the development of solutions based on the integration of nanomaterials, micromechanical, and microelectronic devices. We then employ advanced sensor signal processing algorithms, such as sensor fusion and machine learning, to enhance the performance of our systems. Much of our research is carried in collaboration with industrial partners from automotive, defense, and telecommunication sectors.

delecta is an AI consulting company based in Vancouver. delecta assembled a world-class AI team to help our clients win. We are a team of physicists, data scientists, mathematicians, and machine learning experts with a passion for innovation. Through our relationships with the world’s leading research institutions, we conceive, prototype, and benchmark Data Science applications in a wide selection of industries. Our team brings 60+ years of combined expertise in providing interdisciplinary solutions to partners in academia, finance, energy, and the life sciences. The deep technical knowledge of our management team provides us with a unique advantage in providing disrupting Big Data solutions to our clients. At delecta, we are committed to creating a thriving and respectful environment and culture for our teammates. This culture enables us to provide an exceptional experience for our customers and helps to build a lifelong relationship with them.

axSense Technologies Inc. is a Canadian corporation engaged in the design and sale of high-performance accelerometers based on Micro-Electro-Mechanical Systems (‘MEMS’) technology. axSense Technologies aims to commercialize a technology developed through a large-scale research project over nearly 5 years of intensive research. axSense has developed ultra-sensitive vibration sensors with unprecedented performance with a small footprint. These devices are so sensitive that they can detect vibrations of air or water molecules due to acoustic or sonar waves. The sensitivity of the device allows detection of whale sounds as far as 65km away from them!

Introduction: Keeping 25,000 Elevators Safe in BC

Elevators represent an integral facet of transportation in metropolitan areas, allowing for fast and safe movement of people and goods within multi-storey residences, large commercial buildings (e.g. malls and offices) and public infrastructures (e.g. hospitals and schools).

There are over 25,000 elevators across British Columbia (BC) and their safety is a joint responsibility between asset owners, vendors, maintenance contractors and Technical Safety BC, the mandated provincial safety regulator for technical systems.

The first step to preventing an incident is to ensure that elevators are installed correctly and receive regular and properly-conducted maintenance. This is typically performed by certified elevator technicians, often working for an elevator vendor or a third-party maintenance company on behalf of asset owners.

The second step involves routine periodic inspections by Technical Safety BC’s elevating device safety officers then ensure that the work of the technician is completed on time and meet required standards. A major part of that safety regulation involves the inspection, approval and incident investigation of local elevators, especially those that have previously been reported to cause injuries, entrapment of passengers, or other anomalous behaviors.

With the rapid development of real estate and infrastructure in BC, more elevators will be installed throughout the province. Both the installation of new elevators and the aging of existing elevators contribute to an ever-increasing demand for maintenance and inspection personnel that cannot be satisfied purely by the training of new workers (Figure 1).

Increasing Demand for Elevator Maintenance and Inspection

Figure 1. Challenges faced by the elevator safety industry and the role of the Internet-of-Things (IoT) in improving the allocation of existing resources

To ensure the quality and safety of elevators in BC over the foreseeable future, without incurring additional human resource requirements or other financial costs, a redesign of the technical safety environment must be taken into consideration.

The main challenge and opportunity facing elevator safety is that there is a lack of a process to systematically predict and prevent the occurrence of an elevator incident without continuous in-person monitoring. Such continuous monitoring could be more efficiently and effectively achieved through the implementation of Artificial Intelligence (AI) on Internet of Things (IoT) devices. This would not only ensure more reliable and scalable monitoring of all of these elevators but would also be able to predict and prevent future incidents that may occur. As pioneers of technical systems monitoring, Technical Safety BC would be the first public regulator advancing the use of IoT and AI, in elevator incident monitoring and prediction in Canada.

In this report, we describe an Internet of Things (IoT) approach for monitoring elevator health and predicting performance issues before they develop into serious incidents. The IoT prototype that was co-developed with Technical Safety BC and tested in partnership with local municipalities and building stakeholders has so far been successful in collecting data from the sensors from the participants’ elevators across the BC lower mainland. Using this data, Technical Safety BC was able to identify and predict when elevator anomalies occur with the help of machine-learning and AI algorithms implemented by their Research and Analytics team.

Introduction: Internet-of-Things (IoT) and elevators

IoTs are systems in which sensors are coupled to everyday objects, such as elevators. This “smart” elevator is then able to connect to the internet, allowing asset owners to remotely monitor the status of these objects and their environment, and in certain situations, even send instructions back to these objects.

Sensors on an IoT elevator collects performance data

Figure 2. Sensors on an IoT elevator collects performance data which is then sent to a digital cloud server for storage and further computation

Attaching sensors to elevators is not a new idea; existing elevators are already equipped with basic sensors for detecting excessive speeds of travel, measuring electrical consumption, and for accurately aligning the elevator to the correct floor height to prevent tripping hazards. However, the novelty of IoT elevators is that they are connected to the internet, allowing for real-time performance monitoring from a remote location (Figure 2). The sensors attached to IoT elevators also encompass a wider variety of elevator health measurements including but not limited to temperature, noise, vibration, humidity, speed and acceleration in addition to providing a higher level of sensitivity and accuracy.

Given these advantages, it is likely that IoT elevators will be integrated and installed within BC in the near future. Elevator giants such as OTIS, KONE, Schindler, ThyssenKrupp and even smaller players like Hitachi have begun introducing IoT-connected elevators as their latest line of elevators in major cities across the world (Table 1). Utilizing real-time data, these vendors are now able to identify existing elevators problems and predict future issues, thereby saving building owners and maintenance contractors valuable time and labor cost.

Company

Known IoT elevator services

KONE

IoT service: KONE 24/7 Connected Services
IoT-Cloud platform: IBM Watson IoT
Number of IoT elevators: 1.1 Million elevators

ThyssenKrupp

IoT service: ThyssenKrupp MAX
IoT-Cloud platform: MAX Box (in-house device) + Microsoft Azure
Number of IoT elevators: 120,000 elevators (10% of total)

Otis

IoT service: Otis ONE
IoT-Cloud platform: Microsoft Azure
Number of IoT elevators: 300,000 elevators (15% of total)

Schindler

IoT service: Schindler Ahead
IoT-Cloud platform: Schindler Ahead Cloud + Huawei + General Electric
Number of IoT elevators: Data not available

Hitachi

IoT service: No specific name for IoT elevator service
IoT-Cloud platform: Lumada
Number of IoT elevators: Data not available

Table 1. Overview of IoT elevator services offered by 5 global elevator companies

As the technical safety regulator for elevating devices in BC, Technical Safety BC stands to benefit from the implementation of IoT elevators in this province. First and foremost, Technical Safety BC can use the data from IoT elevators to identify elevators with the most urgent technical and safety needs. This allows Technical Safety BC to allocate its safety officers more efficiently and ultimately, cover more elevators that need inspection. Secondly, IoT elevators provide Technical Safety BC’s teams an opportunity to understand the IoT technology landscape and its potential and challenges. As technical equipment becomes increasingly digitized, technical safety stakeholders need to be able to make decisions with confidence and experience that can only be gained by early exposure to technologies such as IoT.

Despite the clear benefits provided by IoT elevators, shown in Table 2, the option to install new IoT elevators is frequently not available or financially acceptable to many asset owners in BC. Additionally, the information within each IoT elevator is currently owned and controlled by each individual elevator vendor, making this data exceedingly inaccessible. Thus, there is a need and opportunity to develop an IoT-based performance-monitoring solution for elevators that is affordable and independent of the major vendor companies.

1 Altoros, “KONE Monitors 1.1M Elevators and Escalators with IBM Bluemix and Watson IoT”, January 2018, https://www.altoros.com/blog/kone-monitors-1-1m-elevators-and-escalators-with-ibm-bluemix-and-watson-iot/
2 IBM, “Case studies: KONE Corp.” ,https://www.ibm.com/case-studies/kone-corp
3 ThyssenKrupp, “Maintenance with MAX: predicting elevator break downs – before they happen”, August 2018, https://engineered.thyssenkrupp.com/en/cloud-maintenance-solution-max/
4 ThyssenKrupp, Techforum magazine, January 2016, https://www.thyssenkrupp.com/media/publikationen/thyssenkrupp_techforum/2016_6/thyssenkrupp-techforum_1_16_en.pdf
5 Otis, “Otis Launches ‘Otis ONE’ IoT Service Solution for the World’s Largest Elevator Service Network”, August 2018 http://www.otis.com/en/us/about/news-and-media/press-releases/otis_launches_otis_one_iot_service_solution_worlds_largest_elevator_service_network.aspx
6 eWeek, “Otis Elevator Hikes Role of IoT in Customer Service, Satisfaction”, July 2018, http://www.eweek.com/networking/otis-elevator-hikes-role-of-iot-in-customer-service-satisfaction
7 Schindler, “Schindler and GE Enter Strategic Partnership to Power Digital Urban Mobility”, July 2016, https://www.schindler.com/com/internet/en/media/press-releases-english/press-releases-2016/schindler-and-ge-enter-strategic-partnership-to-power-digital-ur.html#button
8 Schindler, “Schindler and Huawei sign global agreement on cooperation for IoT connectivity”, September 2016, https://www.schindler.com/com/internet/en/media/press-releases-english/press-releases-2016/schindler-and-huawei-sign-global-agreement-on-cooperation-for-io.html#button
9 Hitachi, “Implementation and Future Plans for Global IoT Maintenance System”, 2018 http://www.hitachi.com/rev/archive/2018/r2018_04/02b04/index.html
10 Hitachi, “Integrating MathWorks MATLAB and Microsoft Dynamics 365 Using Hitachi’s Lumada IoT platform”, June 2018, http://www.hitachi.ca/about/press/details/06202018.html

 

Stakeholders

 

Technical Safety BC

Asset Owners

Maintenance Contractors

Benefits of IoT-connected elevators

  1. Increased inspection coverage across all elevators
  2. Ability to identify and flag at-risk elevators for inspection with higher accuracy
  3. Increased efficiency of resource allocation
  4. Opportunity for internal teams to understand and utilize IoT systems and IoT data
  1. Ability to monitor the health of elevators in real-time or close-to-real-time
  2. Reduced long-term maintenance costs
  3. Potential development of an early warning system
  4. Ability to verify installation and maintenance work
  1. Increased ability for predictive maintenance
  2. Reduced time and labour cost due to efficient problem detection

Challenges

  1. Optimization of fault detection algorithms to be sufficiently accurate and reliable
  2. Training and/or recruitment of highly qualified personnel to install and maintain the IoT system.
  3. Development of new standard operating procedures (SOPs) and response measures for an IoT system
  1. Increased short-term costs from the installation and setup of the IoT system.
  2. Minor long-term costs for maintenance of the system
  3. Concerns from other building stakeholders over data privacy and security
  1. Increased training time associated with accessing and understanding the IoT data

Table 2. Benefits and challenges of IoT elevators

Project goals: Identifying and predicting abnormal elevator performance using IoT

The objective of this project was to design an IoT system that could collect raw data from elevators which can then be used to predict which elevators were potentially at-risk of technical issues. Developing an IoT system generally requires three main components (Figure 3):

  1. Sensors coupled to the specific object/infrastructure to collection of raw data
  2. Broadcasting device to transmit the data via the internet or other communication technologies
  3. Analytical tools to transform the raw data into useful information and insights
The three minimal components of an IoT system

Figure 3. The three minimal components of an IoT system

With respect to the current project, the first component involved collecting the raw elevator performance data via accelerometer-sensors attached to the elevators over the span of a few weeks to a few months. The second component involving ‘data transmittance’ was excluded from this evaluation as there was a lack of time and resources for proper setup. Hence, instead of transmitting, the data from the elevator IoT devices were stored locally within each device and retrieved at a later date for analysis. The third component involved the analysis of data using machine-learning algorithms to identify and predict mechanical problems, ride quality issues, and even estimate the remaining lifespan of elevators and their critical components. Therefore, the overall objective of this project is twofold:

  1. Identify, evaluate and test suitable sensor devices that are capable of collecting data from elevators that can be used to identify potential technical performance issues
  2. Identify, evaluate and test suitable data-mining and machine-learning methods that are able to automatically detect abnormal elevator behavior and alert building stakeholders.

Project methodology: A three-phase IoT process

To achieve the objectives mentioned in the previous section, the project work was divided into three phases:

Phase 1 focused on understanding the indicators of elevator performance. This phase involved clarifying how elevators function, the characteristics of elevators that exist in BC, and the mechanical and safety issues that are known to Technical Safety BC and across the elevator industry. Having identified the most common elevator issues, all possible data that could be feasibly retrieved from an elevator was then described. Acceleration data was determined to be the most feasible and essential data to collect. Lastly, eleven device vendors, academic groups, and elevator consultants were contacted to source and evaluate a suitable consumer-ready sensor device that could capture the necessary acceleration data.

Phase 2 focused on collecting preliminary data from elevators. The selected sensor device from phase 1 was tested in 20 elevators across the lower mainland in locations managed by an anonymous mix of municipalities, hospitals and building management companies.

Phase 3 began with the manual processing and exploration of data to identify visually-obvious patterns. Having explored the data, the next step was to automate the detection of anomalies in elevator behavior that would indicate a technical issue. Using anomaly detection algorithms and machine-learning methods, a number of anomalies were able to be collected from both the simulated elevator data as well as authentic elevator use. There are, however, further opportunities available to enhance the existing detection pipeline before it can be used to advise day-to-day operations.

For further details of each phase, please refer to Appendices 1, 2 and 3.

Project results: A working IoT data pipeline

The following sections provide a high-level overview of the key project results. For a more detailed explanation of the results and the nature of the work performed in each, please refer to Appendices 1, 2 and 3.

Phase 1

In phase 1, the criteria for selecting test elevators, as well as any potential anomalies identified as being probable leading indicators of performance issues and the type of data to be collected was successfully determined. Additionally, two consumer-ready accelerometer devices that met the specific requirements for capturing elevator acceleration data in public buildings were identified (Table 3).

Questions

Answers

What types of elevators should be investigated?

Older elevators (30+ years)

Electrical and hydraulic elevators

Within the lower mainland

Preferably with known technical issues

What are the common technical issues?

Leveling mechanism

Door lock mechanisms

Elevator brakes

Elevator motor

Elevator car alignment

What elevator data can be captured by a sensor and is potentially useful?

Elevator acceleration – selected feature

Vibration

Humidity

Temperature

Sound of moving parts

Electrical current

What device can capture the selected data type?

A small, lightweight, 3-axis accelerometer-integrated device with sufficient battery life and data storage. Device must also be FCC or CE-certified.

Final device choice:

Axivity AX3, designed by Open Lab of Newcastle University

Table 3. Results of phase 1

Phase 2

Phase 2 of the project was made possible through the combined efforts of multiple departments within Technical Safety BC by collaborating with various building stakeholders (municipal, hospital and commercial) to allow for the access and sensor installation within their elevators. This led to a successful installation of over 20 sensor devices in elevators across the lower mainland for data generation. Phase 2 also resulted in the identification of device settings optimized for longer durations of recording while still maintaining sufficient data quality (Table 4).

Questions

Answers

Which elevators in BC were installed with the sensor device?

Municipal buildings,

Hospitals,

Commercial buildings

How long were they installed for?

Preliminary testing: 10 minutes – 1 day

Pilot project: 2- 4 weeks

Where were the devices attached to the elevator?

Sensors were attached to the ceiling of the elevators or the walls where they could not be removed by bystanders

What device settings were used for recording?

Frequency of recording:

Preliminary testing: 12.5Hz, 25Hz, 100Hz

Pilot project: 25Hz

Range of measurement: ±2G

Table 4. Results of phase 2

Phase 3

Exploratory analysis of the data in phase 3 indicated that the elevator acceleration data collected was of both high quality (high signal to noise ratio) and volume (millions of timepoints). The data cleaning steps were successful in removing some of the noise in the data. In addition, visual inspection of the data showed that elevator trips form recognizable acceleration patterns.

The Technical Safety BC Research and Analytics team in collaboration with research partners was able to apply a number of machine-learning methods to automate the detection of anomalous behavior from the collected elevator data (Table 5). Qualitative evaluation of the anomaly detection results indicated that the methods were successful in detecting anomalies. However, due to the current lack of secondary data for verification (i.e. a labelled dataset), the accuracy of the method for broader scope of work outside simulation was not possible. Based on the limited set of simulated data, the methodologies explored were able to detect the labelled abnormal patterns at 100 percent.

Questions

Answers

What volume of data was collected per elevator?

At 25Hz (25 measurements/second) for 2 weeks, there is data for approximately 30 million time points per elevator.

What features were collected?

Device ID, timestamp, x, y and z-axis acceleration value

What data pre-processing was performed?

“z-normalization” centers the mean at 0 and sets the standard deviation to 1, enabling comparison of values and patterns between different elevators

“low-pass filter” removes noise at higher frequencies, creating a smoother time series graph and reducing errors. However, applying it too strongly can also unintentionally remove real anomalies that should be captured.

Were there obvious patterns in the data?

Visual inspection of collected data shows that elevator trips form human-recognizable patterns of acceleration over time.

What machine learning methods were attempted to automate anomaly detection?

Hidden Markov Model (HMM)

Self-organizing Maps

SAX+-VSM*

Standard Anomaly Detection

Luminol: Bitmap Detector

Wavelet

What were the results of the machine-learning (ML) methods? What were the limitations of these method?

All methods were successful at extracting anomalies from elevator acceleration data.

However, there was no reliable secondary data to verify that the anomalies detected were caused by actual technical issues. Thus, the accuracy of each ML method could not be validated.

What are the recommendations for improving the value of the machine-learning methods?

Anomalies detected by machine-learning algorithms need to be verified by a secondary data source.

Verification data could include incident reporting by passengers, managers and technicians and/or other sensors that capture passenger experience or elevator performance (utilizing a measurement other than acceleration). Verification data would validate detected anomalies and enable measurement of the accuracy of the methods used.

What other data analyses were performed?

An algorithm to extract elevator trips from raw acceleration data. Preliminary results show that between 80 – 100% of elevator trips and anomalies could be extracted using this algorithm. Further work needs to be done to complete the algorithm and validate it across various types of elevators.

Analyzing and predicting elevator leveling issues. They were able to predict elevator leveling offsets to a 10 – 15 cm margin of error. The team suggests that further improvements to the data collection and analysis process could improve the accuracy to an error margin of only 5 cm.

Signal processing techniques such as Wavelet was performed on simulated leveling data. The analysis was able to apply Wavelet transformation to detect and classify various anomalous and normal elevator travel patterns.

Table 5. Results of phase 3
+ SAX: Symbolic Approximate Aggregation
* SAX-VSM: SAX-based Vector Space Model

Analysis Summary

The data collected from the sensors was of sufficient quality and reliability for advanced machine learning analysis in order to differentiate abnormal travel patterns from normal travel patterns. An anomaly is something that can occur in the sequence of events in acceleration (e.g. a normal travel ending with an abrupt quick stop), or in the magnitude of speed (e.g. over-speeding), or in the amount of displacement (e.g. leveling issues). The data collected had the adequate level of resolution at the sampling frequency of 25 Hz and also had high signal to noise ratios, which indicates when elevators travel normally, the corresponding data signature is very distinct.

Figure 4 displays examples of normal patterns. Figure 5 displays examples of abnormal patterns. The algorithms explored show promising results in detecting anomaly and have great overlap in the overall detection of anomaly, effectively capturing the distinct data signatures.

More details on the AI and machine learning algorithms used in detection of anomaly for elevating data signature are available in Appendix: A3.4 Machine-learning and anomaly detection.

Examples of normal travel patterns in elevators

Figure 4. Examples of normal travel patterns in elevators

Examples of abnormal patterns detected in elevators

Figure 5. Examples of abnormal patterns detected in elevators

Next steps: Expanding the IoT project

We identified a number of observations and limitations that could be used to improve the outcome of potential future IoT projects.

The main observations of interest were the following:

  1. Building stakeholders are interested and comfortable in monitoring elevator performance and safety under the following conditions:
    1. The installed sensors are safe and certified
    2. The installed sensors do not capture audio, video or pictures of passengers
    3. The data collected remains anonymous unless permission is granted by the building stakeholder to release the data
    4. Sufficient notice, information and instructions are provided to building stakeholders and their employees prior, during, and after installations
  2. Existing elevators can be converted into IoT-like elevators with relatively low-cost and easy-to-install sensors
  3. Data collected from consumer-ready sensors is of sufficient quality and reliability for advanced machine learning analysis

The main limitation that was observed during this assessment was the following:

  1. The reporting of elevator incidents and performance issues by passengers and technicians is currently sparse and unreliable. Reliable data regarding incidents and performance issues is needed to verify the link between detected anomalies and the actual elevator incidents and to validate machine-learning models.

Based on these observations, the main opportunities for next steps in advancing the IoT research are the following:

  1. With the cooperation and support of building owners and managers, Technical Safety BC can expand and scale up the current project in order to build a robust IoT network that can accurately detect and predict elevator patterns and anomalies across the province.
  2. The insight and research method from this project can be shared with all stakeholders in the elevating industry who are interested in analyzing and improving elevator safety. As elevators affect all cities globally, the insights here could also be further explored and used by other jurisdictions and the global industry to inform elevator IoT guidelines and policies.
  3. The reporting of elevator incidents and performance issues could be encouraged by standardizing and simplifying the process in which incidents are reported. For example, Technical Safety BC is planning to release an elevating performance feedback app in 2019. This crowd sourcing approach will allow Technical Safety BC to more effectively gather information that can validate machine learning models.

In light of the promising results in this pilot project and its synergy with Technical Safety BC’s efforts to become a leader in data-driven safety, the main recommendation is to expand this project with particular focus on exploring various IoT infrastructure pipelines (sensor fusion, cloud platform services, as well as different options for wireless gateway) and improving the accuracy of machine learning techniques explored by the Research and Analytics team. The experience gained by the cross-departmental teams within Technical Safety BC and their exposure to IoT technologies, experts and partners has provided Technical Safety BC with an advantage and a strong foundation to become an industry leader in this new age of IoT-connected technical equipment.

APPENDIX

Appendix 1: Phase 1: Research

A1.1 Identifying suitable elevator models to diagnose

With over 25,000 elevators in BC, the first requirement was to reduce the numbers for practical testing purposes. Four criteria were established:

  1. Elevators within the lower mainland
  2. Preference for elevators with known issues
  3. Preference for older elevators (30+ years)
  4. An equal number of hydraulic and electric elevators

Ultimately, the key deciding factor was whether permission was obtainable from building owners and managers to install the proposed devices.

A1.2 Identify existing elevator data

Before collecting new information, the data that was already within Technical Safety BC’s database, was identified. This included:

  1. Age of elevator
  2. Model of elevator
  3. Elevator breakdowns and incidents in BC
  4. Known causes of elevator breakdowns and incidents in BC

This information was useful in identifying which elevators were more likely to have technical issues. Older elevators are more likely to have accumulated wear-and-tear damage while certain models of elevators are known to be prone to technical issues1. For example, single-speed elevators move at a constant speed throughout the entire travel journey which prevents the elevator car from decelerating smoothly at the end of each trip2. This causes an issue called “leveling”, where the elevator car floor is not level with the building floor.

A1.3 Identifying the minimal number of data points needed to diagnose an issue

While the age and model of an elevator are useful in predicting the general performance of a generation of elevators, these do not allow for risk analysis and predictions on a per-elevator basis. Incident history and investigation data is more specific but covers very few elevators and is unreliable due to under-reporting of elevator issues by passengers, building managers, and technicians.

1 Technical Safety BC, “Case Study: Elevator Leveling Incidents (2009 – 2015)”
2 Technical Standards and Safety Authority, “Requirement for Passenger Elevators with Single Speed Controls, Director’s Order 269/17”,

To complement the data that Technical Safety BC currently has on BC elevators, the following list of potentially useful data features that can be captured by sensors or are currently being collected by existing IoT elevators was compiled:

  1. Number of times doors open/closed
  2. Average duration the elevator door stayed open
  3. Acceleration of elevator
  4. Speed of elevator
  5. Distance traveled by elevator
  6. Frequency of each floor destination
  7. Motor current and electrical consumption
  8. Outlier vibrations
  9. Number of times the elevator was overloaded or near maximum capacity
  10. Number of times the elevator door was held open by a user
  11. Number of emergency button presses
  12. Sound signatures in the machine room, hydraulic pump or elevator motor (depending on elevator type)
  13. Temperature of moving parts
  14. Humidity of elevator shaft, machine room, motor

The decision on which data features to collect required consideration of a number of factors:

  1. The ease of obtaining a suitable sensor
  2. The cost of a sensor device
  3. The ease of collecting and storing the data generated by the sensor
  4. The ease of analyzing the sensor data
  5. The likelihood that the data contained anomalies that can be detected by data-mining and machine-learning algorithms
  6. The domain knowledge and analytical expertise available to the analysis team

Based on these factors, the decision was made to collect elevator acceleration data using accelerometers. The rationale included the following:

  1. Accelerometers are one of the most common, low-cost, low-energy sensors with a wide range of sensitivities and add-on functions.
  2. The data generated would consist only of a timestamp and a numeric acceleration value for each timepoint, allowing for smaller data storage sizes.
  3. Accelerometer data is in the format of a time series of single values and relatively easy to process and understand.
  4. There is a significant body of research and machine-learning tools, both academic and industrial, focused on accelerometer-like data (i.e. waveform, time-series data) analysis especially in applications such as drones, automotive, medicine (cardiograms), and music.
  5. Based on 3 and 4, there was sufficient expertise in the Technical Safety BC Research and Analytics team to learn and apply machine learning methods to analyze accelerometer data from elevators.

A1.4 Re-confirming the value of accelerometer data with a safety officer

The decision to collect accelerometer data, required confirmation that elevator acceleration values could be linked to true technical issues. An interview with a Technical Safety BC elevating device inspector highlighted five common elevator mechanisms that can be monitored for elevator performance issues:

  1. Leveling mechanism
  2. Door lock mechanism
  3. Elevator brakes
  4. Elevator motor
  5. Elevator alignment

Of the five common mechanisms, acceleration over time data was chosen to identify problems with:

  1. Elevator brakes
    - slower deceleration and/or acceleration patterns indicative of slippage suggest worn-out brakes
  2. Elevator motor
     - slower acceleration or noisy accelerations suggest worn-out motor or traction grooves
    - extremely high acceleration/deceleration values suggest a mis-calibrated motor or incorrect speed settings
  3. Elevator alignment
    - significant vertical/lateral vibrations and/or acceleration suggest worn-out guide rollers/sliding shoes or bumpy guide rails

In summary, it was determined that acceleration data would be both feasible for collection (Appendix A1.3) and valuable in identifying and predicting technical issues (Appendix A1.4).

A1.5 Considerations for device parameters and instalment

Acceleration data is typically collected using a sensor called an accelerometer. A vast variety of accelerometers exists on the online marketplace and are used by academic, governmental, and commercial groups. Accelerometers are used for various purposes including measuring the speed and direction of flying drones, triggering airbags during a car crash, and capturing the magnitude of earthquakes.

Accelerometers are typically sold as individual sensor components instead of a complete consumer device; the latter includes a battery pack, a flash memory for storage, a motherboard for mounting and connecting all the components and a port for uploading/downloading information. Another component that can often be found on a complete consumer device is a wireless transmittance module for remote connections to the device.

Due to the time and technical constraints of this project, a complete consumer device was selected as the optimal piece of equipment. In the process, over 10 accelerometer products and vendors were evaluated based on the following requirements:

  1. Measures acceleration
  2. Measurement range of ± 2G (g-force)
  3.  Battery life (3-6 months)
  4.  Cost (under $100 per piece)
  5.  Minimal modifications required (i.e. adding a battery park, custom software, etc.)

Four feasible options were identified (Figure 6):

  1. Axivity’s AX3 data logger
  2. MicroDAQ’s Pendant G data logger
  3. axSense’s AE-3x acceleration data logger
  4. axSense’s AE-3x1-L acceleration data logger with sensor fusion

The Axivity AX3 device was selected for testing as the device specifications met the requirements for the data collection. The axSense AE-3x1-L device was also tested on a subset of elevators for comparative analyses and to explore the potential of advanced sensor features.

Axivity AX3

Option 1: Axivity AX3 Supplier: Axivity, UK Link: https://axivity.com/product/ax3 Cost: ~$99 (Euro) Software: Free, open-source Sampling rate: 12.5Hz to 3200Hz Accelerometer range: ±2,4,8,16G Accelerometer resolution: Up to 13 bits Memory: 512MB (Flash, non-volatile) Battery life: 30 days @ 12.5Hz; 14 days @ 100Hz

MicroDAQ – Pendant G data logger

Option 2: MicroDAQ – Pendant G data logger Supplier: MicroDAQ, USA Link: https://www.microdaq.com/onset-hobo-pendant-acceleration-data-logger.php Useful specs: Cost: ~$80 (USD) Software: Separate purchase ($99) Sampling rate: Up to 100Hz Accelerometer range: ±3G Accelerometer accuracy: 2.5% FS Memory: 64K measurements (21.8K per axis) Battery life: 1 year @ typical usage, 7 days @ 100Hz continuous sampling


Figure 6a. Two of the top four device options for measuring acceleration data

Axsense Model AE-3x Acceleration Data Logger And Model AE-3x1-L With Sensor Fusion

Option 3: axSense acceleration data logger (model AE-3x):
Supplier: axSense, BC, Canada Link: https://www.axsense-technologies.com Cost: ~$300 (CAD) Software: None needed (CSV data format with time and 3-axis acceleration) Accel. sampling rate: 1Hz to 6600Hz Accel. resolution: 16 bits Accel. accuracy: 1% FS Memory: MicroSD Card (4GB provided, ~100M measurements; expandable to 64GB) Battery life: 45 days @ 25Hz, >90 days adaptive usage Additional options: Wireless data transmission Extended battery life Cloud connectivity via axSense data hub

Option 4: axSense acceleration data logger with sensor fusion (model AE-3x1-L):
Supplier: axSense, BC, Canada Link: https://axsense-technologies.com Cost: ~$800 (CAD) Software: None needed (CSV data format with time, 3-axis acceleration, height information) Accel. sampling rate: 1Hz to 6600Hz Accel. resolution: 16 bits Accel. accuracy: 1% FS Height sampling rate: 1Hz to 25Hz Height resolution: < 10cm (5-7cm typical) Memory: MicroSD Card (4GB provided, ~80M measurements; expandable to 64GB) Battery life: 45 days @ 25Hz, >90 days adaptive usage Additional options: Wireless data transmission Extended battery life Cloud connectivity via axSense data hub

Figure 6b. Two of the top four device options for measuring acceleration data

Other considered options

PASCO smart cart (for educator use, wireless 30m)
https://www.pasco.com/prodCompare/smart-cart/index.cfm

IOlabs (for educator use, wireless 100ft)
http://www.iolab.science/

Bosch Sensortec (sensor only, no device)
https://www.bosch-sensortec.com/bst/products/motion/accelerometers/overview_accelerometers

Mcubemems (sensor only, no device)
http://www.mcubemems.com/technology/sensor-fusion/

MSR electronics (powerful devices, battery extension available, very expensive >$500 pp.)
https://www.microdaq.com/manufacturers/msr-electronics.php

Sierra Wireless (wireless gateway only, no sensors)
https://www.sierrawireless.com/

Phase IV engineering (wireless sensor for temperature and pressure only, no accelerometer)
https://www.phaseivengr.com/wireless-technologies/about-rfid-wireless-data-loggers/

Safetach (elevator movement monitoring device, no long-term monitoring)
http://maxtonvalve.com/products/index.php?route=product/product&product_id=105

axSense (SFU spin-off company, high-performance accelerometer, too expensive)
https://www.axsense-technologies.com/

Platforms for purchasing sensor and IoT devices

Digikey
https://www.digikey.ca/

AliExpress
https://www.aliexpress.com/wholesale?catId=0&initiative_id=SB_20180602105124&SearchText=data+logger+

A1.6 Pilot Implementation: Device setup and retrieval

Attaching the Axivity AX3 device required an accessible, stable surface (to reduce noise) but one that is still out-of-sight and reach of an average person. Available options included the elevator ceiling, the inside of the fireman’s locked box in the elevator car, and the roof of the elevator car. Early attempts to install the device determined that it was easiest to tape the device to the ceiling of the elevator in an inconspicuous corner or edge. For easy installation and retrieval, the device was attached to the surface using double-sided tape.

Two retrieval schedules were proposed during this project:

  1. For short-term data collection, the device was installed in the elevator for a period of 2 – 4 weeks and then retrieved for data extraction and recharging.
  2. For long-term monitoring, the recommendation for retrieving, downloading, and recharging is every 3 months for 1 year+. This allows Technical Safety BC to cover a longer time range (1 year+) and save safety officer time (4 retrievals a year). Although the device battery does drain out within a month, it is assumed that the periodic samples of data would be sufficient to detect long-term declines in elevator performance.

 A1.7 Recommended device settings

Preliminary data collected from elevators in the Technical Safety BC building demonstrated that both the 25Hz and 100Hz sampling rates were able to detect the stationary and traveling patterns of elevators. In an elevator installed simultaneously with two sensors, patterns from data captured at 25Hz matched those captured at 100Hz. Recording frequencies lower than 25Hz were available but were not tested in this project. Therefore, the recommendation is to use 25Hz with the lower sampling rate to reduce battery and memory consumption. As for acceleration sensitivity, the recommendation is to use ±2G for the sensitivity as the maximum and minimum values for g-force appears to be well within ±2G.

Appendix 2: Phase 2: Device testing and preliminary data collection

A2.1 Installation of devices in BC elevators

The placement of sensors or other devices in a public infrastructure such as elevators is often a challenge as there are valid concerns about privacy, security, and safety. In this project, Technical Safety BC initiated discussions with local asset owners to obtain permissions for installing the Axivity AX3 sensor device in local elevators.

Their discussions have yielded a list of conditions that are acceptable to asset owners for installing a sensor device in their elevators. Below are a list of questions and concerns from building stakeholders and the answers that were deemed satisfactory (Table 6).

Questions / Concerns from building stakeholders

Answers from Technical Safety BC (modified from original answers)

Is there a preference or limitation for the types of elevators?

Older, issue-ridden elevators are preferred. A mixture of hydraulic and electrical elevators would also be preferable for optimal testing.

How many elevators are tested?

10 – 20 elevators in this pilot phase

Will the device monitor, record or transmit conversations of passengers, take photos or videos of passengers within elevator cars?

No. The device will only capture the acceleration and deceleration of elevator cars, estimate travel time and monitor any lateral (side-to-side) vibrations.

What is the safety of this device?

The device used to measure elevator performance is approximately the size of a key fob, containing 3 accelerometers which are used to measure the movement and vibrations of an elevator. The device is lightweight (11g), runs on a single coin cell battery, is FCC and CE-certified and will be secured out of reach and sight of regular building occupants. Proposed location for the device is the ceiling of the elevator.

Will the device be transmitting the collected data?

The device will not transmit. Instead, the data will be logged in memory until the device is retrieved later.

If the collected data will be shared with others (beyond Technical Safety BC), what level of privacy will be protected (such as building locations, name of building owners, etc.)?

Building owners/managers will have access to their own data and data from other buildings on an anonymized basis.

Individual building or property managers will want to know the time (of installation/placement), location, and duration (of the research) of each of these devices. 

Installation locations will be provided ahead of time. Building managers or representatives will be walked through the installation process. Installation takes, on average, 5 to 10 minutes. Data will be collected for up to a month, at which point, a Technical Safety BC personnel will retrieve the device with your consent and prior knowledge.

Table 6. Questions and concerns from building stakeholders and their corresponding answers from Technical Safety BC

As expected, the main concerns around installing a sensor device are passenger privacy and safety. Additionally, building stakeholders were concerned about the security and safety of transmitted data. Lastly, the installation process is also an investment of time and human resources for the building stakeholder.

A2.2 List of installation locations (SUPPRESSED)

Building types that were included in the elevator monitoring pilot project:

  1. Mall elevators
  2. Hospital elevators
  3. Government building elevators
  4. Other commercial buildings

Appendix 3: Phase 3: Analyzing the data using machine learning

A3.1 Data format

Data collected from the Axivity AX3 was formatted as a csv file with rows representing recordings at each timepoint. 5 columns were included in every file:

  1. Device ID (integer)
  2. Timestamp (yyyy-MM-dd HH:mm:ss.SSS)
  3. X-axis acceleration value (g-force = 9.8m/s2)
  4. Y-axis acceleration value (g-force = 9.8m/s2)
  5. Z-axis acceleration value (g-force = 9.8m/s2)

Because the device was attached to the ceiling of the elevator, the Z-axis values represented the movement of the elevator when it traveled up and down. Hence, the Z-axis and the timestamp columns were primarily selected for the analysis and machine learning portion of this report. The X and Y-axis data were not analyzed in this pilot project although X and Y-axes data could potentially be used to detect anomalous lateral (side-to-side) vibrations.

A3.2 Obvious visual patterns in the data

An elevator is typically in one of two states; stationary or traveling. Visual exploration of the elevator’s z-axis data shows that there are obvious humanly-recognizable patterns in the acceleration values when the elevator is traveling. Figure 7 demonstrates three examples of elevator trips: a) elevator traveling up 1 floor, b) elevator traveling down 1 floor, c) elevator traveling down 3 floors.

Visually-obvious patterns during elevator trips

Figure 7. Visually-obvious patterns during elevator trips

The plots above show acceleration over time when an elevator is traveling up 1 floor, b) traveling down 1 floor and c) traveling down 3 floors

Within the traveling state are three main phases:

  1. acceleration phase
  2. constant velocity phase and
  3. deceleration phase.

These three phases are most obvious in longer trips (2 or more floors traveled). For example, figure 8 shows a 3-floor elevator trip divided into the three phases. These patterns allow us to break apart a single elevator trip into smaller phases for analysis.

Three phases of a 3-floor elevator trip

Figure 8. Three phases of a 3-floor elevator trip

Favorably, the existence of obvious travel patterns makes it probable that travel patterns can be automatically detected, extracted, and analyzed via algorithmic methods.

A3.3 Data quality and normalization

A better understanding of the data structure enables more determinant judgement whether a change in acceleration values is a real signal or noise. The next step is to standardize the data and to remove as much noise as possible while retaining signal.

A common first step is to perform a Z-normalization (Figure 9) which is composed of two simple steps:

  1. First, subtract all values in the dataset by the average (mean) value – centers the data at 0
  2. Secondly, divide all values by the standard deviation – sets the standard deviation at 1
Demonstration of a Z-normalization

Figure 9. Demonstration of a Z-normalization

With time-series and waveform-like data, another common step is to remove certain frequencies within the data, which are likely noise, to smoothen the data. In the case of the acceleration over time data, a low-pass filter to remove high-frequency noise can be utilized. A low-pass filter is a type of bandpass-filter that allows only frequencies below a certain threshold to “pass” through by attenuating higher frequencies. Controlling the threshold of a low-pass filter determines the level of noise that is removed. A threshold that is too high will not remove much noise, however a threshold that is too low might end up removing true signals and anomalies in the data.

Figure 10, below, shows an example of a low-pass filter applied to raw accelerometer data from an elevator in the pilot project. At a 5Hz filter threshold, a lot of the noise has been removed while maintaining most of the shape of the time-series graph. As demonstrated at 1Hz and 0.1Hz, the threshold is too low and “over-filtered”, removing valuable patterns in the time series.

Effects of different thresholds

Figure 10. Effects of different thresholds of a low-pass filter on the shape of a time-series graph of elevator acceleration values

A3.4 Machine-learning and anomaly detection

The next four subsections describe the process and results of data exploration and machine learning performed by four analytical teams to detect and measure anomalies from simulated and actual elevator acceleration data.

  1. A3.4.1 Elevator travel feature extraction
  2. A3.4.2 Machine-learning approaches – Time Domain
  3. A3.4.3 Machine-learning approaches – Frequency Domain
  4. A3.4.4 Elevator leveling detection
A3.4.1 Elevator-specific feature extraction method
A3.4.1.1 Designing a SAX-based elevator-specific feature extraction algorithm

Given a raw dataset of elevator acceleration data, the task was to experiment with a model/algorithm that could detect anomalies within the data. In machine learning, there are generally two approaches to this type of exploratory analysis, supervised learning (where data is labelled) and unsupervised learning (where the data has no labels and the algorithm often attempts to assign its own label). Both learning methods are explored by Technical Safety BC’s Research Analytics team in section A3.4.2.

In this section, the focus was on extracting individual elevator trips using an assumption-heavy algorithm that was invented during the project. This algorithm works specifically, and possibly exclusively, for elevator acceleration data. Below are a few core assumptions that make this algorithm possible:

  1. Elevator acceleration is mainly centred around zero or some constant offset value
  1. Elevator accelerations must always be followed by elevator decelerations and vice versa (i.e., What goes up must come down)
  1. For each elevator trip, the shape and duration of the accelerating, constant velocity, and decelerating phases are mostly consistent

With these assumptions, the following logical arguments can be made:

  1. Since zero (or some constant offset value) is when the elevator is mostly stationary (except for states of constant velocity), extraction of individual elevator trips is possible by splitting the raw time series by long sections of zeroes (or constant offset values).
  1. Since elevator trips have an acceleration/deceleration phase, it is feasible to pair disjointed acceleration and deceleration phases (from the initial split) together as long as the duration between the two are within reasonable limits.
  1. Since the general shape of elevator travel phases stay consistent, breaking down each elevator trip into travel phases for subsequent analysis can be achieved. Also, similar elevator trips can be clustered together by the duration of each phase and the duration of the total trip.

Ultimately, accurately extracting individual trips allows precise measurement of the duration of each elevator trip and the duration of the travel phases. This duration of travel can then be used as a backbone for clustering similar elevator trips and subsequent anomaly detection.

A3.4.1.2 Steps for feature extraction

This feature extraction method, called SAX-EFX (SAX-based Elevator Feature eXtraction), follows a few simple steps. As of the time-of-writing, SAX-EFX has not been completed and validated. The following two lists show the algorithmic steps that have and have not yet been implemented as code:

Already implemented:

  1. Z-normalize the data
  2. Apply Piecewise Approximate Aggregation (PAA) on the data
  3. Convert PAA values into a sequence alphabets using SAX
  4. Identify top 3 most frequent alphabets
  5. Identify “flat” areas (near zero) using regular expressions for the top 3 most frequent alphabets. 90% of a subsequence needs to consist of the top 1 alphabet to be called a “flat” area
  6. Split the entire time series by the flat areas
  7. The non-flat areas are labelled as travel patterns

Not yet implemented:

  1. Pair disjointed acceleration and deceleration phases together
  2. Trim flat areas on both ends of each travel pattern
  3. Cluster similar travel patterns
  4. Label acceleration patterns with paired acceleration/deceleration phases as complete trips
  5. Label remaining acceleration patterns as anomalies for manual review
  6. Break down complete trips into acceleration and deceleration phases
  7. Calculate duration of travel phases and duration of overall trip
A3.4.1.3 Sample output of the implemented portion of the SAX-EFX algorithm

As of the time-of-writing, SAX-EFX has not been completed and validated. However, the output of the algorithm at step 7 is extremely promising, as SAX-EFX is able to label and extract 87.5% (14/16) of all visually-obvious patterns within the first data snippet shown below (Figure 11 and 12). In another data random snippet, the algorithm is able to extract up to 100% of all visually-obvious patterns (Figure 13) where travel phases are clearly visible.

Visually-obvious patterns detected

Figure 11. Visually-obvious patterns detected and highlighted by SAX-EFX

Visually-obvious patterns from figure 11

Figure 12. Visually-obvious patterns from figure 11 extracted by SAX-EFX into individual plots

Visually-obvious patterns extracted by SAX-EFX

Figure 13. Visually-obvious patterns extracted by SAX-EFX from another data snippet

A3.4.2 Machine-learning methods in Time Domain
A3.4.2.1 Hidden Markov Model

Hidden Markov models 3 (HMM) are powerful finite state machines that characterize a system using its observable parameters. HMMs are widely used for sequence modelling and sequence anomaly detection. The assumption behind HMM is that a given time series O = O1 . . .On is the indirect observation of an underlying (hidden) time series Q = Q1 . . .Qn, where the process creating the hidden time series is Markovian, even though the observe process creating the original time series may not be so. Thus, the normal time series can be modeled using a HMM, while the anomalous time series cannot be.

Given a training series, we can build a single HMM (λ) model, which consists of parameters that describe the normal data, such as the initial state distribution and state transition probabilities . Every training time series can have a specific model or there can be a unified model for all the training time series. A HMM based anomaly detection technique operates as follows:

  1. Given a training time series, Otrain = O1 . . .On, it is considered as a sequence of indirect observations of the HMM model. There exists a well-established procedure which is able to determine the HMM parameters by maximizing the probability P(Otrain|λ) using a technique called the Baum-Welch re-estimation procedure.
  1. In the testing stage, given an unknown time series Otest = O′1 . . .O′n′, the probability P(Otest|λ) is computed using the trained model. Among the test time series, we can say that the anomalies are those which have the minimum value of P(Otest|λ).

In the elevator pilot, the acceleration patterns for a normal elevator travel are consistent and fits the assumption that there exists a hidden process which is markovian and generates the normal time series. Figure 14 shows a normal trip for an elevator going up from the parking floor to the 3rd floor. When the elevator starts moving up, it first accelerates, then deaccelerates to remain a uniform speed. When it is about to reach a certain floor, it deaccelerates first to slow down the car and the acceleration with gravity recovers to 1g. When the elevator starts moving down, it is a similar mechanism, but the acceleration is in an opposite direction.

However, abnormal elevator travels do not follow a similar pattern (Figure 15). When we apply HMM to the abnormal data, the assumption is that there are three hidden states, the acceleration climbing up to the high peak, the acceleration dropping down to the low peak and the acceleration remaining flat. For normal elevator travels, the acceleration will transition among these three states with a certain probability. Training the HMM model on the normal sequences, provides the highest log likelihood score for the training set. When the model is then used on test sets which contain both normal sub-sequences and abnormal sub-sequences, high log likelihood scores from the model indicate that a subsequence is normal while low scores indicate that a subsequence is abnormal. A threshold is set to classify subsequences into normal or abnormal.

3 Lawrence R. Rabiner , “A tutorial on hidden Markov models and selected applications in speech recognition”, Proceedings of the IEEE 77.2, pp. 257-286, 1989.

Elevator on a normal trip

Figure 14. Elevator on a normal trip going up from the parking floor to the 3rd floor

Anomalies in elevator travels
Anomalies in elevator travels

Figure 15. Two anomalies in elevator travels detected using a HMM-based algorithm

a) Elevator in a “Roll Away” scenario causing a leveling isue, b) Elevator in a simulated “Overspeeding” scenario

A3.4.2.2 Self-organizing maps

Self-organizing Maps 4 (SOM), are also called kohonen neural network. It is a popular unsupervised machine learning algorithm to discover the topological relationships within the training data. Given the IoT dataset, the SOM is trained best on the normal pattern data first. For each observation in the test data, k-nearest neighbor which is another semi-supervised learning approach is performed with respect to the SOM nodes. The mean distance to the nodes will be calculated. The greater the distance, the more likely that this observation is an anomaly. In the end, all the observations with high anomaly scores are ordered by time. Then the close-anomalized data points are combined and printed out as an anomaly pattern.

 

4 Tian, J., Azarian, M. H., Pecht, M. (2014). Anomaly detection using self-organizing maps-based K-nearest neighbour Algorithm. Proceedings of the European Conference of the Prognostics and Health Management Society.

Anomalies detected by the SOM algorithm

Figure 16. Anomalies detected by the SOM algorithm

A3.4.2.3 SAX-VSM

SAX-VSM 5,6,7 is based on two well-known techniques, Symbolic Aggregate Approximation which is a high-level symbolic representation of time series and Vector Space Model based on Terms Frequency Inverse Document Frequency weighting scheme.

SAX is the first symbolic representation for time series algorithm that allows for dimensionality reduction and indexing with a lower-bounding distance measure. It transforms numeric time series of a single input class into a combined collection of symbolic words, which we call the “bag of words”.

Then this algorithm transforms these collections (one collection for each of the input classes) into class-characteristic weight vectors using term frequency inverse document frequency weighting, which, in turn, are used in classification built upon Cosine similarity.

Our SAX-VSM model built on simulated data which includes the prior knowledge of the elevator normal movement patterns and abnormal movement patterns. To eliminate the variations between elevators, Wavelet and Z-norm were applied before modeling.

 

5 Senin, P., Malinchik, S., SAX-VSM: Interpretable Time Series Classification Using SAX and Vector Space Model, 2013 IEEE 13th International Conference on Data Mining

6 Senin, P., Lin, J., Wang, X., Oates, T., Gandhi, S., Boedihardjo, A.P., Chen, C., Frankenstein, S., Lerner, M., GrammarViz 2.0: a tool for grammar-based pattern discovery in time series, ECML/PKDD Conference, 2014.

7 G. Salton , A. Wong , C. S. Yang, A vector space model for automatic indexing, Communications of the ACM, v.18 n.11, p.613-620, Nov. 1975

Comparison between acceleration patterns

Figure 17. Comparison between the original (left), Z-normalized (middle), and Wavelet De-noised (right) acceleration patterns of an example anomaly detected using the SAX-VSM algorithm

A3.4.2.4 Standard Anomaly Detection

A standard approach to identifying outliers in numerical data is to classify data points that are 2 or more standard deviations away from the mean. A similar approach is used for this method where a sliding window moves over the data and searches for outliers to classify peaks and dips in the data (corresponding to elevator speeding up and slowing down). If several clusters of peaks and/or dips are found in a window of time, then that constitutes an anomaly and the timestamp is recorded.

After looking at some labelled data, it appeared that anomalous elevator behavior (overspeeding, overloaded, etc.) had several larger spikes and dips than under normal operations. Observing this led to this method of anomaly detection by looking for large spikes/dips in the data that occur close together.

This method starts out by moving a sliding window over all data points where the window has a size of 20 seconds (i.e. the model searches for anomalies within 20 seconds worth of data). Within these windows, the data points are checked to see if there is an outlier (more than 3 standard deviations away from the mean in this case). The mean and standard deviation are calculated using the entire dataset. Then, all outliers are tracked as peaks (being above the mean) or dips (being below the mean). If 5 peaks/dips are found within the window, then this is labelled a cluster. If there are 1 or more peak clusters and (optionally) any dip clusters present within the window, then that is labelled an anomaly.

Sample of anomalies in normal elevator data
Sample of anomalies in normal elevator data

Figure 18: Sample of anomalies in normal elevator data using Standard Anomaly Detection

A3.4.2.5 Luminol: Bitmap Detector by LinkedIn

Luminol 8 is a light-weight python library for time series data analysis that was originally designed to detect and identify the root cause of network latency issues at LinkedIn. The algorithmic basis behind Luminol is that it looks for values that deviate (is higher/lower than) consistently from a baseline. Bitmap 9 , one of the algorithms in Luminol, detects if a time series dataset contains any anomalies and gives the user back a time window in which the anomaly happened (Figure 19), a time stamp where the anomaly reaches its severity, and a score indicating the severity of the anomaly compared to others in the time series (Table 7).

 

8 Luminol, an open-source python library by LinkedIn, https://github.com/linkedin/luminol/blob/master/src/luminol/algorithms/anomaly_detector_algorithms/bitmap_detector.py

9 Assumption-Free Anomaly Detection in Time Series by Li et al, Proceedings of the 17th International Conference on Scientific and Statistical Database Management, 2005, pg.237-240, http://dl.acm.org/citation.cfm?id=1116877.1116907

Top anomaly with highest score detected

Figure 19. Top anomaly with highest score detected using Bitmap Detector Model in Luminol

Timestamp and Score of Top 10 Anomalies Detected

Table 7. Timestamp and Score of Top 10 Anomalies Detected

A3.4.2.6 Model comparison

Table 8 below describes the pros, cons and future work for each of the algorithms explored by the Technical Safety BC Research and Analytics team.

Model

Qualitative description

Hidden Markov Model

Pros:

  • Fast training and prediction. The training set can be small and contains only the normal subsequence. Training of the abnormal subsequence for which there are too many different patterns is not needed.
  • Accurate and efficient for anomaly subsequences with huge spikes
  • Suitable for real-time anomaly detection

Cons:

  • HMM cannot detect anomalies that fall in the range of hidden states.

Self-organizing Maps (SoM)

Pros:

  • Computational friendly, training process is fast and results are generated quickly
  • Various anomalies could be detected accurately
  • Can be customized for different elevators

Cons:

  • Thresholds to determine anomalies vary from data to data, which also requires manual determination
  • Some anomalies might be missing due to a larger threshold

SAX-VSM

Pros:

  • High Flexibility
  • Easy to understand

Cons:

  • Requires prior knowledge of anomaly patterns
  • Model need to be rebuilt for different elevators
  • Detection is too slow to run on large dataset

Standard Anomaly Detection

Pros:

  • Fast, no training necessary
  • Easy to understand
  • Low memory consumption

Cons:

  • Sensitive to outliers, leads to high number of false positives
  • Since outliers are classified based on mean and standard deviation, low values of each could lead to multiple false positives while high values of each could lead to false negatives
  • Needs a lot of fine tuning (number of standard deviations to count as a outlier, number of peak/dip to count as a cluster, window size, etc.) which may return varying levels of success for different elevators

Luminol: Bitmap Detector

Pros:

  • No predefined threshold is required for the time series. Each data point is assigned an anomaly score.

Cons:

  • It consumes a lot of resources to train the model.

Table 8. Machine Learning methods explored by Technical Safety BC for anomaly detection

A3.4.2.7 Summary

The current analysis showed that the application of various machine learning models in detecting elevator anomalies is feasible and can yield visually-convincing results. Further improvements can be made to these models by modifying the model parameters, improving data quality by increasing sampling rate, removing noise using alternative filters, and engaging the expertise of elevator domain experts in model building and acceleration data analysis.

Despite the convincing qualitative evidence, there is currently insufficient labeled data (i.e., incident reports, secondary sensor data confirming technical errors) to validate the models quantitatively. One alternative attempted is to manually label anomalies in the dataset by visual inspection. However, given the size of each dataset, this becomes a time-consuming task which also introduces human bias. Ultimately, a secondary labelled data source is needed to validate the anomalies detected by the machine-learning methods.

A3.4.4 Anomaly detection in Frequency Domain

The following sections summarize the literature review conducted, the approach taken, and the results of the approach from analyzing elevator acceleration data in the frequency domain.

A3.4.4.1 Literature Review

Anomaly detection, can be defined as identifying patterns that do not conform to the expected normal patterns10. This can be achieved either by deriving expectations of what is normal, or by putting lower-bounds on the level of abnormality of events. In any case, the problem of anomaly detection boils down to constructing proper distributions of events.

Classification methods along with hidden Markov models (HMM) have been studied in the literature to show great promise in dealing with high-dimensional time-series data11. The main idea of these studies is to classify different events using some unsupervised learning methods, and then apply an HMM as a post processing technique to identify anomalous behavior. The underlying assumption behind using an HMM is that the classified events form a Markov Chain, and that the probability of transition can be either calculated or studied from that past events12.

One of the clustering methods that is discussed in the literature is “fuzzy clustering"13. Fuzzy clustering is a method of assigning continuous numbers in the 0-1 interval as measures of membership, in contrast to the classical boolean membership approaches. This idea combined with assigning likelihoods to the similarity of two time-series14, and the use of wavelet transforms15,16, forms the basis of this study in cleaning the original time-series data, and assigning class values to different events.

The other major approaches for anomaly detection are naive Bayes17, or methods based on Neural Networks18. In both methods, there is a need to summarize data into a set of features that summarize the properties of events and can separate normal events from anomalies.

 

10 Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3):15, 2009.
11 Martin Langkvist, Lars Karlsson, and Amy Lout . Sleep stage classification using unsupervised feature learning. Advances in Artificial Neural Systems, 2012:5, 2012.

12 Pedro Garcia-Teodoro, J Diaz-Verdejo, Gabriel Maci Fernandez, and Enrique Vazquez. Anomaly-based network intrusion detection: Techniques, systems and challenges. computers & security, 28(1-2):18{28, 2009.

13 Hesam Izakian, Witold Pedrycz, and Iqbal Jamal. Fuzzy clustering of time series data using dynamic time warping distance. Engineering Applications of Arti cial Intelligence, 39:235{244, 2015.
14 Eamonn J Keogh and Padhraic Smyth. A probabilistic approach to fast pattern matching in time series databases. In Kdd, volume 1997, pages 24{30, 1997.
15 Ingrid Daubechies. The wavelet transform, time-frequency localization and signal analysis. IEEE transactions on information theory, 36(5):961{ 1005, 1990.
16 Paul S Addison. The illustrated wavelet transform handbook: introductory theory and applications in science, engineering, medicine and - nance. CRC press, 2017.

17 John S Eberhardt III, Todd A Radano, and Benjamin E Peterson. Application of machine learned bayesian networks to detection of anomalies in complex systems, May 24 2016. US Patent 9,349,103.
18 Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, and Gautam Shro . Lstm-based encoder-decoder for multi-sensor anomaly detection. arXiv preprint arXiv:1607.00148, 2016.

Raw sensor data showing the z-axis data


Figure 20: Raw sensor data showing the z-axis data (data along the motion of the elevator).


The provided accelerometer data has a resolution of approximately 0.5%g, with amplitude of events in the 3-10%g range. These characteristics make it possible to extract the events of interest and their features with high signal-to-noise ratios. However, raw data from the accelerometer is contaminated with irrelevant events, and seemingly non-random shapes that occasionally arise in the disturbances of the accelerometers. It is therefore not immediately clear how one can go from this data set to a set of classified events, or a set of features that will be useful for building a machine-learning system (see Figure 20).

After a relatively thorough data exploration and literature review, we decided to use wavelets, and take a maximum likelihood approach for finding the best matches to those wavelets.

In contrast to the Fourier transformation that seeks for structures that are present throughout the whole dataset, wavelet transformation is a powerful method for detecting localized structures in datasets7. Formally, the wavelet transform of a continuous signal with respect to a wavelet function, F(x), is defined as:

Wavelet function

Here, the p vector is the list of all the parameters that define a specific wavelet, ω is a weighting function, and ψ(x, p) is the wavelet function. The wavelet transformation, Eq. 1, is a convolution of the original function with the wavelet function. This can be therefore used to define the inner product of the two functions, F and ψ:

Inner product of the two functions, F and ψ

With this inner product definition at hand, and given certain conditions on the wavelet, one can define a Hilbert space for the set of wavelet functions. In principle, one should then seek for a set of orthonormal functions, to make sure that all the features are extracted. In practice, especially in the case of known events, one takes a matched filter approach, and picks a finite set of wavelets for detecting events of interest.

Based on data exploration, we chose four different sets of wavelets for detecting normal upward travels, normal downward travels, quick upward moves and stops, and quick downward moves and stops. The method used to find such structures in the dataset is to minimize the chi-squared differences between the events in raw data, and these four wavelets. A threshold method was used on the aggregated sum of the data points to prune the random similar structures that might arise due to random noise. The wavelet transformation proves to be a strong method in classifying events in the sensory data. The results of this wavelet transformation are presented in the next section.

Besides classification, it is also helpful to find meaningful features that can classify normal events. Springs and damping systems are integral parts of elevators. When an elevator is perturbed from equilibrium, it returns to a stable position through a damped oscillation. The oscillation frequency is characteristic of each spring. When a spring starts to fail, its oscillation frequency decreases. Anomalies were detected by searching for any oscillations that have angular frequencies in atypical ranges. Oscillations with excessively high amplitudes were also detected, which indicates overstressing the elevators' springs. These may cause premature fatigue in the elevator springs. The effectiveness of this approach is shown and are presented in the following sections.

A3.4.4.3 Data Analysis Results

Using the four wavelets explained in the previous section, the relevant localized events in data are detected, classified, and put into a sequence. Figure 21 summarizes the classification results based on wavelet detection. Any trip consists of a pair of events in the accelerometer data, an upward and a downward acceleration. In Figure 21, a base-line zero corresponds to no events, and thin positive or negative lines correspond to upward or downward accelerations, respectively (see caption for more details). These classification results are now ready to be used as inputs for methods such as HMMs to detect abnormality in the sequence of events, or can be used in combination with other techniques for increasing the prediction power of anomaly detection in speed, or displacement.

Event Classification using wavelets

Figure 21: Event Classification using wavelets. The base blue line at zero indicates no events. A thin blue bar with magnitude 1 or +1 indicates a downward or upward acceleration, respectively. A thin red bar with magnitude 2 or +2 indicates a quick downward or upward acceleration, respectively. The first three “wide bands" correspond to the initial noisy period. After that, all travels happen in pairs. E.g., the rst travel is a downward travel and ends with a quick stop. The next trip is an upward travel consisting of a normal upward and a normal downward acceleration and so on.

As was explained before, the team has also explored the idea of feature extraction from the events detected in the accelerometer data for clustering purposes. Like any other oscillatory system, oscillation frequency and amplitude are two crucial features that contain valuable information about the overall health of the system, as well as the abnormality of specific events. The simulated leveling data was then used to t damped oscillations to the vertical acceleration data. The goal is to seek for events of interest in the data set and find the best t parameters of their oscillation to obtain the scatterplot shown in figure 22. From this figure, it becomes clear that there are two distinct classes of oscillations.

With this scatterplot, it is now convenient to define what is outside of these two classes as anomalies. Oscillations with amplitude and/or frequencies outside of what is typically observed become warning signs.

To further understand these two classes of oscillations, the time series of oscillations from each of the two classes can be further investigated. Figure 23 shows an oscillation selected from the top left population of the scatterplot shown in figure 22. This population can be characterized as events that have a relatively high amplitude (i.e. abrupt drops in velocity), and high frequency. Consistent with this categorization, figure 23 depicts a trip that ends uncomfortably. These results and features can now be used to make Bayesian inferences, or as features for machine-learning algorithms.

Event Classification using wavelets

Figure 22: A scatterplot of the amplitudes and angular frequencies of various ts in a snippet of the data.

Plot of an oscillation with high amplitude and angular frequency

Figure 23: A plot of an oscillation with high amplitude and angular frequency.

A3.4.4.4 Summary and Future Work

In this study, the basis for anomaly detection was set in accelerometer data captured from elevator movements. Based on the characteristics of the provided data (i.e. the signal to noise ratio), it became clear that there is less need for advanced probabilistic models for identifying, and characterizing events of interest. Wavelet transformation was used for identification purposes, and a damped oscillator archetype to t for the amplitude and frequency of the oscillations that occur in the time-series data.

The results of this study can now be used to build machine-learning algorithms and make statistical inference models for the various types of anomalies that can happen during the lifetime of an elevator. Design and development of such models and algorithms is a natural next step for this project.

As a final note, it is worth noting that the initial results indicate that using accelerometers with sampling rates of 25Hz is most probably sufficient for detecting anomalies in oscillation frequency and amplitudes. However, this claim needs further studies.

A3.4.3 Analysis of Elevator Leveling Issues

A research group from the Intelligent Sensing Laboratory of Simon Fraser University led by Dr. Behraad Bahreyni was invited to provide advice and independent analysis on elevator leveling issues. Dr. Bahreyni’s group was known for their expertise in designing, testing and implementing high-performance accelerometers in various industrial and research applications. Thus, he and his team were invited to install their own data-logging device, axSense, in parallel to the Axivity AX3 device that was installed by Technical Safety BC. After a data-collection period of approximately two weeks, he and his team analyzed the data using the methodology described briefly below and was able to estimate elevator leveling offsets to within an impressive 10 - 15 cm margin of error.

A3.4.3.1 General approach for estimating leveling offset

Leveling issues occur when an elevator car floor does not level perfectly with the building floor when it arrives on its intended floor, leaving the elevator car a few inches higher or lower than the building floor. To calculate this height difference or leveling offset, Dr. Bahreyni’s team used a combination of sensors within the axSense data-logger to calculate the elevator’s vertical height at any given time. These sensors include:

  1. Altitude-meters – typically work by measuring atmospheric pressure to estimate the height of an object from sea level or other points of reference. The limitation with altitude-meters are that they are affected by changes in air pressure, which can be a problem in certain elevator cars/shafts that pressurize and depressurize due to very tightly-sealed elevator designs.
  2. Accelerometers – the distance of travel can be estimated by performing mathematical integration twice on acceleration. The first integration of acceleration over time yields velocity while the subsequent integration yields distance. The limitation with accelerometers is that they suffer from the uncertainty in sensitivity and drift over time during mathematical integration calculations, leading to relatively large errors in the estimated distance.
A3.4.3.2 Results of the leveling offset analysis

Using the data from the axSense sensors, Dr. Bahreyni’s group was able was to estimate floor height (Figure 24) and the likely offset for each elevator trip (Table 9). The error column in table 9 shows that the error of their estimated offsets was, on average, only 9 cm from the actual measured offsets. It was also noted that the error size could be significantly lowered if accurate measurements of the elevator offset were available and if the axSense device was properly mounted onto the elevator.

Another suggestion proposed was to use algorithms to optimally combine signals from multiple similar sensors with the goal of improving the overall leveling offset estimate. One such example is the Kalman filter, which is an algorithm commonly used in vehicle guidance, navigation and control.

Floor height

Figure 24. Floor height estimated using a combination of altitude-meter and accelerometer data

Time

Level

Measured offset (m)

Calculated offset (m)

Error (m)

10:44:28 AM

3*

1.8

1.617

0.183

10:51:43 AM

5*

0.9

0.545

0.355

10:54:46 AM

6*

0.06

0.059

0.001

10:55:32 AM

6*

0.04

0.133

0.093

10:57:47 AM

3*

0.05

0.06

0.01

10:59:00 AM

3*

0.45

0.397

0.053

11:00:20 AM

3*

0.25

0.335

0.085

11:01:15 AM

6*

0.11

0.011

0.099

11:02:33 AM

6*

0.07

0.034

0.036

11:05:06 AM

6*

0.41

0.35

0.06

11:07:18 AM

6*

0.13

0.071

0.059

11:08:40 AM

6*

0.05

0.006

0.044

11:10:30 AM

6*

0.06

0.046

0.014

11:11:25 AM

6*

0.08

0.09

0.01

11:40:15 AM

6*

3.62

3.355

0.265

11:42:16 AM

6*

3.75

3.666

0.084

11:45:54 AM

6*

3.85

3.693

0.157

11:47:27 AM

3*

0.15

0.161

0.011

11:48:33 AM

3*

0.2

0.36

0.16

11:49:27 AM

3*

0.05

0.108

0.058

Table 9. Elevator leveling offsets estimated from axSense data

In this table, measured offset is the value measured/guesstimated by the operator and calculated offset is what was extracted from the on-board sensors. The numbers in the error column are the differences between the measured and calculated offsets.

A3.4.3.3 Summary

In summary, this independent test and analysis by the axSense group shows that leveling offsets can be estimated using a combination of sensor types placed in data-logging devices. This approach of integrating the data from multiple sensors can also be extended to the detection of other elevator and technical system anomalies to improve the accuracy of all anomaly predictions.