Development and evaluation of an intelligent student assessment system in a remote laboratory for embedded systems education (RLESE)

This research designs, implements, and evaluates an Intelligent Student Assessment (ISA) System for a Remote Laboratory for Embedded System Education (RLESE) in the School of Information and Communication Technology (ICT) at the University of Tasmania (UTAS). The UTAS’ RLESE aims to increase students’ learning in embedded system concepts by providing physical infrastructure that can be remotely accessed.
Remote laboratories are laboratories with physical hardware, but that are controlled and managed online. Students can log into a university’s remote laboratory and access the hardware from their home or school, while watching the results via real-time streamed video.
Embedded systems are a complex interdisciplinary field that uses software to program and control hardware devices, such as sensors and actuators. The vast usage of embedded systems results in high industrial demands for engineers with background knowledge in such systems. An example is the Industry 4.0, the latest revolution in industry, a concept on the industrial increasing interconnectivity and smart automation, such as Internet of Things (IoT). The establishment of remote laboratories in embedded systems supports a lifelong learning process, and assure the relevant engineering competences for the realisation of the Industry 4.0 concept.
Remote laboratories are valuable, especially when remote learning is the only option. In 2020, the COVID-19 pandemic reinforced the importance of remote learning, and that a sudden shift to remote learning is sometimes required. Any study on how to improve student experience on remote laboratories, such as the work conducted in this thesis, assists in the preparation for future learning, under a “new-normal” scenario. In this scenario, teachers will not only have to adapt their coursework and teaching method to the remote environment, but will also have to re-think how to assess the students.
The learning outcomes of students when conducting laboratory activities are only measurable by means of assessment. An assessment based on written evidence, laboratory reports, and final exams is not an effective way to evaluate a student’s learning process. Moreover, focusing on the final grade, and not on the learning process, does not evaluate a student’s competence. Therefore, a holistic approach when assessing students conducting laboratory activities is required. Continuous assessment requires a higher involvement and dedication from teachers than traditional assessment methods. This high involvement and dedication are even more challenging when it comes to instrumental competency – a learning competency predominant in technical studies.
Another challenge in remote laboratories is a possible failure of a hardware component. A failure can happen due to many reasons, such as natural wear of components, harsh environmental conditions, and a short lifespan of components. Because of the lack of a tutor present at all times in a remote laboratory, a hardware failure may occur without being noticed. An unnoticed hardware failure impacts a student’s learning experience and motivation, and it also affects how the student is assessed.
The ISA System developed in this thesis is composed of two integrated subsystems: The Student Performance Assessment (SPA) Subsystem and the Hardware Fault Detection (HFD) Subsystem. The SPA Subsystem applies a probabilistic reasoning approach, based on Bayesian Belief Networks (BBN), to continuously assess students’ learning performance when using the UTAS’ RLESE. To assess the students’ learning performance, the SPA Subsystem virtually observes students’ behaviour while they develop and execute their experiments in the UTAS’ RLESE, measuring the students’ acquired knowledge, understanding, abilities, and skills. The HFD Subsystem determines if there is a fault associated with each electronic device (hardware) in the UTAS’ RLESE when students are performing experiments. A fault in a electronic device can affect the students’ learning performance and is, therefore, necessary to make the ISA System and the UTAS’ RLESE reliable. The HFD Subsystem applies fuzzy logic, an artificial intelligence technique, to assess data and identify a possible fault in the UTAS’ RLESE hardware. The SPA and HFD Subsystems are joined to form the ISA System, which provides an intelligent feedback of students’ performance during and in the end of their work.
The methodology used in this investigation involves a research strategy using a case study approach with an objective ontology and a positivist epistemology. The research design consists of three phases: design, implementation, and evaluation. Quantitative data is collected, analysed, interpreted, and discussed in all phases.
The first phase of the research design designs the ISA System according to the research questions, identified problems, and defined aims. In this phase, the HFD Subsystem and the SPA Subsystem are designed. The two subsystems are then integrated to form the ISA System. In addition, this design phase also involves obtaining ethical approval to conduct this investigation, collecting data from experts to design the system, and designing the course in embedded systems using the UTAS’ RLESE.
The second phase of the research design implements the ISA System in the UTAS’ RLESE. First, students’ synthetic data and hardware fault data are generated to pilot and test the ISA System. After the piloting and testing, data from experts is collected, analysed and interpreted to calibrate the BBN that is part of the ISA System. Then, a live experiment is ran collecting students’ data in an Embedded Systems Course using the UTAS’ RLESE. In this experiment the students are randomly divided into two groups – Group A (control group) and Group B (intervention group).
The third phase of the research design evaluates and validates the ISA System and its impact in student performance. The ISA System is evaluated and validated on its effectiveness as a tool to assess students, on its reliability, and on its acceptance by the users. In this phase, quantitative data from experts, students, the BBN, and the UTAS’ RLESE hardware is collected, analysed, interpreted, and evaluated. The quantitative data is analysed using descriptive and inferential statistics. The results are then analysed and discussed within the context of the current knowledge in the field of RLESE and student assessment.
The key research findings of this investigation highlight the importance of an ISA System for a holistic assessment of students. The ISA System benefits both students and teachers, by assessing the students’ performance in the UTAS’ RLESE through observation of students’ virtual behaviour. In addition, the ISA system provides the students with intelligent feedback, based on the BBN output, helping them to improve their performance and their learning experience. The HFD Subsystem, based on fuzzy logic inference, improves the performance of the ISA System and the reliability of the UTAS’ RLESE, supporting a fair assessment of the students’ performance. Moreover, the ISA System developed in this investigation is well accepted by students. According to the survey taken in this study, students are more satisfied with the UTAS’ RLESE when they receive support and feedback provided by the ISA System. In addition, most of the students note that the UTAS’ RLESE support and feedback enhances their understanding about embedded system education, supporting that the UTAS’ RLESE is a good tool for teaching and education.