A Project ReportOnSocially Assistive Robotics and Assistive TechnologyAbstractSocially assistive robotics (SAR) is a field of robotics that focuses on assisting users through any kind of a social rather than a physical interaction. Just as a good coach or teacher can provide motivation, guidance, and support, socially assistive robots attempt to provide the mentioned appropriate cues to encourage development, learning, or therapy for an individual.One of the applications of SAR would be play. Play is? an essential component for every child’s development. It provides infants and toddlers opportunities to increase their understanding of the world around them through engagement of physical, cognitive, language, and social skills. Thus, enhancing their sense of autonomy, self-confidence and achievement of critical developmental milestones. The aim is to develop a Social Assistive Robot (SAR) as a clinical tool to provide long term and intense intervention for children with disabilities or developmental delays through play.Assistive technology for the disabled, defined as any item, piece of equipment, or product system, whether acquired commercially off the shelf, modified, or customized, that is used to increase, maintain, or improve functional capabilities of a person with a disability. Unemployment being a major problem in terms of job availability and tax dollars spent for them. A major obstacle to employability would be navigation in an indoor facility. This sub-project would be to help design an indoor autonomous navigation bot using RFID tags. The system presented is intended to be used in a warehouse as an automated job coach to assist in the job of order filler.ContentsTitle Page…………………………………………………………………………………………………………………….1Certificate……………………………………………………………………………………………………………………2Abstract……………………………………………………………………………………………………………………… 31. Introduction…………………………………………………………………………………………………………….52. Emotion Recognition for WABBS..……………………………………………………………………………73. Autonomous Warehouse Navigation Bot…………………………………………………………………94. Assistive Technology Course……………………………………………………………………………………11Conclusion………………………………………………………………………………………………………………….14Bibliography……………………………………………………………………………………………………………….15IntroductionSocially Assistive Robots: Play is essential to child development which offers to them an opportunity to create, imagine, and practice while interacting with their environment. Through this interaction, children develop physical, cognitive, language, and social skills, thus enhancing their sense of autonomy, self-confidence, and achievement of critical developmental milestones [1, 2, 3]. Play also has repeated experiences which develop sensory feedback through exploration [4]. These repeated experiences also assist in learning due to an increase in association between neural processes and an overall increase in “efficacy of synaptic transmission along specific brain pathways” [5].Participating in play is crucial for all children, but for children with disabilities this is relatively inaccessible compared to their typically developed peers. The delays may be in any skillset, which may result in a lack of the physical ability to reach for a toy as well as diminished awareness of a toy due to visual or hearing deficits [6]. Children with disabilities also often have physical, cognitive, and sensory limitations, which increase the difficulty in accessing the opportunities experienced by typically developing children [4]. Deficits in developmental stimulation and play experiences can result in issues later in a child’s life in various areas including gross motor, fine motor, language, cognitive, and social skills. Without these experiences, a child may have difficulties reaching certain developmental milestones, which in turn can prevent them from reaching their full potential.Play for all children involves toys, which perform an essential role in enhancing play skills since children are naturally attracted to them. It has been suggested that toys may have a greater impact for children with severe disabilities when they have educational value. However, young children with disabilities are less likely to actively engage themselves with objects or other people, which results in the need for more frequent and exciting play opportunities [7]. To meet this need, toys specifically adapted for better educational access and greater interaction- to provide multi-sensory input and allow for repetitive interaction- have been developed for children with all kinds of disabilities [8, 9, 10, 11]. Additionally, assistive technologies designed for adapted play and social interaction can prove to be fundamental in enabling children with physical disabilities to play, as well as facilitate learning in those with cognitive disabilities [12].Recently, the term “Socially Assistive Robots” (SAR) has entered the literature to describe a unique cross section between Assistive Robots (AR), designed to directly support the physical needs of a patient, and Socially Interactive Robots (SIR), designed to entertain or to form a social bond with an individual. SAR are designed to assist the human user through social interaction, while creating measurable growth in learning and rehabilitation. Defining what features characterize “socially assistive” emphasizes the importance of the human participant and of assistance to human users, like AR. This distinction also specifies the assistance is through social interaction, like SIR [13]. Put simply, SAR seek to replicate, but not replace, the therapeutic and educational benefits that stem from the relationship between a caregiver and an individual with a disability.The goal of this research project is to develop a semi-autonomous SAR for children with complex cerebral palsy- to increase effectiveness of rehabilitation therapy as it improves independence in daily activities, while also improving quality of life and reducing caregiver burden. An adaptive SAR designed for a specific developmental delay, or set of developmental delays, can be used in a home or community setting. Such a system can also be effective in assisting children with complex cerebral palsy to reach developmental milestones during the most critical time of neural development: birth to three-years-old [5]. To make the robot adaptive it should be able to discern the different emotions of a child. The first step would be to identify levels of emotion on a spectrum between happy and sad. Since the main processing device on WABBS is an HTC Sensation smart phone an App that does that will be the first objective.Assistive Technology: Focusing on Job coaching to assist the disabled in employability in a warehouse environment. A major technological hurdle would be the positioning or knowing where he exactly is. Knowing where an employee is located helps provide feedback and direction in a nonlinear manner to assist him in real time. Apart from just localization navigation is also an important part of this project. Learning to navigate in your new workspace is a challenge for all employees of all cognitive levels but is a much more difficult for those with disabilities. The combined necessity of positioning for the as well as navigation for the benefit of the employee are closely intertwined technologically. The purpose of this project is to address these issues in a proof of concept job scenario of warehouse order picking.Emotion Recognition For WABBSSpecific Aim: Create a mobile application that will deduce emotion based on visual inputs that can be trained specifically for each child with a cognitive disability.This project provides a basic design and a prototype for the SAR WABBS Bot which is a bot that stimulates play in children with cerebral Palsy. The current design aims at storage of video data and sensory inputs from an HTC Sensation phone which will be the face of the bot (Figure 1). These inputs will then be used to further train the phone to recognize the children’s emotions to see if children are engaged and to behave accordingly. Work has begun making the app for this which monitors data and recognizes emotions.     Figure 1. The current design of the WABBS Bot.Android Application: The app is going to be an android application on SDK 13 for Android version 4 and above. The app takes in the sensor readings as well as camera input and stores this data locally. The app also processes the video feed in real time using the training generated graph file to classify the emotions. All the video processing is done using OpenCV libraries for android. The app then displays a visual of a happy rabbit or a sad rabbit based on the classification.Learning: The learning part of the bot happens offline on a separate more powerful computer. Using OpenCV and Tensorflow each frame of the video feed will be manually classified by the parents or the caregivers first. This is then passed to the Mobilenet model by Tensorflow to retrain the last layers of it for custom classification. MobileNets are small, low-latency, low-power models parameterized to meet the resource constraints of a variety of use cases. They can be built upon for classification, detection, embeddings and segmentation similar to how other popular large-scale models, such as Inception, are used. MobileNets can be run efficiently on mobile devices with TensorFlow Mobile.  FUNCTIONAL REQUIREMENTS:USABILITY REQUIREMENTSThe app must be robust enough to withstand child use. It needs to be easy enough for a clinician to use. It needs to keep a child’s attention, so it needs to be attractive to a child and draw the child to it. (bright and colourful, expressive, cartoon rabbit face, bright red fur on robot, high contrast).Ability to adapt to any child.PERFORMANCE REQUIREMENTSThis app will have to process camera footage in near real time to decipher the emotion of the child. It should also have a data storage capability.Minimum consecutive run-time of 30 minutes.SUPPORTABILITY REQUIREMENTSIf the program has any problem, it shall allow the users make a report to the development team with availability file logs.This project needs to be very well documented so that a future group can expand on this project.This project should work with a single day of training to facilitate quick adoption. Autonomous Warehouse Navigation BotSpecific Aim: Create a navigation application (Real – time positioning and Graphical Navigation) that will deduce position based on RFID fingerprint and display location graphically on a map with the ability to generate and outline routes to a chosen destination while tracking progress and adjusting accordingly.The software components were broken into three components: environmental learning, real – time positioning, and graphical navigation. Environmental learning components required the capability of acquiring and storing information about the positioning area, which include what the radio fingerprint looks like at given locations. Real-time positioning components required the ability to both examine the current locations radio fingerprint on the fly as well as take information stored by the environmental learning component and use it to infer a current location. Finally, the graphical navigation components needed be able to take current location data, as well as destination data, and create a route through the room to be graphically displayed. It also needed to be able to know when the user had arrived at destinations and move forward to the next task at that point.Real – time positioning: The real -time positioning package takes the XML reference file and EPC master list (From the Environmental Learning Package) as inputs, using an XML reader to load them into memory. It uses periodic acquisitions from the Speedway RFID reader to determine location. The positioning system uses a weighted k-nearest neighbour (WKNN) algorithm to infer location based on the reference fingerprints. The method began by using the reader to acquire a fingerprint using the same read duration and strength as the fingerprints before. The reading was passed through the same parser as the references.To determine position, the current fingerprint was compared to all reference fingerprints iteratively to determine cosine similarity values for each. Cosine similarity uses a spatial comparison of the two vectors by taking the cosine of the angle between them according to Formula 1, where f1 is the current fingerprint and f2 is the reference.cos?=f1.f2|f1f2|    (1)Graphical Navigation: The graphical navigation package used the positioning system, in tandem with destination inputs to navigate to predetermined goals positions. As an input, the navigation package required Cartesian destinations in the form of an array of coordinates. To assist in navigation, a set of 9 nodes were established at intersection points within aisles of the warehouse, as shown in Figure 2A. To navigate, the software used a Nearest Neighbour algorithm to find a shortest path. It first took the user position from the real-time positioning system as well as the first input destination position. It then found the nearest intersection node to each, making them the active nodes. The software then iterated through the node list and found the next nearest nodes that are between the start and destination. Each node was removed from the node list until the active nodes are either adjacent or identical. Using this method, the path was built from both ends, meeting in the middle to complete it.A       B   Figure 2: Mock warehouse schematics showing the node layout (A) and the eligible nodes for navigation (B). Nodes are represented by green circles, with a red circle for the user location and a red X for the destination. Nodes within the reddened area are eligible for navigation use in the given scenario.  Assistive Technology Course Specific Aim: To get a good understanding of assistive technology and client interaction to design appropriate custom AT and the rules and ethics revolving around this.Topic 1 – Introduction: According to the Individuals with Disabilities Education Act Amendments (IDEA) of 2004, an assistive technology (AT) device refers to “any item, piece of equipment, or product system, whether acquired commercially off the shelf, modified, or customized, that is used to increase, maintain or improve the functional capabilities of a child with a disability. The definition is quite broad and encompasses devices that are electronic (e.g., computer, scanner, tape recorder) and non–?electronic (e.g., pencil grip, large print books), so long as those devices are “used to increase, maintain or improve the functional capabilities of individuals with disabilities.” There are several AT products available that can help compensate for a wide range of difficulties. These products include computers, tablets, smartphones, and other mobile electronic devices.Writing Difficulties: AT devices to enhance writing skills include word processors, spell checkers, proofreading programs, speech synthesizers/screen readers outlining programs, graphic organizers, word prediction programs alternative keyboards, paper-based computer pens and speech recognition systems.Reading Difficulties: Several AT devices can be used to compensate for reading LD,including audio books and publications, variable speech control tape recorders speech synthesis/screen reading systems, paper-based computer pens, and optical character recognition/speech synthesis systems.Organizational and Memory Difficulties: Numerous AT devices can be used to help people with LD who have organizational and memory problems associated with LD. These devices include personal information/data managers now integrated into mobile devices like smartphones.Topic 2 – Ethics: There are several ethical considerations one must take into while prescribing an AT and even after prescription. One area is video surveillance. The use of video surveillance installed in homes of people living with dementia may provide a more economic and efficient means for caring for those occupants who wish to maintain their independent living. For example, such a video surveillance system would be available to family caregivers and would provide a rapid means of ascertaining the wellbeing of their family member with dementia. However, formal healthcare providers have been reluctant to make use of video surveillance because of ethical concerns in capturing and storing media of people living with dementia as well as others in the home, including formal care staff and other family members that may include other vulnerable people or those who object in principle to video surveillance in their relative’s home.   Topic 3 – User Persona: A persona is a fictional character that represents a subset of the market we want to address. A persona typically has: a name, a picture, relevant characteristics such as age or income group, behavioural traits, common tasks; and, a goal that describes the problem the persona wants to see solved or the benefit the character wants to achieve. Personas should help develop empathy for users and customers. This avoids the fallacy of a solution-centric approach: worrying more about the product and its features and technologies than the reason people would want to buy and use it in the first place.Project: We were each given our own clients with a disability to develop something for them. The first step of which is to write a user persona.Caryl:* Lives in Elizabeth with husband Jim, who is her caretaker. Also has a son.* T4 – L1 SCI paraplegic, has neuropathy from the waist down which is a side effect of surgeries (32 years post-injury).*Used to work for a non-profit organization.* Hobbies include quilting, computer, some cooking, going out to eat with Evelyn.* Has lowered kitchen counter heights as she wants to be able to cook.* Accessible 2014 Toyota Sienna ($72000) with a Braun ramp (sliding action) which has kneel assist and obstacle detection (stops when it physically contacts an object), as well as an EZ Lock for her power wheelchair to anchor into the driver position. Drives herself or her husband drives by swapping the seat positions.* Owns a Quantum Q6 Edge PWC (midwheel drive, standard joystick, hinged armrests, JABA seat cushion) [came to interview on this], Invacare Pronto PWC, and a Smartphone.* Medicare is her Insurance.* She gets her products from United Healthcare.Goals:* To increase carrying capacity and reduce back pain, she wants a stable lap tray, which (unlike standard ones) does not slip or tilt when moving.* As both she and her husband use the kitchen, having the stovetop at a lower height is an inconvenience for her husband, so some form of height adjustable stove would be helpful.Evelyn:*Lives in Parker has two sons, husband recently passed so lives at home alone.* Neighbour Rob usually helps with the mail, trash, and occasional fixes.* High functioning SCI quadriplegic (40 years post-injury). Has limited right hand function, osteoarthritis in her left hand/wrist, and some scoliosis.* Handles her own transfers at home with a transfer board.* Has lowered kitchen counter heights as she loves cooking.* Has dogs.* Lives at the top of a hill where her driveway extends to the bottom.* Used to work for a non-profit.* Loves to keep her house/porch/yard clean and going out to eat with Caryl.* Used to raise/ride therapeutic horses, now owns race horses (stopped riding because of potential bone injuries).* Accessible 2014 Toyota Sienna ($70000) with Braun ramp (folding action) with kneel assist and a rotating driver’s seat with adjustable footrest for her to transfer to/from her manual wheelchair. Drives herself; her husband used a separate vehicle that does have EZ Lock in the driver position.* Owns a Quickie 2 HP Folding MWC (JABA seat cushion), Tilite ZRA MWC (rigid frame), Jazzy-Air PWC, SmartDrive, Flip Phone, Power Gate (for her dogs), and a Leaf Blower.* Numotion supplies the DME that she doesn’t buy OOP.* Medicare is her Insurance.* She gets her products from United Healthcare.Goals*To increase carrying capacity and reduce back pain, she wants a stable lap tray, which (unlike standard ones) does not slip or tilt when moving. This would be more useful on an MWC, as that’s what she uses in the house.* Lack of accessible location to pick up mail and drop off trash. Currently must drive down the hill/driveway in her van to collect mail/packages and drop trash. FedEx does deliver to the door, but they often forget to lock the power gate that she uses for her dogs. This is especially problematic when she gives them the power gate code during her absence, and they don’t enter the code again when leaving to close the gate. So, she either wants to be able to collect the mail herself or wants some way to lock the gate remotely when she is not at home.* Has osteoarthritis in her left (more functional) hand/wrist and wears a wrist brace. This makes transfers painful. She is interested in electrostimulation therapy rather than a surgical solution.* TiLite ZRA would be her preferred chair to use, but the Frog Leg casters are not levelled, causing one to swivel freely in the air. Numotion has assessed it and deduced that it is due to the gradient of her floors. However, she says the same caster swivels everywhere in the house, which makes it highly unlikely to be a terrain issue.Topic 4 – Usability Testing: Usability testing is a way to see how easy to use something is by testing it with real users. Users are asked to complete tasks, typically while they are being observed by a researcher, to see where they encounter problems and experience confusion. If more people encounter similar problems, recommendations will be made to overcome these usability issues.Conclusion With the current objective of the Socially Assistive Robotics Lab focusing around children with Cerebral Palsy. As these children often experience limitations in their quality and quantity of play, as compared to their typically developing peers. Introducing a bot which can play with these children to reach the developmental milestones as their typically developing peers is what we are aiming for. My current project which aims to detect the emotions of these children will serve as a basic driving module for the bot’s actual motion and engagement. This app will detect two simple emotions of happy and sad with custom training for each child.References [1] Cardon, Teresa A. 2011. "Caregiver Perspectives About Assistive Technology Use With Their Young Children With Autism Spectrum Disorders." Infants and young children 24 (2):153[2] Lee, H., Song, R., Shin, H. 2001. "Caregiver burnout." In The Encyclopedia of Elder Care, edited by M.D. Mezey, 114-116. New York: Springer Publishing[3] Roussou, Maria. 2004. "Learning by doing and learning through play: an exploration of interactivity in virtual environments for children." Computers in Entertainment (CIE) 2 (1):10-10[4] Valadao, C., Bastos, T.F., Bortole, M., Perim, V., Celino, D., Rodor, F., Goncalves, A., . . .Ferasoli, H. 2011. “Educational robotics as a learning aid for disabled children.” In Proceedings from ISSNIP 2011: Biosignals and Biorobotics Conference (BRC). Vitoria, Brazil[5] Berlucchi, G. and Buchtel, H. A. 2009. “Neuronal plasticity: historical roots and evolution of meaning.” Experimental Brain Research. 192: 307.[6] Cook, A., Encarnação, P., and Adams, K. 2010. “Robots: Assistive technologies for play, learning and cognitive development.” Technology & Disability. 22(3), 127-145. doi:10.3233/TAD2010029721[7] Brodin. J. 1999. “Play in children with severe multiple disabilities: Play with toys-a review.” International Journal of Disability, Development and Education, 46(1), 25-34[8] Stiehl, W., Lieberman, J., Breazeal, C., Basel, L., Cooper, R., Knight, H., Lalla, L., Maymin, A., and Purchase, S. 2006. “The Huggable: A Therapeutic Robotic Companion for Relational, Affective Touch.” In Proceedings from IEEE Consumer Communications and Networking Conference (CCNC) 2006. Las Vegas, Nevada, United States.[9] Barakova, E. I., Gillesen, J. C. C., Huskens, B. E. B. M., Lourens, T. 2013. “End-userprogramming architecture facilitates the uptake of robots in social therapies.” Robotics and Autonomous Systems 61 (2013) 704-713. doi: 10.1016/j.robot.2012.08.001[10] Wainer, Jacques, Ben Robins, Farshid Amirabdollahian, and Kerstin Dautenhahn. 2014. " Using the humanoid robot KASPAR to autonomously play triadic games and facilitate collaborative play among children with autism. " Autonomous Mental Development, IEEE Transactions on 6 (3):183-199.[11] Costa, Sandra, Hagen Lehmann, Ben Robins, Kerstin Dautenhahn, and Filomena Soares. 2013. "” Where is your nose?”: developing body awareness skills among children with autism using a humanoid robot.” In Proceedings from ACHI 2013: 6 th International Conference on Advances in Computer-Human Interactions.[12] Salter, T., Werry, I., & Michaud, F. 2008. Going into the wild in child-robot interaction studies: Issues in social robotic development. Intelligent Service Robotics, 1(2), 93-108. doi:10.1007/s11370-007-0009-9