It seems global domination in the social media space and designing 360-degree cameras for virtual reality is not enough for Mark Zuckerberg, who has also quietly been working on his own ‘personal challenge’ of building a bespoke home automation system with artificial intelligence for his home.
Inspired by ‘Jarvis’ in the Iron Man franchise, Zuckerberg’s home automation system incorporates technology from the likes of Crestron, Sonos and Nest and took around 100 hours to build. “My goal was to learn about the state of artificial intelligence -- where we're further along than people realise and where we're still a long ways off,” says Zuckerberg. He added that the project gave him a ‘thorough’ view of home automation, which he shared in an online post explaining how he approached building the system.
Zuckerberg has control of lighting, temperature, appliances, music and security in his home via a phone or computer with the system, that is able to learn new words and concepts as time goes on thanks to natural language processing, speech recognition, face recognition, and reinforcement learning, written in Python, PHP and Objective C.
In his note, published on Facebook on 19 December, Zuckerberg was met with the industry-wide issue of ‘connecting and communicating all of the different systems in my home’ and their individual protocols when building the system – which he identifies as his biggest challenge. “I had to reverse engineer APIs for some of these to even get to the point where I could issue a command from my computer to turn the lights on or get a song to play.
Zuckerberg states he found some products lacking with his smart home venture; “For assistants like Jarvis to be able to control everything in homes for more people, we need more devices to be connected and the industry needs to develop common APIs and standards for the devices to talk to each other.”
He claims there is opportunity in the marketplace for dealing with open-ended requests, as his own system works to offer an improved music-listening experience with context (i.e. looking at past listening patterns and learning the classification of songs). He adds: “I've found we use these more open-ended requests more frequently than more specific asks. No commercial products I know of do this today, and this seems like a big opportunity.”
Zuckerberg built a messenger bot to communicate with Jarvis once it was programmed on his computer – something he states is vital for AI products in addition to voice to consumers’ preference for a private messaging interface.
Voice and speech recognition are integrated into the system for the speed and efficiency of which requests can be dealt with. “The Messenger bot is great for many things, but the friction for using speech is way too much. My dedicated Jarvis app lets me put my phone on a desk and just have it listen.”
He continues; “I could also put a number of phones with the Jarvis app around my home so I could talk to Jarvis in any room. That seems similar to Amazon's vision with Echo, but in my experience, it's surprising how frequently I want to communicate with Jarvis when I'm not home, so having the phone be the primary interface rather than a home device seems critical.”
He adds that although speech recognition systems have improved dramatically at late, no AI system is at the level it can understand conversational speech without structure commands and they are still very much geared to solving specific problems. He claims systems being more specialised than they first appear means the industry still has many developments still to make.
What’s next for Mark Zuckerberg and Jarvis? “In the near term, the clearest next steps are building an Android app, setting up Jarvis voice terminals in more rooms around my home, and connecting more appliances.”
“In the longer term, I'd like to explore teaching Jarvis how to learn new skills itself rather than me having to teach it how to perform specific tasks.”
He concludes that the industry still has a long way to go; “Even if I spent 1,000 more hours, I probably wouldn't be able to build a system that could learn completely new skills on its own -- unless I made some fundamental breakthrough in the state of AI along the way.
Could Facebook make a play for the smart home market? Don’t rule it out. Zuckerberg states “it would be interesting to find ways to make this available to the world.” He adds: “If I ever build a layer that abstracts more home automation functionality, I may release that. Or, of course, that could be a great foundation to build a new product.” Watch this space.
Read Mark Zuckerberg’s detailed run-through of his Jarvis home automation system.