Registration for this special topics course is by permission of
the instructor only, and enrollment is limited. If you are
interested in enrolling in this course, please follow the
instructions below precisely:
Read through this entire webpage in detail to understand what
this course is about and what work it will entail. As a
fair warning, this will be a challenging and time-consuming
course, but (hopefully!) very rewarding.
Our focus in Fall 2017 will be on creating service robots that can continually
roam the hallways of Penn, performing tasks for people.
Make certain that you meet the course prerequisites. At
this time, we do not anticipate that there will be any
exceptions to these prerequisites.
If you are interested in enrolling in the course, complete the
form to apply for a slot in the roster.
Since there are a limited number of enrollment slots, please
note that just because you completed the form above does not
mean that you will be automatically enrolled in the course.
Although slight priority will be given to students who
completed the form early, students will be selected for this
course in order to ensure that the roster contains people with
a variety of backgrounds (AI/learning, electrical engineering,
mechatronics, control, etc.). Having students with a
variety of backgrounds is essential for the interdisciplinary
nature of this course.
You will be
notified as soon as possible (most likely in mid April or early May)
whether or not you have been given permission to register
for this course.
permission to register has been granted, you will be
expected to register in a timely fashion. If you
delay significantly, the department reserves the right
to reallocate your slot to another student.
If you change
your mind about enrolling in the course, out of
consideration for your fellow students, please complete
the enrollment application form again, choosing the option
to delete your application.
In order for robots to operate alongside humans in complex,
unstructured, uncertain environments, they require substantial
intelligence. However, the field of artificial intelligence (AI)
has fragmented into various subfields, each studying different
aspects of intelligence in relative isolation. The problem of
designing intelligent robotic systems that persist in everyday
environments provides an opportunity to reintegrate these
different aspects of AI into a complete intelligent system.
In this project-based seminar course, students will study and
develop an intelligent personal robot assistant, integrating
perception, manipulation, learning, planning, and interaction. The
resulting versatile robot will be capable of learning and
performing a variety of tasks in real-world environments and
collaborating effectively with humans. In addition, students will
study a variety of advanced AI topics, including high-level
perception and reasoning, scalable knowledge representation,
lifelong/multi-task learning, integration of perception and
control, learning from demonstration, and human-robot interaction.
At least TWO (2) of the following courses:
CIS 519 or 520 - (Intro. to) Machine Learning
CIS 521 - Fundamentals of AI
CIS 580 or 581 - Machine Perception or Computer Vision
ESE 650 - Learning in Robotics
MEAM 520 - Introduction to Robotics
MEAM 620 - Robotics
This course will include two major components:
A seminar-style discussion of various topics in integrated
We will study a variety of advanced AI techniques for
perception, manipulation, reasoning and learning, and methods
for integrating these techniques together into a complete
Students will be expected to come to each class prepared to
discuss the required reading, having prepared reading
Students will take turns leading the discussion.
Discussion leaders will also be responsible for preparing and
delivering a brief presentation on the daily topic.
A semester-long project developing an integrated
intelligent personal assistant robot
Students will work in interdisciplinary teams (balancing
backgrounds in AI, control, mechatronics, etc.) to design and
build intelligent personal assistant robots that are capable
of performing a variety of tasks in real-world environments
and interacting with humans.
These robots will be expected to operate in the uncontrolled
environments of the hallways, elevators, and various rooms of
the engineering buildings at Penn.
There will be various milestones, including an initial
proposal, design, status reports/demos, and a final report
In addition to developing the intelligent robot, this
project will give students experience with developing research
proposals, crafting development plans, conducting literature
reviews, conducting evaluations, and presenting research
Further details on the project will be discussed during the
We will study the following topics:
The Robot Operating System (ROS)
Case studies in integrated AI: Stanford STAIR,
DARPA urban & robotics challenges, RoboCup, Robo Brain
Project, UPenn MAGIC, CMU CoBots
Scalable learning: Online learning, transfer and
Integrating perception, learning & control:
architectures, deep learning for object recognition/scene
understanding, and deep reinforcement learning
Novel object recognition & manipulation: novel
object grasping and manipulation, grasping in cluttered
Localization & hierarchical planning: localization
and HTN planning in continuous deployment, anytime path
planning/replanning, planning in dynamic environments
Human-Robot Interaction: collaborative problem solving,
learning from demonstration
Evaluation methodologies and experiment design
Social, economic, safety, & privacy considerations/ethics
of personal robots
Project formulation, motivation, design, and presentation
These topics and due dates are all subject to change. Readings
for each of these topics will include a variety of journal
articles, conference papers, and technical reports.