★ View example final projects from the Spring 2023 semester here.

Overview

This is an opportunity for students to design and implement an interactive system system based on spatial user data. By spatial data we mean information about the pose of pedestrians on an indoor building on campus (e.g., AKW or Becton). The data will be captured by a Kinect Azure sensor, which has a color and a depth camera. The sensor SDK can detect and track individuals in front of it.

In addition to the Kinect sensor, the hardware for the system includes a big TV and a computer (hidden in the image above). The computer will read the data from the sensor, complement its tracking information with conversational group predictions (using the model from this paper), and output the information on a web server for you to use in your system. The computer will also run the interactive system and display its visual output on the TV.

The interactive system should be designed from a user-centered perspective, aiming to provide value to users based on findings from design research. The project focuses on systems that highlight moral principles, help users understand real-time or historic spatial data, or provide novel opportunities to relate nearby pedestrians. Example systems that fit the theme for this project can be seen in this website, which showcases projects from the Spring 2021 edition of this course.

As part of the project, students need to identify the activities that their proposed design will support and the type of questions that the system would allow users to answer. If there are existing solutions for the chosen activites/questions, it is important to identify what does not work for them and how the proposed design provides better support.

We expect to have 40 projects total with teams of 4 students each (or 3 students in a few cases depending on final enrollment).

Technical Details

There are 4 installation setups, each with a TV display, Kinect sensor and a computer to run students’ prototype systems. Each project group will be assigned to one installation setup for the whole semester.

The computer driving a display runs Ubuntu 20.04. In this computer, there will be software that collects data from the installation setup and makes it accessible via a websocket connection within Yale’s network. The data includes estimated human poses for the pedestrians visible from the Kinect (in the form of skeleton keypoints) and group predictions (e.g., based on spatial behavior). Students are expected to implement interfaces that leverage this data using web technologies (HTML, Javascript). Python can be used to create local test environments for the project, as in the demos below:

The applications developed by students will be shown in a Chrome web browser window that is displayed in full-screen mode on the corresponding TV. We recommend using the Phaser 3 library for composing different types of graphics and creating animations.

If students have no prior experience with JavaScript (JS) or object oriented programming, we recommend this tutorial by Mozilla. In particular, this section is helpful ƒor understanding object-oriented programming with JS.

Questions? The course staff created this Google Doc for common/general Q&A related to the system prototype (including working with GitHub). Students may also post questions about their implementation in Ed Discussion.

Project Components

Late Policy: Any assignment submitted after the official deadline, will receive a penalty of 50% of the assignment grade. Assignments will not be accepted more than 24h after the deadline.

NOTE 1: For all assignments, we encourage teamwork, positive interdependence, and individual accountability. Students will have to report on their team collaboration and individual contributions on each assignment.

NOTE 2: Using ChatGPT or automatic text generation algorithms for writing responses to the assignment questions outlined below is not allowed. One of the main goals of the semester-long project is to give students practical experience with HCI design, research and prototyping methods. This requires doing the work: gathering data, looking for insights, synthesizing findings, and creating prototypes that can be tested from a user-centered perspective. Additionally, by dedicating effort to writing assignment responses, students will practice their communication skills. These skills are important for all sorts of academic activities and critical for a successful future career, especially in areas related to computer science.

☞ Please see this tutorial page for more technical information about the displays.

Assignment 1: Problem Ideation (due on Wednesday, Feb 1 at 12pm)

The goal of the first assignment is for students to think of problems that their installation could address. What role can the interactive installation take to help solve these problems? At a high-level, what will the group create?

During class, students will get started thinking about some ideas: Students need to think about three starting points for brainstorming domains, problems, and goals that might be supported by their interactive installation. By domain, we mean an aspect of life, such as wellness. By problems, we mean something negative that needs to be solved (not unsolvable bad things). To identify goals and problems, it is often useful to ask “why” (e.g., within the university domain: Why does a person care to socialize with nearby pedestrians or the Yale community?).

For this assignment, each team will conduct their own brainstorming session to come up with a set of domains, problems and goals that are relevant to the team members. When brainstorming ideas, priority should be given to the quantity (and diversity) of the ideas, not to the quality of any one of them. The brainstorming effort may include ideas from existing products or solutions, but no pair of ideas should be alike. Students may take advantage of the ideas that came up during class to begin this brainstorming session.

Students may use online tools to brainstorm using virtual post-its, e.g., LucidChart (which provides free Education accounts) or Miro (which also offers Education accounts).

Once the brainstorming session by the team ends, students should create an affinity diagram for the ideas that were brainstormed together. Then, the team should choose one problem space to investigate further and look for related, existing solutions.

The submission for this assignment is a pdf document (with the students’ name and netid at the top) that needs to be submitted to Gradescope. A template in Google Docs is provided for students’ convenience. Importantly, students should complete each section of the assignment in a new page of the document in order to facilitate grading. The pdf submitted by the team should include four main sections and a collaboration record:

  1. A picture of an affinity diagram generated from brainstormed ideas by all team members. (1.25 pt)

  2. A summary of key ideas/themes/clusters that emerged from the affinity diagram (at least N key ideas should be proposed, where N is the number of people in the project group). Each idea should be one or two sentences, identifying the domain, and the problem or goal. A person familiar with the project but who is not a member of the group should be able to understand each idea. (1 pt)

  3. A chosen problem space to investigate further in this project based on the key ideas from above (Section 2). The description of the problem should include:
    • an explanation of what the problem is (0.25 pt),
    • why it is an interesting problem (0.25 pt), and
    • why it is not trivial to solve (0.25 pt).

  4. A brief description of related solutions to the chosen problem (1.5 pt). If there are existing solutions to the chosen problem, the submission should clearly describe their issues and highlight any positive aspect worth retaining in the future. When appropriate, examples of existing systems and practices can be used to support claims. If there are no existing solutions, the report should indicate explicitly so and report on the closest possible solution(s) that the team was able to find. We expect at least one related solution to be provided by each team member.

The collaboration serves to document what each member of the team worked on for each deliverable and encourage accountability within the team. The absence of this record in Canvas will result in a deduction of 10% of the assignment grade (0.5 pts).

☞ We recommend that the team prepares to succeed in the assignment by nominating a team member who will submit the pdf to Gradescope. Only one group member has to submit the group’s assessment – students must discuss and agree who will do this before the deadline. Keep a record in a shared notebook or collaborative document, and make sure you support each other by reminding yourselves when the deadline is approaching.

The person submitting the assignment will take responsibility for uploading the correct file, submitting it before the deadline, and for adding all other group members to the submission from a digital list.

Other group members should keep an eye on Gradescope to ensure that the correct file was submitted and the they were properly added to the submission.




Assignment 2: Design Research & Initial Ideation

Each group member should conduct one contextual inquiry or interview that provide insights into the perspectives, needs, and desires of potential users for their installation. Students should reflect on what was learned from this experience. Then, the group should summarize the insights from this need-finding process in their report and devise tasks (two per team member) that are integral to the overall design goal based on what has been learned about the chosen problem thus far.

Students should think carefully about the target population for their research methods (the contextual inquiries or interviews). This target population should be relevant to their display. That is, students should think of: Who will likely interact with the team's display on campus? These should be the people that students seek insight from for this assignment.

The submission for this assignment is a pdf document (with the students’ name and netid at the top) that needs to be submitted to Gradescope. A template in Google Docs is provided for students’ convenience. Importantly, students should complete each section of the assignment in a new page of the document in order to facilitate grading. The pdf should include the three items below in addition to the collaboration record:

  1. For each participant of a contextual inquiry or interview:
    • Which team member conducted the activity?
    • Who was observed or interviewed? (don’t provide their name, but demographics characteristics) What is their background? What was the environment?
    • What was learned from the research?
    • What tasks, problems, or opportunities were uncovered?
    • Were there difficulties establishing rapport or getting the desired information?

  2. A brief discussion of the high-level themes and problems that the participants shared.

    NOTE: If it is hard to identify high-level themes, problems, practices at this point, it is possible that additional understanding through more design research needs to be done. Students should ensure that their design research has provided insights and perspectives needed to proceed to the next stage of the project.

  3. What tasks are important to design for based on the themes/problems from above? Each team member should suggest 2 tasks each, such that maximum 1 task can overlap with a task proposed by another member. This means that the team will have between N to 2*N possible tasks to focus on moving forward, where N is the number of team members.

The collaboration serves to document what each member of the team worked on for each deliverable and encourage accountability within the team. The absence of this record in Canvas will result in a deduction of 10% of the assignment grade (1 pts).

☞ Similar to Assignment 1, we recommend that the team prepares to succeed in the assignment by nominating a team member who will submit the pdf to Gradescope — ideally, this person would be different from the one chosen for the prior assignment. The person submitting the assignment will take responsibility for uploading the correct file, submitting it before the deadline, and for adding all other group members to the submission from a digital list. Other group members should keep an eye on Gradescope to ensure that the correct file was submitted and the they were properly added to the submission.




Assignment 3: Low-fi Prototype

Students brainstorm and sketch distinct designs for their interface in order to explore the design space. More specifically, each student in the group should propose one design (short text description with one hand-drawn sketch) that addresses at least two tasks that are important for the problem space chosen by the group. These tasks are ideally from Assignment 2, although they may be modified if feedback was received by the course staff about how they could be improved. Then, the group should choose one design (i.e., one solution) to move forward with (potentially mixing multiple ideas). The team should also develop a low-fi (wireframe-style) prototype using Figma.

To reduce the chances that students will get distracted with aesthetic details about their low-fi prototype at this stage of their project, the Figma prototypes should be created in grayscale (the whole look of the installation design has to be created using black, gray tones, or white colors). Also, students are encouraged to use hand-drawn elements in their prototype and simple vector shapes rather than complex graphics. See this tutorial to get started with your prototype.

The submission for this assignment is a pdf document (with the students’ name and netid at the top) that needs to be submitted to Gradescope. A template in Google Docs is provided for students’ convenience. The pdf should include the items below in addition to the collaboration record:

  1. The proposed design by each student (short text description and sketch)

  2. A brief description summarizing the chosen design by the group. Please explain why the group thought that this was a good design given everything that the group has learned about their problem space and the constraints that they have for building their installation (e.g., physical location, display size, sensing capabilities, etc.).

  3. Pictures and a link to the Figma low-fi prototype.

☞ Similar to the prior assignments, we recommend that the team prepares to succeed in the assignment by nominating a team member who will submit the pdf to Gradescope — ideally, this person would be different from the one chosen for the prior assignments. Remember that this team member has to add all other members to the submission and that other members should double check that everything looks right in Gradecope (e.g., the right version of the assignment was submitted, etc.).




Assignment 4: Updated Prototype

Groups of students help evaluate each other’s prototypes. More specifically, a member of one group would provide heuristic evaluations for a prototype from another group, and receive evaluations from a member of another group as well. To facilitate this process, students have been grouped already into pairs. Each pair of students should use this evaluation sheet for ranking the other person’s prototype in accordance to the 10 heuristics discussed in class. Once all heuristic evaluations are completed, the group should use the feedback from other students to make improvements to their prototype from Assignment 3.

The submission for this asssignment is a pdf document (with the students’ name and netid at the top). The pdf needs to be submitted to Gradescope. A template in Google Docs is provided for students’ convenience.

The first section of the pdf submission should document changes resulting from the inspection-based analysis. The document should include a list of the results from the heuristic evaluations. For each identified issue, include in the report:

  • the name of the team member whose evaluator identified the issue,
  • an image of the relevant portion of the low-fi prototype,
  • a text description of the identified issue (including the heuristic that is violated),
  • the severity of the issue (0 for there being a violation but it not being a problem; 1 for “usability blemish”; 2 for minor usability problem; 3 for major usability problem; and 4 for critical usability problem), and
  • an image and explanation of the revision of the prototype that was implemented.

The second section of the submission should include a new overview image of the updated Figma prototype and a link to the new version in Figma. In addition to taking care of the issues identified from the heuristic evaluations, teams should now improve the visuals of their prototype in preparation for their system implementation (i.e., the next assignments). That is, students should increase the fidelity of their Figma prototype such that they can better plan for what needs to be done when implementing the actual system.

The third section of the submission should include a record about which prototypes were evaluated by team members. For each team member, please indicate the group number for the prototype that this person evaluated. Additionally, indicate the heuristics that were violated by the prototype that this team member inspected.

Finally, at the end of the report, include a collaboration record.

☞ Similar to the prior assignments, we recommend that the team prepares to succeed in the assignment by nominating a team member who will submit the pdf to Gradescope — ideally, this person would be different from the one chosen for the prior assignments. Remember that this team member has to add all other members to the submission and that other members should double check that everything looks right in Gradecope (e.g., the right version of the assignment was submitted, etc.).




Assignment 5: Prototype System Milestone (3%)

Students begin implementing a working prototype using spatial data from the installation setup. Prototyping should focus on designing and implementing key system components to support the chosen tasks.

The submission for this assignment is a pdf document (with the students’ name and netid at the top). The pdf needs to be submitted to Gradescope. A template in Google Docs is provided for students’ convenience. The pdf should include:

  1. A textual description of the system, the two tasks it addresses and its key implementation components. We recommend that you include a block system diagram in your answer to communicate your system components.

  2. A short discussion of the aspects of the system that were successfully implemented as planned and any challenges that were encountered during the implementation of the prototype components. Include screenshots of the existing system in your answer to communicate what you’ve been able to implement thus far. The screenshots should convey the state of your actual system implementation, not your high fidelity prototype from Assignment 4.

  3. A description of the aspects of the system that remain to be implemented.

  4. A plan (e.g., in the form of a schedule) of how the team will complete the prototype system. The plan description should indicate how the remaining activities (or items in the group’s to-do list) are divided among the group members. Explain why your distribution of next activities within the team is fair in your report.

Finally, at the end of the report, include a collaboration record. This collaboration record is worth 33% of the grade of this assignment because it is really important that all team members begin contributing to the implementation of the system prototype.

☞ Similar to the prior assignments, we recommend that the team prepares to succeed in the assignment by nominating a team member who will submit the pdf to Gradescope — ideally, this person would be different from the one chosen for the last assignment. Remember that this team member has to add all other members to the submission and that other members should double check that everything looks right in Gradecope (e.g., the right version of the assignment was submitted, etc.).




Assignment 6: Final Prototype System (10%)

The group finalizes the design and implementation of their prototype system. The group submits their final code as a zip file to Gradescope such that the corresponding teaching staff can upload the latest version to the TV setup before live project demos.

This assignment will be evaluated based on:
a) Whether the code can run on the display (1 pts)
b) The system prototype addresses the group’s two tasks from Assignment 5 (2 pts per task; 4 pts)
c) The code includes a README text file that explains how to install any dependencies and run the project (2 pts), provides a brief description of the project and what tasks the installation addresses (1 pts), indicates explicitly if there are any constraints from the deployment environment (0.5 pts), and includes a collaboration record that specifies what each team member contributed to the prototype (1.5 pts).

By including constraints from the deployment environment in the README, we mean that the text file should explain any physical constraints that are important to consider when the course staff evaluates the system prototype. For instance, would users need to be able to stand at least a minimum distance away from the Kinect for the application to run as intended?

Worth noting, Python projects must be implemented such that they can run in a virtual environment. These projects need to include in the source code a requirements.txt file that includes all Python dependencies (so that they can be instealled with pip as pip install -r requirements.txt).

☞ Similar to the prior assignments, we recommend that the team prepares to succeed in the assignment by nominating a team member who will submit the zip file with the code and README text file to Gradescope — ideally, this person would be different from the one chosen for the last assignment. Remember that this team member has to add all other members to the submission and that other members should double check that everything looks right in Gradecope (e.g., the right version of the assignment was submitted, etc.).




Assignment 7: User Testing and Video Presentation (8%)

Each student performs one user test of their final prototype (e.g., usability testing) and the group documents their project and final findings in a 3 minute video that summarizes their effort creating the installation. All video presentations will be showcased in the course website at the end of the semester.

The submission for this asssignment is twofold: 1) a pdf document (with the students’ name and netid at the top) that documents the findings from the user tests and includes a collaboration record; and 2) a short video (with the link included in the pdf document). The pdf should include for each user test:

  • Who conducted the test
  • Who was the participant (don’t provide their name but rather provide demographics and background information)
  • A summary of key findings from this test (both positive and negative).

The video should concisely describe:

  1. The goal of the project. What does the design aim to achieve?

  2. A short demo (approximately 1 min long) of the resulting system. Consider using short clips to demonstrate various aspects of the system or speeding up recordings of the system if necessary.

  3. A brief summary of the lessons learned in the design research process and prototype implementation.

Students have flexibility in terms of how they make their video as long as the final submission in an mp4 container with H.264 format, which is portable and easily playable. The video submission should not exceed 80MB in size.

Tip 1: HandBrake can be used for free to encode video as an mp4 container with H.264 format and reduce its file size.

Tip 2: The video can be made in Zoom using presentation slides (e.g., PowerPoint or Keynote), or using any other video editing software that the students are familiar with.

A template for this assignment submission is available for students’ convenience. Similar to all prior assignments, the answers to this assignment should be submitted to Gradescope as a pdf.

☞ Similar to the prior assignments, we recommend that the team prepares to succeed in the assignment by nominating a team member who will submit the zip file with the code and README text file to Gradescope — ideally, this person would be different from the one chosen for the last assignment. Remember that this team member has to add all other members to the submission and that other members should double check that everything looks right in Gradecope (e.g., the right version of the assignment was submitted, etc.).