Overview

This is an opportunity for students to design and implement an interactive system system based on spatial user data and/or using a Large Language Model (LLM). By spatial data we mean information about the pose of pedestrians on an indoor building on campus (e.g., AKW or Becton). This spatial data will be captured by a Kinect Azure sensor, which has a color and a depth camera. The sensor SDK can detect and track individuals in front of it. If student opt for implementing an interactive system with an LLM, then they should use the free Gemini Pro API by Google.

In addition to the Kinect sensor, the hardware for the system includes a big TV and a computer and keyboard (hidden in the image above) and push-to-talk microphones. The computer will read the data from the sensor, complement its tracking information with conversational group predictions (using the model from this paper), and output the information on a web server for you to use in your system. The computer will also run the interactive system and display its visual output on the TV.

The interactive system should be designed from a user-centered perspective, aiming to provide value to users based on findings from design research. The project focuses on systems that highlight moral principles, help users understand real-time or historic spatial data, or provide novel opportunities to relate nearby pedestrians. Example systems that fit the theme for this project can be seen in this website and this other website, which showcases projects from the Spring 2021 and 2023 edition of this course.

As part of the project, students need to identify the activities that their proposed design will support and the type of questions that the system would allow users to answer. If there are existing solutions for the chosen activites/questions, it is important to identify what does not work for them and how the proposed design provides better support.

We expect to have 40 projects total with teams of 4 students each (or 3 students in a few cases depending on final enrollment). The groups will be defined by the course staff, based on student’s responses to this form (responses can be provided up until Thursday, Jan 25 at 2:30pm).

Technical Details

There are 4 installation setups, each with a TV display, Kinect sensor, push-to-talk microphones and a computer to run students’ prototype systems. Each project group will be assigned to one installation setup for the whole semester.

The computer driving a display runs Ubuntu 20.04. In this computer, there will be software that collects data from the installation setup and makes it accessible via a websocket connection within Yale’s network. The data includes estimated human poses for the pedestrians visible from the Kinect (in the form of skeleton keypoints), group predictions (e.g., based on spatial behavior) and text stream of the user’s speech (any utterances made while using push-to-talk microphones). Students are expected to implement interfaces that leverage this data using web technologies (HTML, Javascript). Python can be used to create local test environments for the project, as in the demos below:

The applications developed by students will be shown in a Chrome web browser window that is displayed in full-screen mode on the corresponding TV. We recommend using Processing JS for composing different types of graphics and creating animations.

If students have no prior experience with JavaScript (JS) or object oriented programming, we recommend this tutorial by Mozilla. In particular, this section is helpful ƒor understanding object-oriented programming with JS.

Questions? The course staff created this Google Doc for common/general Q&A related to the system prototype (including working with GitHub). Students may also post questions about their implementation in Ed Discussion but please follow the instruction in the Google Doc when posting these messages.

☞ Please see this tutorial page for more technical information about the displays.

Project Components

Late Policy: Any assignment submitted after the official deadline, will receive a penalty of 50% of the assignment grade. Assignments will not be accepted more than 24h after the deadline.

NOTE: For all assignments, we encourage teamwork, positive interdependence, and individual accountability. Students will have to report on their team collaboration and individual contributions on each assignment.



Assignment 1: Design Research & Initial Ideation (10% of the grade, due Feb. 6 at 12pm ET)

Given the project brainstorming that took place in class (and additional brainstorming if necessary), the group should choose a problem space to investigate further in this project. Also, each group member should conduct one contextual inquiry or interview that provide insights into the perspectives, needs, and desires of potential users for an installation focused on this problem space. Students should reflect on what was learned from this experience. Finally, the group should summarize the insights from this need-finding process in their report and devise tasks (two per team member) that are integral to the overall design goal based on what has been learned about the chosen problem thus far.

Students should think carefully about the target population for their research methods (the contextual inquiries or interviews). This target population should be relevant to their display. That is, students should think of: Who will likely interact with the team's display on campus? These should be the people that students seek insight from for this assignment.

The submission for this assignment is a pdf document (with the students’ name and netid at the top) that needs to be submitted to Gradescope. A template in Google Docs is provided for students’ convenience. Importantly, students should complete each section of the assignment in a new page of the document in order to facilitate grading. The pdf should include the three items below in addition to the collaboration record:

  1. A chosen problem space to investigate further in the semester-long project (0.75 pt). The description of the problem space should include:
    • an explanation of what the problem space is (0.25 pt),
    • why it is an interesting problem space (0.25 pt), and
    • why it is not trivial to solve (0.25 pt).

    NOTE: If the team does not have ideas for what problem spaces to work on, they should brainstorm further!

  2. Findings from user research (4 pts). For each participant of a contextual inquiry or interview:
    • Which team member conducted the activity?
    • Who was observed or interviewed? (don’t provide their name, but demographics characteristics) What is their background? What was the environment?
    • What was learned from the research?
    • What tasks, problems, or opportunities were uncovered?
    • Were there difficulties establishing rapport or getting the desired information?

  3. A brief discussion of the high-level themes and problems that the participants shared (1 pts).

    NOTE: If it is hard to identify high-level themes, problems, practices at this point, it is possible that additional understanding through more design research needs to be done. Students should ensure that their design research has provided insights and perspectives needed to proceed to the next stage of the project.

  4. Tasks descriptions (2 pts). What tasks are important to design for based on the themes/problems from above? Each team member should suggest 2 tasks each, such that maximum 1 task can overlap with a task proposed by another member. This means that the team will have between N to 2*N possible tasks to focus on moving forward, where N is the number of team members.

A collaboration record must be provided to document what each member of the team worked on for each deliverable and encourage accountability within the team (2.25 pts).

☞ We recommend that the team prepares to succeed in the assignment by nominating a team member who will submit the pdf to Gradescope. Only one group member has to submit the group’s assessment – students must discuss and agree who will do this before the deadline. Keep a record in a shared notebook or collaborative document, and make sure you support each other by reminding yourselves when the deadline is approaching.

The person submitting the assignment will take responsibility for uploading the correct file, submitting it before the deadline, and for adding all other group members to the submission from a digital list.

Other group members should keep an eye on Gradescope to ensure that the correct file was submitted and the they were properly added to the submission.




Assignment 2: More Research & Low-fi Prototype (10%, due Feb. 19 at 8pm ET)

Students identify related solutions to their problem space. Then, they brainstorm and sketch distinct designs for their interface in order to explore the design space. More specifically, each student in the group should propose one design (short text description with one hand-drawn sketch) that addresses at least two tasks that are important for the problem space chosen by the group. These tasks are ideally from Assignment 1, although they may be modified if feedback was received by the course staff about how they could be improved. Then, the group should choose one design (i.e., one solution) to move forward with (potentially mixing multiple ideas). The team should also develop a low-fi (wireframe-style) prototype using Figma.

Students are not allowed to create their hand-drawn sketches with Generative AI tools (e.g., Stable Diffusion or ChatGPT or Bard). Part of the goal of sketching by hand in this assignment is to help students think about their design. This thought process naturally occurs as people sketch out ideas by hand.

To reduce the chances that students will get distracted with aesthetic details about their low-fi prototype at this stage of their project, the Figma prototypes should be created in grayscale (the whole look of the installation design has to be created using black, gray tones, or white colors). Also, students are encouraged to use hand-drawn elements in their prototype and simple vector shapes rather than complex graphics. See this tutorial to get started with your prototype.

The submission for this assignment is a pdf document (with the students’ name and netid at the top) that needs to be submitted to Gradescope. A template in Google Docs is provided for students’ convenience. The pdf should include the items below in addition to the collaboration record:

  1. A brief description of related solutions to the chosen problem (1 pt). If there are existing solutions to the chosen problem, the submission should clearly describe their issues and highlight any positive aspect worth retaining in the future. When appropriate, examples of existing systems and practices can be used to support claims. If there are no existing solutions, the report should indicate explicitly so and students should report on the closest possible solution that they were able to find. One related solution should be provided by each team member.

  2. The proposed design by each student (short text description and sketch)

  3. A brief description summarizing the chosen design by the group. Please explain why the group thought that this was a good design given everything that the group has learned about their problem space and the constraints that they have for building their installation (e.g., physical location, display size, sensing capabilities, etc.).

  4. Pictures and a link to the Figma low-fi prototype.

☞ We recommend that the team prepares to succeed in the assignment by nominating a team member who will submit the pdf to Gradescope — ideally, this person would be different from the one chosen for the prior assignment. The person submitting the assignment will take responsibility for uploading the correct file, submitting it before the deadline, and for **adding all other group members** to the submission from a digital list. Other group members should keep an eye on Gradescope to ensure that the correct file was submitted and the they were properly added to the submission.




Assignment 3: Updated Prototype (5%, two deliverables)

Groups of students help evaluate each other’s prototypes. More specifically, a member of one group would provide heuristic evaluations for a prototype from another group, and receive evaluations from a member of another group as well. To facilitate this process, students have been grouped already into pairs and we will dedicate class time to do the evaluation (on Thursday Feb. 22). Further, all groups have provided information about problem space, tasks and any additional request(s) for the evaluators:

Once the heuristic evaluations are completed or the deadline to evaluate them has passed (Feb. 26 at 8pm), the groups should use the feedback from other students to make improvements to their prototype from Assignment 2.

The submission for Assignment 3 is organized into two deliverables:

Deliverable I (deadline: Mon., February 26th at 8pm): Students must complete an online gradescope assignment individually to confirm that 1) they completed the heuristic evaluation considering the tasks and request(s) made by the group that created the protototype, 2) confirm that they emailed the group that authored the prototype, and 3) submit a report of the heuristic violations so that the course staff can grade it. The report must be uploaded to gradescope as a pdf document. To facilitate this part of the assignment, students can use this report template.

NOTE: It is possible for an issue in an interface to correspond to multiple heuristic violations. In that case, students should document all the violations that they identified for the issue in their report. Also, there may be issues with a prototype that are not heuristic violations. In that case, please use the last section of the report template to provide any additional feedback to the authors of the prototype.

Deliverable II (deadline: Fri., March 1st at 8pm): The group must submit a pdf document (with the students’ name and netid at the top). The pdf needs to be submitted to Gradescope. A template in Google Docs is provided for students’ convenience.

The first section of the pdf submission should document changes resulting from the inspection-based analysis. The document should include a list of the results from the heuristic evaluations. For each identified issue, include in the report:

  • the name of the evaluator who identified the issue,
  • an image of the relevant portion of the low-fi prototype,
  • a text description of the identified issue (including the heuristic that is violated),
  • the severity of the issue per this evaluation sheet (0 for there being a violation but it not being a problem; 1 for “usability blemish”; 2 for minor usability problem; 3 for major usability problem; and 4 for critical usability problem), and
  • an image and explanation of the revision of the prototype that was implemented.

The second section of the submission should describe your problem space, the tasks that you are focusing your design on, and include a new overview image of the updated Figma prototype and a link to the new version in Figma. In addition to taking care of the issues identified from the heuristic evaluations, teams should now improve the visuals of their prototype in preparation for their system implementation (i.e., the next assignments). That is, students should increase the fidelity of their Figma prototype such that they can better plan for what needs to be done when implementing the actual system. Using color is now allowed. You can worry about details like specific font choices, icons, etc. that will be included in your final prototype.

NOTE: Students can update their problem space and tasks in this assignment to a) more clearly convey what problem space they are focusing on, b) better describe their tasks so that it’s more clear how they are focusing on user actions that contribute to a useful objective in relation to the problem space, or c) slightly pivot in their project based on the feedback that has been received thus far from the course staff, other students, and their own learning throughout the semester.

Finally, at the end of the report, include a collaboration record.

☞ Similar to the prior assignments, we recommend that the team prepares to succeed in the assignment by nominating a team member who will submit the pdf to Gradescope — ideally, this person would be different from the one chosen for the prior assignments. Remember that this team member has to add all other members to the submission and that other members should double check that everything looks right in Gradecope (e.g., the right version of the assignment was submitted, etc.).




Assignment 4: Prototype System Milestone (6%, due Wednesday April 10 at 8pm ET)

Students begin implementing a working prototype, which should take user input either via spatial data or using the microphones attached to the displays in the installation setup. Use of the Gemini LLM is optional.

Prototyping should focus on designing and implementing key system components to support two critical tasks for the chosen problem domain. Supporting more tasks is allowed, but discouraged because groups only have 1 month to complete the prototype (through Assignments 4 and 5). Two critical tasks should be enough to scope the prototype in a way that a) the group can show a tangible solution to address their problem space b) users can test them on a display.

Each group should begin to develop their prototype for their corresponding TV. There are 4 TVs in total: TV1 is in the 3rd floor of AKW, TV2 is in the entrance of Becton from Prospect Street, TV3 is outside of the Davies auditorium, and TV4 is by the entrance of 17 HH. Groups 1-9 are assigned to TV1, groups 10-18 are assigned to TV2, groups 19-27 are assigned to TV3, and groups 28-36 are assigned to TV4.

The submission for this assignment is two-fold: 1) students should submit a pdf document (with the students’ name and netid at the top); and 2) students should submit their existing code for the prototype following the “Step 1: Upload your project” instructions in the “Upload Your Project” section of the tutorial page. The pdf needs to be submitted to Gradescope. A template in Google Docs is provided for the pdf submission for students’ convenience. The pdf should include:

  1. A brief description of the group’s problem space, the tasks that the prototype addresses, a textual description of the proposed solution (the prototype that is being built), and a description of the prototype’s key implementation components. We recommend that you include a block system diagram in your answer to communicate your system components.

  2. A short discussion of the aspects of the system that were successfully implemented as planned and any challenges that were encountered during the implementation of the prototype components. Include screenshots of the existing system in your answer to communicate what you’ve been able to implement thus far. The screenshots should convey the state of your actual system implementation, not your high fidelity prototype from Assignment 3.

  3. A description of the aspects of the prototype system that remain to be implemented.

  4. A plan (e.g., in the form of a schedule) of how the team will complete the prototype system. The plan description should indicate how the remaining activities (or items in the group’s to-do list) are divided among the group members. Explain why your distribution of next activities within the team is fair in your report.

  5. A collaboration record. It is really important that all team members begin contributing to the implementation of the system prototype.

The code should be submitted as a zip file. Follow the Step 1 in the tutorial page to upload your code. Note that the the zip file has specific requirements per the /remote page of the group’s display (which is shown in the image below):

You do not need to complete Step 2 of the upload instructions for Assignment 4 – you only need to upload your code to the display, not test it on the display. You will be testing your code on the display for Assignment 5.

☞ If a project is using the Gemini API, then the group should read the Google terms for the API to ensure that they use the API per the official terms. Also, a disclaimer must be added to the prototype interface that says three important things:

1. The Gemini API by Google is being used by the application.
2. Information generated by the Gemini API may be erroneous, and this can result in errors in the application.
3. Information provided to the application may be sent to Google, and Google may use this data to improve and develop Google products.

☞ Similar to the prior assignments, remember that the team member that uploads the report to Gradescope has to add all other members to the submission and that other members should double check that everything looks right in Gradecope (e.g., the right version of the assignment was submitted, etc.).




Assignment 5: Final Prototype System (10%)

The group finalizes the design and implementation of their prototype system. The group submits their final code as a zip file to Gradescope such that the corresponding teaching staff can upload the latest version to the TV setup before live project demos.

This assignment will be evaluated based on:
a) Whether the code can run on the display (1 pts)
b) The system prototype addresses the group’s two tasks from Assignment 5 (2 pts per task; 4 pts)
c) The code includes a README text file that explains how to install any dependencies and run the project (2 pts), provides a brief description of the project, problem space, and what tasks the installation addresses (1 pts), indicates explicitly if there are any constraints from the deployment environment (1 pts), and includes a collaboration record that specifies what each team member contributed to the prototype (1 pts).

By including constraints from the deployment environment in the README, we mean that the text file should explain any physical constraints that are important to consider when the course staff evaluates the system prototype. For instance, would users need to be able to stand at least a minimum distance away from the Kinect for the application to run as intended?

Worth noting, we expect the projects to either:

  • Have an index.html file is present in the top-level folder.
  • Have a text file named web is present in the top-level folder to indicate the path of the homepage for the website within the source code. This file should contain the url to access the application. See this post as an example.

☞ Similar to the prior assignments, we recommend that the team prepares to succeed in the assignment by nominating a team member who will submit the zip file with the code and README text file to Gradescope — ideally, this person would be different from the one chosen for the last assignment. Remember that this team member has to add all other members to the submission and that other members should double check that everything looks right in Gradecope (e.g., the right version of the assignment was submitted, etc.).




More assignments will appear here in the future… Their description has been removed from here to avoid confusion about which assignments are officially out.