(y) Virtual and Augmented Reality

Course Summary

VR and AR sits across graphics, tracking, computer vision, interaction, hardware, and human perception. The course pulls those threads together into systems that are responsive, comfortable, and registered to the real world.

Lecture Notes

• (y-00) Course Organization
  Course philosophy, structure, project, quizzes, and oral exam requirements.

• (y-01) Introduction and History
  VR/AR definitions, immersion and presence, history, applications, and the mixed reality continuum.

• (y-02) Human-Centered Design
  Iterative design, analysis, constraints, VR-specific requirements, prototyping, and testing.

• (y-03.1) Stereo Rendering and Hardware
  Depth cues, stereo rendering, parallax, off-axis projection, HMD optics, latency, and display challenges.

• (y-03.2) AR Displays and Input Hardware
  Optical/video see-through AR, display-space taxonomy, spatial AR, haptics, and multimodal input.

• (y-04) Interaction in VR and AR
  3D interaction tasks, selection, manipulation, navigation, system control, and AR-specific interaction.

• (y-05) Tracking in VR and AR
  Registration, calibration, tracking, coordinate systems, sensor technologies, optical tracking, and sensor fusion.

• (y-06) Computer Vision for AR
  Marker tracking, homographies, multi-camera IR tracking, natural features, SIFT, pose estimation, SLAM, and outdoor tracking.

• (y-07) Calibration and Registration
  Camera calibration (lens distortion, undistortion, targets), display calibration (shooting gallery, boresight, SPAAM, pointing device, eye tracking), hand-eye calibration, and registration errors from error propagation and latency.

• (y-08) Visual Coherence
  Occlusion (phantoms, edge refinement, probabilistic, depth, segmentation), illumination (environment maps, light probes, photometric registration), shadows (volumes, differential rendering, path tracing, photon mapping), reflections and refractions, diminished reality and stylized AR, and camera effects (lens distortion, vignetting, post-process emulation).

• (y-09) Perception and Psychology
  What perception is, why 2D graphical perception research does not transfer cleanly to 3D, depth and size underestimation, colour interference in AR and VR, visual attention and its limits (post-attentive amnesia, change and inattentional blindness), preattentive features and Feature Integration Theory, attention guidance in 2D and in AR/VR (added elements, blur, automatic locomotion, virtual actors, motion, dichoptic disparity), gestalt laws, and clinical applications of VR/AR psychology.

• (y-10) Adverse Health Effects in VR
  VR sickness umbrella (motion, simulator, cybersickness), theories of motion sickness (sensory conflict, evolutionary, postural instability, rest frame, unified model), eye strain and accommodation-vergence conflict, readaptation, physical fatigue, fit and hygiene, injury, latency sources and compensation, system / application / individual factors, measurement (SSQ, postural stability, physiological), and design guidelines for reducing adverse effects.

• (y-11) Immersive Analytics
  Immersive analytics at the intersection of data visualization and VR/AR, visual analytics versus data-driven storytelling, brief history (Super Cockpit, scientific versus abstract data, term coined 2015), eight advantages over the desktop (extra visual channel, immersion, spatial memory, collaboration, embodiment, multimodal input, hybrid interfaces, situated visualization), the evaluation and 3D-for-infovis debates, and situated visualization (definition, situated versus situated analytics, non-situated / situated / embedded, forms and examples, enabling technologies, constraints).

• (y-12) Haptics
  Tactile versus kinesthetic feedback and the skin’s receptor types, grounded versus ungrounded devices (full trade-off table), active versus passive feedback, the device taxonomy (vibration, pressure, skin stretch, propeller, string, mechanical arm, encountered-type, EMS, gloves, pseudo), pseudo-haptics and haptic retargeting, automotive applications (Strives), and the perfect-system question.

• (y-13) Evaluation and Future Trends
  Experiments on human subjects (variables, ethics), evaluation of versus with VR/AR, the method toolbox, operationalizing hypotheses into measures, the novelty effect, VR sickness and the SSQ, SUS and NASA TLX, interviews and the limits of self-report, pilot studies and good practices, descriptive versus inferential statistics, control and confounding variables, the asymmetric AR assembly case study, and future trends (displays, more senses, smart objects, robots, brain-machine interfaces, ML/AI, semantic scene understanding, standards).

Assessment Map

(y) Back to Notes