Slide1 : Computer Vision in Graphics Production
Visual Media Research Group
Centre for Vision, Speech and Signal Processing
University of Surrey
Slide2 : Overview Where is computer vision useful in content production?
Where is vision used?
What can and can’t current computer vision do?
Slide3 : Computer Vision in Content Production Computer vision: analysis & interpretation of real images/video
Match moving
Matting
Model building
Human motion capture
Studio Production
Facial animation
Image-based illumination
Slide4 : Match Moving Reconstruction of camera movement to composite CG
Products: 2d3, RealViz … (semi-automatic camera tracking)
… standard tool in film production Oxford University/2d3
Slide5 : Matting Separation of foreground and background objects
- actor/background separation
- wire/set removal
Studio: chroma-key (solved)
Post-production: Imagineer, RealViz, …. (open-problem)
Slide6 : Model Building Capture of real shape for CG models
objects
environments
characters
Shape capture
active sensors (laser/structured-light)
- accurate 3D surface measurements
- static objects or environments
structure from images
- low-accuracy
- static objects
Problem: unstructured surface measurements
Slide7 : Model Building (2) Walking with Dinosaurs - FrameStore
Slide8 : Structured Models
Slide9 : Courtesy Stanford Computer Graphics Lab. Animation of David
Slide10 : Human Motion Capture Capture of real motion
Marker based systems
widely used in performance animation whole-body/face
character animation (Golum)
‘realistic’ motion characteristics
accurate
real-time?
widely used in film production (with post-production)
Slide11 : Human Motion Capture(2) Markerless human motion capture
advantages: unintrusive; simultaneous appearance capture
model-based visual tracking
low-accuracy
visual ambiguity (uniform apperance, non-rigid shape)
Slide12 : Human Motion Capture(3)
Slide13 : Studio Production Modelling actors
3D animated models
animation
re-illumination
loss of visual realism to captured images
Free-viewpoint video
post-production of arbitrary camera views/paths
‘matrix’ flowmo shots
aim: quality equivalent to captured video
Slide14 : Studio Production (2)
Slide15 :
Slide16 : Limitations:
- shape detail face/hands/hair
- not video quality Studio Production (3)
Slide17 : Studio Production: Free-view video
Slide18 : Studio Production: Free-view video
Slide19 : Studio Production: Free-view video
Slide20 : Studio Production: Free-view video
Slide21 : Facial Animation Production of photo-realistic faces
2D video-based approaches
video rewrite [Bregler’97]
concatenative synthesis [Cossato’98,Ezzat’02]
photo-realistic
limited viewpoint, illumination
3D markers
Performance animation [Phigin’98]
shape only
3D video (shape+appearance)
concatenative synthesis
photo-realistic
control of viewpoint, illumination
Slide22 : 3D Video
Slide23 : 3D Video Database 51 people: Expressions + Speech (short sentence)
Slide24 : Initial Face Synthesis from Speech Input: Output:
Slide25 : Image-based illumination Illumination of CG scenes with real illumination
high-dynamic range capture of illumination
environment mapping illumination to CG scene
widely used in production Debevec SIGGRAPH’00 FiatLux
Slide26 : Computer Vision in Content Production Match moving - yes, automatic camera tracking available
Matting - partial, many unsolved problems
Model capture – yes, tools for semi-automatic restructuring
Human motion capture – no, inaccurate
Studio Production – partial, free viewpoint video
Facial animation – yes, 3D video
Image-based illumination – yes, widely used
Other applications of computer vision….
Slide27 : Future Directions Video-based rendering
photo-realistic, free-viewpoint rendering of dynamic scenes
relighting required
Video-based animation
animation from captured video (face,whole-body…)
control of motion, viewpoint, illumination
Slide28 : People
Jon Starck, Joel Mitchelson, Gordon Collins,
Eng-Jon Ong, Ioannis Ypsilos, Rob Dilks,
Michael Kalkavouras Collaborators
BBC, BT, Sony, Canon, Philips,
Hensons, Framestore, Snell&Wilcox,
3D Scanners, AvatarMe
EPSRC – DTI