MOTION CAPTURE OF A RED-WINGED BLACKBIRD IN FREE FLIGHT
A Thesis Presented to the Faculty of the Graduate School
of Cornell University in Partial Fulfillment of the Requirements for the Degree of
Master of Science
by Brendan Alec Holt
May 2009

© 2009 Brendan Alec Holt ALL RIGHTS RESERVED

ABSTRACT
We developed a motion capture system for the purpose of reconstructing the deforming wing surfaces of a bird performing sustained flapping flight. Our facility consisted of a 13-m long free flight tunnel which hosted two calibrated 500-fps digital cameras and multiple high powered lamps. The experimental setup was tested with several species. The best results were obtained with a wild, immature red-winged blackbird (Agelaius phoeniceus) whose wing joints and remiges were labeled with 3-mm retroreflective markers. For each camera’s recording of the flight we digitized the 2D marker paths and from them reconstructed the 3D trajectories using the direct linear transformation method. We fit a virtual wing model consisting of bones and feathers to the marker trajectories and extracted approximate joint kinematics from the shoulder, elbow, and wrist. These time-dependent angle data of the joints compared favorably with those of the European starling (Sturnus vulgaris) obtained in wind tunnel experiments [DJJ91]. The wing model also matched the silhouettes of the red-winged blackbird when superimposed over the original footage. Although more experiments are needed to verify the accuracy of our method, future results would find diverse application in simulating the complex aerodynamics of bird flight, quantitatively comparing flight behaviors among different species for the benefit of ornithology, and animating virtual birds with heightened realism in the visual effects industry.

BIOGRAPHICAL SKETCH Brendan Holt was born on October 3rd, 1983 in Boston and grew up in the suburbs along the outer beltway of Massachusetts’ capital. In 2006, he graduated from Cornell University with a B.S. in Biological and Environmental Engineering, but not before he was bitten by the animation bug during his senior year. After experiencing Professor Donald Greenberg’s animation courses, the visual world of computer graphics became a new passion. His undergraduate friends attempted to warn him that staying another few years in Ithaca’s frigid gorges would lead to permanent hypothermia or worse—nevertheless, he enthusiastically joined the Cornell Program of Computer Graphics and earned his master’s degree in May 2009.
iii

ACKNOWLEDGEMENTS
Don Greenberg has been a terrific advisor. His multitude of interests and enthusiasm for interdisciplinary work have made this project all the more exciting. Don taught me to try out new ideas without worrying about their success or failure—every experiment was an opportunity to learn. Kim Bostwick from the Lab of Ornithology spent countless hours teaching us about birds and how to handle them. She gave us insights for our experimental design, and we braved all sorts of weather to fix the flight tunnel and run motion capture trials. Through this collaboration I have gained a new appreciation for the avian species.
We thank David Winkler for the use of his flight tunnel, and for granting all of our requests to modify that venerable structure, and for tolerating our construction projects that became accidents. In addition, we are grateful to the Cornell Laboratory of Ornithology for lending us their high speed cameras. Another key tool came from Ty Hedrick, who generously made his digitizing software available to us, without which this project would have been far more difficult. My minor advisor, Doug James, sparked my interest in physically based animation and gave numerous helpful suggestions for improving our analyses and the presentation of our results. Near the onset of our project investigation, Dan Riskin gave us a tour of his bat facility and wind tunnel at Brown University, and introduced us to the calibration, digitization, and reconstruction workflows. Thanks to Francisco Valero-Cuevas for his early advice on the project and for the knowledge I gained from his inspiring Neuromuscular Biomechanics course.
Jeff Wang showed me the ropes during my first year of graduate school. He also devoted a lot of his time to this project and was always there to lend a hand and an encouraging word. Jeff’s expertise in photography saved us many hours when setting up a working camera system. Fellow office mate and machine
iv

shop wizard David Kaplan transformed chunks of scrap aluminum into incredibly useful light mounts for our camera setup. More importantly, he provided most of the energy snacks necessary to get through those late nights of brainstorming on projects. Whenever I reached the limits of my numerical methods knowledge, Konstantin Shkurko was always happy to point me in the right direction. Todd Harvey suggested a critical change in our light rig which greatly improved the visibility of the bird. Undergraduate researcher Leeann Louis, along with Jeff, David, and Todd all assisted on numerous occasions with the lengthy process of catching, marking, and flying birds in the flight tunnel.
Hurf Sheldon and Martin Berggren gave indispensable technology support for our research and teaching duties, and they supplied us with ample workshop tools for upgrading the flight tunnel. Over lunch I enjoyed listening to Hurf’s adventurous tales of war, formula car racing, and plasma physics. Thanks especially to Linda Stephenson and Peggy Andersen for their administrative help, invitations to go skiing, and friendly reminders to “finish up!”
Gamma Alpha Co-op has been a home away from home, and the students there have become like a second family, looking out for each other. I will never forget the wonderful home-cooked meals that would await every night when coming back from a long day’s work. Finally, I’d like to thank my parents and siblings for their love, support, and encouragement these past few years.
This research was funded by the Program of Computer Graphics, the Department of Architecture, and the National Science Foundation ITR/AP CCF-0205438. No birds were harmed during the course of this project.
v

CONTENTS

Biographical Sketch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

1 Introduction

1

2 Background and Previous Work

3

2.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.1 Wing morphology . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.2 Direct linear transformation method . . . . . . . . . . . . . 6

2.2 Motion capture of vertebrate flight . . . . . . . . . . . . . . . . . . 11

2.2.1 Kinematic and dynamic studies . . . . . . . . . . . . . . . . 13

2.3 Related works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.3.1 Motion capture of deformable surfaces . . . . . . . . . . . . 21

2.3.2 Simulating winged flight . . . . . . . . . . . . . . . . . . . . 24

3 Flight Tunnel and Motion Capture Methods

28

3.1 Equipment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.1.1 Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.1.2 Calibration cube . . . . . . . . . . . . . . . . . . . . . . . . 33

3.1.3 Markers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3.1.4 Lights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.2.1 Obtaining a bird . . . . . . . . . . . . . . . . . . . . . . . . 36

3.2.2 Marking the bird . . . . . . . . . . . . . . . . . . . . . . . . 37

3.2.3 Flight and retrieval . . . . . . . . . . . . . . . . . . . . . . . 38

4 Data Processing

41

4.1 Camera calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.2 Marker digitization . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

4.3 Interpolation of image-space marker trajectories . . . . . . . . . . . 45

4.4 Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5 Results

49

5.1 Marker trajectories . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.2 Wing visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

5.3 Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

vi

6 Discussion

63

6.1 Data analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

6.2 Proposed improvements . . . . . . . . . . . . . . . . . . . . . . . . 67

6.2.1 Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

6.2.2 Bird selection . . . . . . . . . . . . . . . . . . . . . . . . . . 68

6.2.3 Recording technology . . . . . . . . . . . . . . . . . . . . . . 69

6.2.4 Markers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

6.2.5 Marker layout . . . . . . . . . . . . . . . . . . . . . . . . . . 72

6.2.6 Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . 74

6.2.7 Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . 74

6.3 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

6.3.1 Model improvements . . . . . . . . . . . . . . . . . . . . . . 76

6.3.2 Research questions and applications . . . . . . . . . . . . . . 77

A Anatomical Measurements

80

vii

LIST OF TABLES 3.1 High speed camera specifications . . . . . . . . . . . . . . . . . . . 31 A.1 Measured arm bone lengths of A. phoeniceus specimens. . . . . . . 80 A.2 Measured flight feather lengths from a plucked wing . . . . . . . . 82
viii

LIST OF FIGURES
2.1 Internal structure of the wing . . . . . . . . . . . . . . . . . . . . . 4 2.2 Ventral wing musculature . . . . . . . . . . . . . . . . . . . . . . . 5 2.3 Simple camera model . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.4 Epipolar geometry of two cameras . . . . . . . . . . . . . . . . . . 8 2.5 Alignment of a 3D bird skeleton to cineradiography . . . . . . . . . 12 2.6 Two-triangle wing model . . . . . . . . . . . . . . . . . . . . . . . . 14 2.7 Four-camera setup for a turtle-dove flying in a wind tunnel . . . . 15 2.8 Moments of inertia and masses of cockatiel wing segments . . . . . 16 2.9 Deflections of steppe eagle wing coverts . . . . . . . . . . . . . . . 17 2.10 Vertebral, global, and gravitational frames of reference . . . . . . . 18 2.11 Fundamental wing beat constancy . . . . . . . . . . . . . . . . . . 19 2.12 Decomposing complex motion into constituent orthogonal modes . 20 2.13 Marker-based estimation of facial muscle activations . . . . . . . . 22 2.14 Estimating missing motion capture data with neighboring markers 23 2.15 Bird model for simulating physically based flight . . . . . . . . . . 25 2.16 Animating the ivory-billed woodpecker . . . . . . . . . . . . . . . . 26
3.1 Flight tunnel and early camera setup . . . . . . . . . . . . . . . . . 28 3.2 Preliminary video footage of a red-winged blackbird in level flight . 30 3.3 Flight tunnel schematic (top view) . . . . . . . . . . . . . . . . . . 31 3.4 Sample video frames from usable flight data . . . . . . . . . . . . . 32 3.5 Calibration object . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.6 Marker layout of the avian wing and body . . . . . . . . . . . . . . 39
4.1 Calibration tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 4.2 DLT calibration residual diagnostic . . . . . . . . . . . . . . . . . . 44 4.3 Digitizing the wing markers in both camera views . . . . . . . . . . 45 4.4 Interpolating 2D data to reconstruct a continuous 3D trajectory . . 46 4.5 Interpolation of the manus marker trajectory with smoothing splines 47
5.1 Wing marker trajectories over a flap cycle . . . . . . . . . . . . . . 50 5.2 Looped marker trajectories for the leading edge of the wing . . . . 51 5.3 Deforming wing mesh, looped over one flap cycle . . . . . . . . . . 52 5.4 Bone lengths inferred from joint marker distances . . . . . . . . . . 54 5.5 Modeling simplifications for the wing bones and feathers . . . . . . 55 5.6 Interremigial ligament aids in spreading and folding the remiges . . 56 5.7 Fitting a bone-and-feather wing model to the marker data . . . . . 57 5.8 Snapshots of the fitted wing model . . . . . . . . . . . . . . . . . . 59 5.9 Fitted wing model superimposed over the video data . . . . . . . . 60 5.10 Inferred wing joint kinematics . . . . . . . . . . . . . . . . . . . . . 61
6.1 Comparison of A. phoeniceus and S. vulgaris joint kinematics . . . 64 6.2 Natural, trackable wing markings (hairy woodpecker) . . . . . . . . 69
ix

6.3 Marker layout for tracking deformations of a bat wing . . . . . . . 73 A.1 Arm skeleton and feathered wing of an A. phoeniceus specimen . . 81
x

CHAPTER 1 INTRODUCTION
Bird flight is a difficult phenomenon to describe accurately. Vertebrate morphologists have borrowed methods from biomechanics and aerospace engineering to contain the bird and dissect every component of its motion. Most experiments use a wind tunnel to “freeze” its horizontal velocity and high speed cameras to effectively “slow down” the rapid wing beat. To date, most studies have tracked only a sparse set of points on the wing, and arguably a wind tunnel is not a natural flying environment due to reflecting airwakes [HTB02]. Computer graphics researchers and visual effects artists have reproduced the likenesses of flying birds via physical simulation or animation. Although the ability to simulate birds is an attractive prospect, examples in the literature lack ground truth measurements. Instead, single-view video footage for subjective evaluation is frequently used.
Our goal was to design and test a motion capture system and to create an accompanying visualization for a songbird in free flight. Such tools should reveal the rigid motion of the wing bones and the bending behavior of the feathers. In more technical terms, we wish to capture the time-dependent deforming geometry of the bird wing and effectively reconstruct the joint kinematics of level flapping flight in selected passerines.1 Since data from marker-based systems are not easy to interpret, we needed to create a virtual wing model consisting of an arm skeleton and simplified feathers whose motion is guided by the marker trajectories. A successful motion capture system for wild birds would be a boon to the animation industry, which can use the data to create more realistic flight simulations. For ornithologists, the kinematics of the wing could elucidate how birds fly from mechanical and evolutionary standpoints.
1An order of birds (Passeriformes) commonly known as songbirds.
1

Several factors make traditional 3D motion capture equipment ineffective or difficult to use with wild birds. Considerable space is required because of the birds’ fast overall body velocities during flight. Songbird flapping rates are typically 1216 cycles per second, so sensitive cameras with high frame rates are necessary. The wings are deformable surfaces whose feathers mask their underlying bone structure. Lightweight, unobtrusive markers are needed for tracking hidden joints and other landmarks, but they must not interfere with the aerodynamics of the wing or the future health of the bird. The data will have some degree of noise and error due to limited camera views and occluded markers blocked by neighboring feathers. But these obstacles are not insurmountable.
We begin with an overview of avian wing anatomy with comments on the structures responsible for its deformation during flight. This brief biological description is followed by an abridged derivation of the direct linear transformation method, an algorithm which in this context converts 2D imagery from multiple cameras into 3D data. This background information motivates a review of avian motion capture studies and recent methods of bird flight simulation in the computer graphics literature. Chapter 3 describes the flight tunnel and camera equipment, as well as the procedure for catching, marking, and flying birds in the facility. In Chapter 4, we outline our post-processing methods for digitizing the wing markers, repairing gaps in the data, and reconstructing 3D trajectories from the 2D marker data. We present the data from a red-winged blackbird capture session in Chapter 5, and visualize the results with a simple bone-and-feather wing model fit to the marker trajectories. In Chapter 6, we compare our model’s joint kinematics to those of another flight study and propose various improvements for our motion capture methodology. Finally, we close with an outlook on how this work could be applied in the fields of experimental biology, the animation industry, and education.
2

CHAPTER 2 BACKGROUND AND PREVIOUS WORK
2.1 Background
2.1.1 Wing morphology
In order to capture the deforming motion of a flapping bird wing, it is first necessary to understand the internal structure, mechanical properties, and typical behavior of the wing. This knowledge can motivate informed decisions regarding the placement of cameras and markers. We briefly outline the most significant features of a songbird’s wing but more thorough explanations of the functional anatomy can be found in references such as The Manual of Ornithology [PL93] and Form and Function in Birds [KM85].
The wings are homologous to the structure of forelimbs in mammals. The bones of the wing (Figure 2.1) include the humerus, or upper arm, which is linked to the radius and ulna of the forearm. The hand consists of a fused carpometacarpus with a major digit and a smaller alular digit, each with limited flexibility. At the shoulder joint, the humerus meets the furcula (wishbone), which flexes during flight, adding translational degrees of freedom to the shoulder. [Vid05]
The soft portion of the forelimb consists of muscles and connective tissue. Two regions that contribute to extra surface area are the propatagium (patagium in Figure 2.1), which connects the shoulder to the wrist, and the metapatagium (humeral patagium in Figure 2.1), a flap extending from the elbow to the rear of the shoulder [KM85]. The patagia are covered by rows of covert feathers.
The wing bones are connected by ligaments and actuated by contracting muscles, whose elongated tendons provide some passive spring force. The muscles on
3

shoulder elbow

wrist

Figure 2.1: The underside of the left wing of a pigeon, with internal structure revealed. The key bones are the humerus, radius and ulna, and the manus, which includes the carpometacarpus and the digits. [PL93]
the underside of the wing are easily exposed by displacing the covert feathers (Figure 2.2). The major flight muscles include the pectoralis major for wing depression and the supracoracoideus for elevation. Approximately 40 other muscles are involved in controlling various rotational modes of the wing skeleton. [KM85, Vid05]
Feathers are complex structures consisting of keratin. There are many variations in dimension and stiffness, some better suited for thermal insulation, aeorodynamic sensors, and courtship behaviors, but we focus on the primary and secondary flight feathers, also known as remiges. The primaries are attached to the carpometacarpus and are asymmetric—the leading edge vane is narrower than the trailing edge. The secondaries insert along the ulna and are more symmetric. The central shaft or rachis lends strength to each remige without sacrificing flexibility; although relatively stiff for their size and mass, the flight feathers together form an air foil that bends while under aerodynamic pressure. These deflections of
4

Figure 2.2: Ventral wing muscles of a pigeon. [PL93]
the primary feathers have been simulated with a cantilever bending model [PV78]. Although the retrices (tail feathers) were not studied in this project, it is worth noting that they serve as braking and balancing mechanisms in flight. [KM85, PL93]
Birds have a variety of flight styles, but observational and kinematic studies identify common characteristics of flapping wing beats across similar species [Tob07]. Broadly speaking, during the majority of the downstroke the wing joints are extended, the feathers are splayed, and the surface area of the wing is maximal. From a technical standpoint, the upstroke is a challenging portion of the wing beat to video capture—both the elbow and wrist are in flexion and the patagium collapses inward. The surface of the wing is nearly folded at the wrist such that the primaries hide from view the underside of the secondaries. These characteristic movements in birds will help us plan where to place the cameras and accept certain limitations in our data.
5

2.1.2 Direct linear transformation method
Extracting 3D data from a single camera image is an indeterminate problem, but additional viewpoints can rectify ambiguities. The fact that multiple views are necessary for accurately reconstructing a 3D object should come as no surprise. Besides the problem of obtaining depth information, only a portion of a complex 3D surface is seen by the camera—nothing is known about the rear-facing surfaces unless the object rotates into view. Thus, the vantage points must be well-distributed in order to provide sufficient coverage of the subject.
A typical camera can be defined by its position and orientation, as well as a number of internal parameters such as zoom, focal length, and aperture. A simplified camera can be modeled with two geometric components: the camera center is a point offset from an image plane whose size depends on parameters such as the field of view, a principal point determining the orientation of the camera, and an image plane with its own 2D coordinate system (Figure 2.3). Objects in the scene are projected onto the plane; in reality this is not a simple transformation because modern cameras have compound lenses, and some conversion is made into pixel-unit screen space.
Epipolar geometry describes the relationship between two cameras. The epipolar plane is defined by the centers of each camera (C1 and C2) and a point in the scene, X. The following elements shown in Figure 2.4 all lie in this plane:
• A line connects each camera, and the spots where it intersects the two cameras’ image planes are known as epipoles (e and e ). Another interpretation is that the image of the second camera (C2) in the first camera’s (C1) view plane is the epipole e [Let04].
• A ray from the camera center C1 to the object X intersects the image plane at x; the epipolar line l connects the epipole e to point x.
6

camera
C

b x
a image plane

object
X
a = CX b = Cx

Figure 2.3: A simple camera model. Object X is projected onto the camera’s image plane at point x. The vector from the camera to the object is defined by a, and the vector from the camera to the object’s projection is labeled b. Diagram inspired by Y-H. Kwon [Kwo98].
• Where exactly X is located along the C1 camera ray is unknown, but the projection of X in C2 must lie along the epipolar line l . The correspondence between these two views is expressed by
Fx = l
and the epipolar constraint is defined as
x TFx = 0
with F being the rank-2 fundamental matrix that associates a point x in the first view with its epipolar line l in the second view. [HZ03, Let04]
In real-world applications this reconstruction is not a perfect process. Although two calibrated views should theoretically enable us to determine the exact 3D position of a point, there are inherent limitations in the projected images. Motion capture involves identification of markers in video footage; these markers have a
7

X

x e

x’ e’

C1 l’ l C2

Figure 2.4: The epipolar geometry of two cameras. The projection of X onto the image plane of the second camera (C2) is constrained by the epipolar line l . Diagram inspired by Hartley and Zisserman [HZ03].
finite area and when sampled at limited resolution, the very act of determining their image space coordinates introduces error. Because of this error in measuring x and x on the image plane, the resulting epipolar lines do not coincide with x and x because they fail the epipolar constraint. It follows that if back-projected rays from each camera through x and x do not intersect, we can only estimate the 3D point’s position X. [HZ03]
The direct linear transformation (DLT) method is the backbone for solving the relationship between the two camera views and ultimately reconstructing a 3D model of the bird wing. DLT relates the 3D coordinates of an object to the 2D image space of each camera’s projection plane. The algorithm was first published in 1971 as a photogrammetric calibration process by Abdel-Aziz and Karara [AAK71] and has been widely adopted in biomechanics studies because of its straightforward implementation and flexibility when dealing with multiple cameras [Hed08].
A vector a from the camera center reaches the object of interest, and on the way strikes a position on the image plane at the end of vector b (Figure 2.3). Since
8

both a and b lie on the same ray, they differ by a scalar multiple c (Equation 2.1). However, a must be transformed from object space into image space via the 3 × 3 matrix M.

b = cMa

(2.1)

 x − x0 

a=


y − y0

 

z − z0

 u − u0 

b=


v − v0

 

−d

 r11 r12 r13 

M

=

 

r21

r22

r23

 

r31 r32 r33

Recombining the known elements of a, b, and M yields 11 DLT parameters, L1 to L11. For brevity, a number of steps have been omitted. We solve for image space coordinates u and v by rearranging Equation 2.1 and substituting in the L parameters, producing the basic DLT relationships below.

u = L1x + L2y + L3z + L4 L9x + L10y + L11z + 1
v = L5x + L6y + L7z + L8 L9x + L10y + L11z + 1

(2.2)

Calibration: The calibration process involves solving a system of equations for the vector containing the 11 DLT parameters. In practical terms this means photographing a set of n nonplanar control points of known distances from each other and measuring their projected image space coordinates. In Equation 2.3 the rows of the left-hand coefficient matrix can be grouped in pairs, each duo containing the object coordinates (x, y, z ) of a single control point. The corresponding rows of the right-hand side are proportional to the image plane positions of each marker, u and v. The denominators of several matrix elements have been substituted with R, which is defined below Equation 2.3. [Kwo98]

9


L1

 x1 y1 z1 1
R1 R1 R1 R1

0

0

0

0

−u1x1 R1

−u1y1 R1

−u1z1 R1

  

L2

  

 u1 
R1



0



  

...

0 ...

0 ...

0 ...

x1
R...1

y1
R...1

z1
R...1

1
R...1

−v1x1
R...1

−v1y1
R...1

−v1z1
R...1

     

L3 ...

   
=


v1
R...1

     

  

 xn yn zn 1

 

Rn

Rn

Rn

Rn

0

0

0

0

−unxn Rn

−unyn Rn

−unzn Rn

  

L9

  

0 0 0 0 Lxn yn zn 1 −vnxn −vnyn −vnzn

Rn Rn Rn Rn

Rn

Rn

Rn


 10  

L11



 un 

 

Rn

 

vn

Rn

(2.3)

R = L9x + L10y + L11z + 1

An iterative algorithm is required to solve this system because the elements of the coefficient matrix depend on three of the unknown DLT coefficients grouped in R [Kwo98]. Variations of this algorithm take into account nonlinear lens distortion, but the basic 11-parameter calibration method ignores optical flaws [Hed08].

Reconstruction: After calibration, it is now possible to find the 3D position of a point given its projected image coordinates. The reconstruction equation entails another manipulation of Equation 2.2 such that we are now solving for the object coordinates [x y z ]T given a set of calibration coefficients and 2D digitized locations from m cameras.











 

←











↑ ...
2m×3
ma...trix
↓

 
↑



 

x



  

...

  

 

→

 
 
  



y z





=

 











2m×1
vec...tor

     

 
↓

(2.4)

The left-hand side matrix and the resultant vector have been abstracted, and like the calibration equations, they contain elements that depend on x, y, and z,

10

so the system must be solved iteratively. As in Equation 2.3, the rows of the lefthand matrix are pair-related to the (u, v ) coordinates of the digitized object as seen from camera m. Hatze’s modified technique for solving the DLT equations enforces orthogonality of the transformation from object to image space and improves the accuracy of the reconstruction by an order of magnitude [Hat88].
In summary, given the corresponding image coordinates from two or more cameras, the DLT method linearly triangulates the 3D point location [HZ03]. A full treatment of the DLT method for camera calibration and reconstruction can be found on the Kwon 3D website [Kwo98].
2.2 Motion capture of vertebrate flight
Scientists have developed a range of motion capture techniques to address their research questions regarding avian flight, including evolution of the apparatus, biomechanics, the effect of certain feathers on fluid flow, and energy use. Some of the data collection methods are adapted directly from computer graphics or neighboring aerodynamics fields.
Cinematography is a popular method for studying bird flight. Variants include stop motion or high speed film, cineradiographs (X-ray video), or mounting cameras directly on the bird. Arraying several cameras around the subject make 3D motion studies possible through reconstruction, either in realtime or after postprocessing. Reflective markers are commonly attached to the subject to aid computer algorithms with identifying anatomical landmarks in multiple camera views. The CTX is a realtime X-ray system currently under development at Brown University which overlays 3D models of bones generated from computed tomography (CT) onto cinefluoroscope footage of an animal performing a behavior [Bra07]. Once complete, the technology could potentially give scientists literal insight on
11

the complex skeletal movement of flying birds and other vertebrates (Figure 2.5).
Figure 2.5: A precursor to CTX: The 3D virtual skeleton of a pigeon was manually aligned to the X-ray video frames of its flight in a wind tunnel [GDJ04]. Note the metallic markers (black dots), some implanted close to the bones. New registration methods align bones to X-ray and voxel tissue data automatically [MLC06].
A wind tunnel is a favored experimental setting for several reasons. The bird remains in a steady frame of reference, allowing prolonged recording for gathering the best data and a chance to optimize the lighting conditions and the placement of the cameras. It is also possible to attach sensors and other onboard devices to the bird that require trailing leads in order to measure muscle forces or pressure distributions. Aerodynamic properties of wings can be inferred from digital particle image velocimetry (DPIV), a technique for viewing air flow and vortices generated by the wing beat. Preferred flight styles can be studied by varying the wind speed.
There are several difficulties inherent in a wind tunnel. Not all birds fly well in a wind tunnel, and the species that do, such as the cockatiel or starling, require weeks of training in order to build up stamina and become accustomed to the noise levels. There is evidence that birds do not fly completely naturally in a wind
12

tunnel; the vortices shed from the wings may bounce off the walls of the tunnel and continually influence the wing beat, though a model has been developed to compensate for the effects of the reflecting wakes [Ray94].
Vertebrate morphologists and biomechanics researchers have long been interested in the underlying mechanisms of animal locomotion. However, while terrestrial animals can be easily trained to walk on a treadmill or some force plates, birds pose a different challenge. Measurement devices must be small and light so as not to interfere with the aerodynamics of flight. For example, miniaturization of sensors has now permitted the recording of differential pressures on the wing. A more complete description of flight dynamics is possible using force transducers and strain gauges. Electromyography involves the insertion of electrodes into muscular tissue and can detect when a muscle contracts during a movement. Dial, Goslow, and Jenkins used a combination of high speed cineradiography and electromyography to infer muscle activity in the major flight muscles of a European starling [DJJ91]. These direct measurement devices, although often requiring surgical implantation, provide a more accurate picture of the mechanisms of animal movement rather than estimating aerodynamic forces indirectly from DPIV or simplified airfoil models based on kinematic data [UHB03].
2.2.1 Kinematic and dynamic studies
Hedrick, Tobalske, and Biewener [HTB02] modeled the circulation on the wings of cockatiels and turtle-doves using 3D kinematics and a quasi-steady aerodynamics model. They divided the wing into two areas of interest: a proximal portion defined by the shoulder, wrist, and first primary, and a distal section constructed from the wrist, ninth and fourth primary tips. Trackable markers consisting of circular white tape dotted with black were attached to those locations. The marker positions
13

served as the vertices of the proximal and distal triangles, forming a very rough approximation of the wing surface (Figure 2.6).
Figure 2.6: Layout of the wing markers and the triangular planes used to approximate the proximal (T1) and distal (T2) portions of the wing. Cross-shaped marker #1 was used for tracking whole body movements of the bird. [HTB02]
Four high speed cameras were calibrated using direct linear transforms (DLT) such that the 3D location of a digitized point over time could be determined from corresponding camera views. The cameras were positioned to capture useful angles of the wing beat of the subject as it flew stationary in the wind tunnel. Even with four cameras, some occlusion of the markers was unavoidable due to on-edge angles or feathers covering the dots (Figure 2.7). They interpolated three wing beats of data and estimated the missing marker positions using a quintic spline fit, which facilitated easy calculation of velocity and acceleration curves. Aerodynamic properties such as flow, lift, and circulation were then calculated for the two wing triangles.
In 2004, Hedrick, Usherwood, and Biewener [HUB04] described the variation of aerodynamic forces and upstroke-downstroke energies of cockatiel flight over a range of speeds. They used a similar optical motion capture system consisting of three high-speed, synchronized cameras surrounding a wind tunnel and calibrated with DLT. Three accelerometers were implanted in the dorsal midline of the birds’
14

C1 C2
C3 C4
Figure 2.7: Left: Camera configuration about the motion capture volume. Cameras 1 & 2: dorsal | Camera 3: latero-dorsal | Camera 4: latero-ventral. Right: Snapshot of a turtle-dove flying in the wind tunnel as viewed from each of the four high-speed cameras. The markers are clearly visible on either side of the right wing, though in some views, a few of the markers are occluded. [HTB02]
backs above the center of mass to obtain instantaneous accelerations during flight. A lightweight data cable was attached to the accelerometer block and fed to the recording apparatus. The authors noted that the cable and accelerometers may have contributed to the overall drag and reduction of the cockatiels’ maximum flight speeds, but assumed that these devices had negligible effect compared to the forces generated by the wing motion.
Instead of approximating the wing coarsely with two triangular polygons, the authors modeled the deforming wing at a finer level. They estimated the inertial contributions of the cockatiel wing by weighing 1.3-cm wide slices of frozen cockatiel wings and approximating each of the 18 segments as a point mass at the leading edge of the wing. Using the 3D kinematics data, they distributed virtual wing slices along the digitized limbs to match the video frames of the flapping wing, and then calculated the acceleration of the wing sections. While parallel cuts undoubtedly made it easier to approximate and sum the wing beat acceleration contributions,
15

some of the slices spanned multiple feathers, especially along the distal portion of the wing. This could result in some inaccurate acceleration estimates during mid-to-late upstroke when the wing joints are in flexure and the primary feathers overlap the secondaries.

I (×10–6 kg m–2)

A
6 4 2 0
B
1.5 1.0 0.5
0
C Elbow

Wrist

Section moment Section virtual moment
Section mass Section virtual mass

Mass (g)

23.4 cm
Figure 2.8: Moment of inertia and mass of each wing segment in a typical adult cockatiel wing. [HUB04]
Few projects have attempted to reconstruct the 3D deformation of a bird’s wing while flying, though some researchers used innovative methods to film detailed feather actions during flight maneuvers.
Carruthers, Thomas, and Taylor [CTT07] utilized a combination of standard and high-speed digital video to study the aerodynamic deflections in the coverts and alulas of a captive steppe eagle in England. The covert feathers comprise the leading edge of the wing airfoil on the ventral and dorsal sides, and the alular quills cover the first digit of the manus [PL93]. The subtle movements in these structures were observed during take-off, gliding, flapping, and primarily landing sequences. The alula may help the bird sense air currents or turbulence, while a common theory for the coverts is that together they serve as a lift-generating flap
16

and a stall detector.

The eagle was fitted with a pectoral harness such that two small, lightweight

cameras could be affixed to both shoulder regions of the bird, each aimed at the

left and right alulas and coverts (Figure 2.9). PAL video footage (720 × 576 pixels

at 50 fps interlaced) of the feather deformations was downloaded from the cameras

wirelessly during trial flights. The onboard video data was direct but lacking in

temporal resolution as the deploying action of the coverts occurred in less than 60

ms. Separate high speed cameras (1280 × 1024 pixels at 500 fps) recorded greater

detail in the covert deformations as the eagle glided or flapped to a perch.

lesser upperwing coverts

lesser underwing coverts

alula

Figure 2.9: Left: The steppe eagle soaring, wearing its harness equipped with portable cameras. In a 20 ms interval of the downstroke (A followed by B) the lesser upperwing and underwing coverts visibly deflect. [CTT07]
The footage was used to manually determine the timing of the onset and end of feather deflection events over 37 trials. The authors theorized on several aerodynamic mechanisms for the movements of the alula and coverts, but they did not construct a computer model or simulation to illustrate their claims regarding flow separation and lift on the covert feathers. They concluded that wind tunnel experiments would be necessary to quantify the role of aerodynamics in the passive deformation of these structures.
Dial, Jackson, and Segre [DJS08] used a four-camera motion capture setup in order to obtain accurate 3D kinematic data of partridges performing flap-running sequences up a ramp and flapping minor descents from a perch. While there have
17

been many previous studies on bird flight kinematics using similar motion capture technologies, this paper focuses on flight in an ontological-evolutionary context. They compared these flapping styles to level flight to support their hypothesis that a fundamental wing beat across species and developmental stages is constant with respect to gravity.
Chukar partridges were raised from chicks and trained to run up a ramp covered in sandpaper. The ramps were oriented at 65◦, 70◦, 80◦, and 90◦. The second experiment filmed was a flapping descent from a meter-high perch. Two key data sets included the stroke plane angle (the angle of the plane swept out by the wing during downstroke) and angle of attack (angle of the wing-plane). The following kinematic variables were also calculated: number of wing beats, body angle, wing beat frequency, stroke amplitude, dynamic wing loading, dynamic wing length, and actuator disc loading.

Frames-of-reference

Vertebral

A
y

y

Global
By

x x

g

C
x

Gravitational
yy
x gg

x

Figure 2.10: A. The vertebral coordinate system is a relative set of axes aligned to the spine. B. The global and gravitational systems’ y-axes are aligned with gravity. C. The gravitational frame of reference tracks with the shoulder of the bird unlike the global frame, which remains fixed in the world. [DJS08]

Their experimental setup consisted of mixed brands of high speed cameras (Redlake and Fastec). They calibrated the experimental volume with 24 points, and determined that the positional error in digitized 3D points was less than 5%. The birds were marked with retroreflective 3M tape. Although the point set on the bird was sparse, the authors were not trying to capture the deformation of
18

the entire wing in detail. They estimated the dynamic surface area of the wing by treating the markers as vertices of a triangle mesh. A couple of markers on the rump and shoulder defined a frame of reference for the body. The vertebral, global, and gravitational frames of reference are illustrated in Figure 2.10.

A Vertebral WAIR
Flight

Gravitational

Flight

WAIR

g

B 60

90º WAIR

Angle of attack (degrees)

F aero

0 Flight

Downstroke

Upstroke

−60 0 Time (percentage of wing beat) 100

F aero

Figure 2.11: A. While the body orientation changes during flight or flap-running (WAIR), the wing stroke plane angles remain similar for both activities when viewed from a gravitational frame of reference. B. Matching angles of attack for flight and flap-running relative to a global frame. [DJS08]

When viewed from the global frame, the estimated direction of the aerodynamic force was always within a 19◦ angular slice whether the behavior was level flight, flap-running, or descent. In Figure 2.11, the wingtip traces and angles of attack are extremely similar between level flight and flap-running modes when viewed from the gravitational and global frames respectively. This phenomenon was observed in 20 other species.
The authors drew a clear distinction between their interpretation of the wing stroke and other kinematics studies: previous efforts explained bird maneuvers in terms of their changing wing stroke with respect to the more intuitive vertebral frame of reference. Using a global/gravitational frame of reference instead revealed that chukars and other similar species have a nearly fixed wing-stroke, and they modify the power of their wing beat to execute shifts in body rotations. Finally,

19

[DJS08] suggested that the orientation of the shoulder joint remained fixed in 3D space with respect to the gravitational frame as birds evolved.
Riskin et al. [RWID+08] studied the dimensional complexity of fruit bat (Cynopterus brachyotis) flight over a range of speeds in a wind tunnel. Two additional goals of the study were to determine an optimal marker layout for the bat wing and a minimal set of variables that best described the complexity of the wing beat. Unlike birds, bat wings consist of a leathery membrane with shoulder, elbow, wrist, and five fingers all deforming the wing surface. Since these joints contribute more than 20 degrees of freedom for each wing, reflective markers were distributed along the arm bones and throughout the wing membrane to ensure high spatial resolution. Using proper orthogonal decomposition (POD) on the kinematics data, the authors discovered that just 16 orthogonal modes could explain 95% of the wing motion (see Figure 2.12 for a simplified example) and that flight speed had little effect on dimensional complexity.

AB

C

Figure 2.12: A. Digitized trajectories of five bat wing markers (red, green, blue, cyan, purple). B. The first two principal orthogonal components of the decomposed data. C. When composed, the two modes capture 57.3% of the original motion. [RWID+08]

20

2.3 Related works
2.3.1 Motion capture of deformable surfaces
Related motion capture methods have been used for a variety of purposes in different disciplines. In visual effects, a director may desire an actor’s realistic performance for a fantasy character that is otherwise impossible to portray with costume, animatronics, or puppetry. In biomechanics and sports science, researchers employ motion capture to study human gait, stresses on the skeleton, and post-surgical simulations. Vertebrate morphologists have borrowed much of the same technology to track an animal’s movements exhibited in mating behavior, locomotion, or to compare evolutionary relationships among the musculoskeletal systems in related species.
Recent advances in computer graphics algorithms have enabled highly detailed motion capture of subtle movements, such as facial expression. Sifakis, Neverov, and Fedkiw [SNF05] built a virtual, muscle-actuated human face that was driven by sparse motion capture data. Their base assumption was that since most facial muscles are thin and lie directly under the skin, one can estimate which muscles are activated based on movement of the skin surface. The muscle system was defined by groups of flowing B-splines indicating direction of contraction. The muscle model is not force-generating over time; instead the authors assume a quasistatic end pose and map activations to that result. Dynamics can be layered on top of the solution to improve realism during jerky movements.
Unlike simpler models where the various body tissues and fascia and forces are ignored, the authors modeled the deformation of the flesh with a finite element method and also solved for the sliding orientation of the jaw bone. The sparse marker set cannot resolve wrinkles or tiny bumps; rather, the underlying mus-
21

Figure 2.13: Left: Muscle activations were estimated based on changes in the marker data. Right: The underlying isotropic muscles contract, deforming a mesh representing connective tissue and skin. [SNF05]
cle and flesh system produces realistic, small-scale smile creases, lip pursing, and eyebrow furrowing (Figure 2.13). With any motion capture project, there will be stray markers due to noise. An advantage of their work was that the optimization process excluded faulty marker data—only physically attainable expressions were displayed. On the other hand, the mesh was not based on material properties of skin, and while the deformations were visually plausible, the muscle activations driving those movements may not have corresponded to physiologically meaningful values. [SNF05]
Conventional full body motion capture is intended for gathering rigid body rotational data and estimating joint locations. Park and Hodgins [PH06] developed a method for enhanced reconstruction of bulging, stretching, jiggling, and other surface phenomena of human performance in addition to overall limb kinematics. A volunteer was covered with 350 reflective markers each spaced 4-5 cm from each other. The subject was limited to a 2 m × 2 m × 2.5 m tall region surrounded by 12 near-infrared (NIR) cameras (4 megapixels at 120 fps). After the subject performed several dynamic motions containing subtle secondary movement, such as shadow-boxing, breathing, or jumping, the marker data was processed to fill
22

holes due to occlusions. Using the markers as vertex locations, a surface mesh was constructed with the assumption of constant topology over time. Every vertex was aware of its nearby neighbors and had a local frame. Thus, for each missing marker, a statistical estimation based on the neighboring markers accurately repaired the damaged data (Figure 2.14).

AB

C

Figure 2.14: A. A marker surface suffering from occlusions contained gaps in the data. B. The neighbors-based hole-filling algorithm repaired the mesh. C. The high-resolution model was posed and deformed to fit the marker surface mesh. [PH06]
Unlike [SNF05], no underlying musculoskeletal system was simulated to match the data. Instead, Park and Hodgins used the sparse marker surface to drive the motion of an anatomically detailed, denser mesh. They achieved this by separating out the rigid body motion of the limbs and treating the residual as local deformation due to muscle bulging or skin stretching.
These examples from the computer graphics arena did an excellent job of capturing accurate motion data and reconstructing models that could express detailed deformations. However, the test subjects were cooperative, relatively slow-moving humans who were well practiced at delivering performances for the cameras in tightly controlled areas. In contrast, a bird in free flight breaks all of these constraints. The inherent challenges of capturing, modeling, and validating avian flight may explain the relative scarcity of bird simulations compared to the vast
23

numbers of human studies in computer graphics. There are some important papers on bird simulation from the computer graphics community, and we introduce them in the next section.
2.3.2 Simulating winged flight
There have been numerous efforts to digitally simulate bird flight in the film industry, including artistic animation inspired from reference footage or kinematics data. Examples of computer-generated birds in cinema include the cartoonish pelican in Pixar’s Finding Nemo (2003), the stylized crane in DreamWorks’ Kung Fu Panda (2008), the photorealistic giant eagles in The Lord of the Rings: The Return of the King (2003), and the fictional but convincingly rendered phoenix in the Harry Potter franchise. Accurate wing beat motion generated from physically based models are far less common due to a scarcity of accurate pressure measurements, turbulent aerodynamics, and high computational cost.
In computer graphics, Ramakrishnananda and Wong [RW99] designed a twojoint flapping wing model driven by forward kinematics and dynamic equations. The relevant flight surfaces incorporated only the primary and secondary remiges but neglected the tail feathers. The humerus and forearm were converted into a single segment, and they justified the omission of the patagium, arguing that this region was aerodynamically insignificant. All movement of the wings was symmetric; neither yaw nor roll was possible. The authors defined a flight course by specifying target locations, and their simulator solved for the body pitching angle and aerodynamic forces necessary to reach the targets for a given flapping pattern.
Wu and Popović [WP03] have created the most realistic, physically based bird simulation to date. Their algorithm is capable of producing complicated, asymmet-
24

ric acrobatics such as taking-off, diving, wheeling, and landing with appropriate styles for various-sized species, like the sparrow, raven, and eagle. The individual flight and tail feather models are flexible; each have bending and twisting springs for reacting to aerodynamic forces (Figure 2.15B). Other parameters include feather splay, arm joint configuration and forearm twist, and duration of the wing beat (Figure 2.15A). The user determined the flight path and velocities while the algorithm attempted to find an optimal wing and tail parameterization. External forces included gravity and simplified aerodynamics, based on approximate lift and drag coefficients. An objective function penalized wing motion that would damage feathers. Surprisingly realistic behavior was generated; besides demonstrating a believable wing stroke, the bird model fanned its retrices during landing, a braking mechanism observable in nature.

A
shoulder (quaternion joint)

forearm twist elbow bend
wrist bend

B
twist spring

tail twist

tail spread

tail bend spread angle of attack

feather

bend springs

aerodynamic forces

Figure 2.15: A. The skeleton of the bird model with arrayed flight and tail feathers. B. The feather model responds to bending and twisting forces due to asymmetric feather vanes as observed in the remiges of real birds. [WP03]

In 2007, Jeffrey Wang [Wan07] built an articulated computer model of an ivorybilled woodpecker with the eventual goal of pattern matching his model to video footage of an unidentified bird. The motivation for this project was to settle a debate about the supposed extinction of the species. In this project a controversy erupted in the ornithology community when M. D. Luneau’s [FLMDL+05] blurry

25

footage purporting to contain the once-thought extinct ivory-billed woodpecker was released to the public. Others disagreed with the analysis of the video and argued that the subject was probably the more common pileated woodpecker [SBPE06]. A preserved specimen was digitized using CT scanning and a low-polygon mesh was manually constructed in Alias Maya to fit the volumetric data. Primary and secondary wing feathers were modeled with trimmed parametric surfaces, and the remainder of the plumage was mimicked with texture maps and Maya Fur. The bird geometry was rigged with virtual bones which deformed the skin around the joints. Aerodynamic deflection of the feathers was approximated by mapping predefined bending, twisting, and splay pose functions to various states of the wing armature. In a first test, the wing joints were driven with angles reported in kinematic studies of European starlings [DJJ91]. High speed reference footage of pileated woodpeckers descending from a nest inspired a revised animation of the ivory-billed woodpecker wing beat [WBL+08].
Figure 2.16: Left: A frame from the video of a pileated woodpecker performing a flap-gliding descent. Right: The ivory-billed woodpecker model was animated to approximate the wing beat of the subject in the footage. [WBL+08]
Despite these visually convincing simulations, both animated and physically based, the researchers who created them lacked the support of accurate kinematic
26

and aerodynamic measurements, instead relying on diverse reference footage or hand-picked parameters to supplement their models. Our aim for the past two years has been to expand on existing methods to capture the three-dimensional flight of songbirds so that the forthcoming data could be useful to ornithologists, animators, and others for validating or inspiring new results.
27

CHAPTER 3 FLIGHT TUNNEL AND MOTION CAPTURE METHODS
The experiments took place at David Winkler’s flight tunnel, which is located at the Cornell Experimental Ponds. The facility consists of an outdoor wooden structure approximately 13.4 m long × 1.36 m tall × 1.3 m wide. The entrance is walled but contains a cubbyhole with a trapdoor for releasing the bird subject into the tunnel, after which it flies the length of the passageway, attracted towards the sunlit exit.
Fastec Imaging TroubleShooter (top view)
camera-aligned ood light
EXIT

bird ENTRANCE
bird dropbox

Redlake Imaging MotionMeter (rear view)

Fastec Imaging TroubleShooter (side/right view)
trigger

Figure 3.1: The flight tunnel with the traditional camera setup. Although the notion of filming the bird from from orthographic views initially seemed like a good idea, in practice it failed to produce usable marker imagery due to self-occlusions in the top view and on-edge angles of the wings in the rear view.

Several options exist for positioning cameras. Below the dropbox is a camera port for filming the posterior of the bird. There are two other pyramidal structures
28

jutting from the side and top of the tunnel 2.3 m from the release point, each containing a camera mount as shown in Figure 3.1. With these three ports the tunnel provides the possibility of filming the bird simultaneously from top, side, and rear views. However, the deficiencies in the data resulting from the orthogonal three-camera setup are apparent in Figure 3.2. Most of the wing markers are visible in the top camera view during mid-stroke, but nearly all are invisible in the side view. During the peak downstroke, the markers are maximally visible in the side view but almost completely occluded in the top view by the body. One positive aspect of this preliminary data is that the silhouettes of the body are well-defined and could be used to estimate the bird’s center of mass. The rear view (not shown) suffered from low contrast and maximum intensity clipping due to direct viewing of the tunnel exit. Our initial camera configuration made use of all three mounts, but we eventually opted to move the cameras to new locations in order to improve the visibility of the markers on the bird wing for the majority of the flap cycle.
3.1 Equipment
3.1.1 Cameras
Two Fastec Imaging TroubleShooter cameras equipped with zoom lenses were mounted on mini tripods in the flight tunnel. The TroubleShooters are sensitive to near-infrared light and visible wavelengths. The trials were filmed at 500 fps and 2-ms shutter speeds. Other relevant specifications are listed in Table 3.1. Although the cameras are capable of 1000 fps, choosing the higher frame rate would cut the resolution in half due to storage considerations, and the lighting situation was far too dim to resolve markers at that setting.
One TroubleShooter, the “slave”, was placed on the ground in the rear portion
29

SIDE
frame 30

TOP
frame 30

SIDE
frame 41

TOP
frame 41

Figure 3.2: The side and top views for an earlier red-winged blackbird trial. The upper snapshots occur during the mid-downstroke while the lower images are at the peak downstroke. For certain portions of the wing beat, most of the wing markers are visible in one camera while completely occluded in the other.

of the tunnel, facing the exit, and aimed slightly upwards. The “master” camera sat on the bottom slope of the side pyramid, essentially oriented at the wall with no ports and partially towards the exit (Figure 3.3). The exact locations and orientations of the cameras with respect to the tunnel were unknown, but knowledge of these measurements was not essential for reconstructing the flight of the bird since the calibration step encapsulated the necessary camera information in a set of coefficients. With the help of a calibration object, the focus and zoom of each camera was manually adjusted to maximize the common viewing volume.
The cameras were connected in serial with synchronizing cables. The right-side

30

Table 3.1: High speed camera settings and output specifications. The Redlake camera was used only in the earlier trials.

Frame Rate (fps) Shutter Multiplier Resolution (pixels) Output Format

Fastec Imaging TroubleShooter
500 1X (2 ms) 1280 × 1024 AVI (raw)

Redlake Imaging MotionMeter 500 1X (2 ms) 658 × 496 DV (compressed)

1.3 m 2.7 m

rear port

ENTRANCE

1.85 m

2.3 m

side port

13.4 m
top port
9.25 m
rear camera
(SLAVE)

EXIT

TOP VIEW

side camera
(MASTER)

Figure 3.3: A top view schematic of the flight tunnel. In our most recent experiments, the rear imaging MotionMeter was omitted because of technical issues, and we used a more “freestyle” setup with angled rear and side cameras in order to capture more of the wing beat.

31

SIDE (master)

REAR (slave)

frame #

67
72
77
82
Figure 3.4: Selected snapshots at 10-ms intervals from the latest camera setup. Although the side camera footage was out of focus, there were far more markers visible in both cameras for the majority of the flight compared with the original setup (Figure 3.2). camera was set to master mode, whose status was controlled by a trigger button that sent a stop signal to the master camera (side) and then the slave camera (rear). The cameras recorded continuously, overwriting internal RAM when exceeding their storage until a stop signal deactivated the session.
With this configuration, the resulting images were far superior to those from
32

our initial setup, and we were able to extract useful data (Figure 3.4). There were still marker visibility issues during the upstroke and mid-downstroke; however, there was enough data adjacent to the problematic frames to interpolate the gaps in the marker trajectories.
3.1.2 Calibration cube
Filming an object of known dimensions is necessary for reconstructing 3D data from flat camera imagery. Our calibration “cube” (Figure 3.5) is actually an irregular Tshaped structure used to calculate the direct linear transform (DLT) coefficients of a multi-camera system. The cube was constructed from a plastic molecular modeling kit, and the intersection pieces were marked with retroreflective dots, each regularly spaced at 10.8 cm. The modular nature of these lightweight components allowed portability in the field, as well as the option to easily extend or dismantle the cube. In earlier experiments we used a much smaller version with only 20 markers, but later decided to expand the cube to the shape shown in Figure 3.5 in order to delineate more completely the region viewed from each camera. The asymmetric cube simplified the task of identifying individual markers in all camera views.
3.1.3 Markers
We attempted to track wing deformations by placing retroreflective markers on key landmarks such as the arm joints and along the primary and secondary flight feathers. Passive reflective markers are frequently used in motion capture systems because they do not hamper the subject’s movement nor require electrical leads and additional hardware. We created 3-mm flat markers by hole-punching 3M retroreflective tape and affixing the sticky side to the bird feather. Although a
33

10.8 cm
Figure 3.5: The calibration cube consists of 10.8-cm struts connected by intersection pieces. Attached to each joint is a retroreflective marker, appearing as a white dot. hemispherical marker would permit a larger viewing angle, it would likely interfere with the folding and layering of the feathers and might be preened off too easily. Since both sides of the wing are visible in each camera, we placed markers on the ventral and dorsal surfaces of the wing.
34

3.1.4 Lights
Three MR-16 halogen reflector flood lamps (50 watt, 38◦) were individually attached to the three cameras as close to their lenses as possible. Due to the retroreflective nature of the markers, the lights were also aligned in the direction of the lens. In addition, four to six 500-watt halogen work lights were distributed around the viewing volume to provide extra illumination for the markers and to separate the bird from the darkly-toned tunnel background.
Natural light leaking into the tunnel was of concern. Except for the exit, it was necessary to block all other openings in the tunnel, such as the side pyramids and release box. The goal was to minimize anything that appeared to be “sky” so that the bird would not choose a side tunnel for its attempted escape route or backtrack to the entrance. The brightness of the lamps could also conceivably affect the bird’s choice of route, but their directional nature masked their presence for most of the bird’s flight.
Originally we tested three 6-watt infrared lamps1 (a compound light consisting of an array of IR emitting LEDs) by Wildlife Engineering, and mounted them next to the camera lenses. IR lights are commonly used for surreptitiously filming animal behavior in low light conditions, such as bats roosting in caves or nocturnal predators raiding bird nests for eggs and chicks [BMIJ98]. Unfortunately, the IR lights did not provide sufficient illumination of the markers, feathers, or background at 500 fps to be of practical use. Therefore, we employed the MR-16 halogen lights.
3.2 Methods
The methods described in this section are the result of several months of trial and error, and are by no means foolproof or definitive. We first attempted to film birds
1http://www.batmanagement.com/Ordering/irlight/irlight.html
35

as they made trips between a copse and a series of bird feeders. After capturing and marking several specimens, we released them, hoping that they would resume feeding activity. Instead, either the birds promptly fled the vicinity or they spent the next half hour in the trees, meticulously preening off the markers we had so painstakingly attached. Although anecdotal, this story illustrates the difficulties and unpredictability of working with birds in the wild. Despite the attraction of an all-natural setting, clearly there are advantages for choosing an environment with some controls, the steps of which we outline below.
3.2.1 Obtaining a bird
For each day of the measurement experiments, new wild birds were caught in mist nets. Thus, none of the birds used in our tests had prior experience in the flight tunnel, although each specimen was used for multiple runs. Birds were handled according to IACUC protocol 2001-0051 under the supervision of Cornell University ornithologist Dr. Kimberly Bostwick.
Mist nets are a typical research tool used by ornithologists for harmlessly catching medium-to-small birds. We unfurled several mist nets around a common feeding area where multiple species gathered. The nets are so fine that most birds do not notice them (or they are more preoccupied with eating the free food), until they bounce off the mesh, fall a short distance, and become trapped in a pocketlike region at the base of the net. It is important to remove the birds quickly lest they become completely entangled or suffer prolonged exposure.
At first we flew in the tunnel almost any species that fell into the net, including the common grackle, goldfinch, robin, red-winged blackbird, song sparrow, and downy woodpecker. Generally, the larger visitors, especially the blackbird and grackle, had slower wing beats; this property was advantageous since motion blur
36

was a significant concern. In addition, their sizeable appendages were easier to resolve in our limited camera resolutions. In contrast, the filmed footage of the finches and sparrows was blurrier due to their faster wing beats. Furthermore, the ratio of wing size to marker diameter was smaller, which increased the error in their reconstructed trajectories. The downy woodpecker, also a small bird, refused to cooperate and instead swooped above the cameras and alighted upside down on the ceiling of the tunnel. This was particularly disappointing since the plumage of the downy woodpecker is speckled with a pattern of natural white markers, a characteristic which would have enhanced our reconstruction methods. As we observed the habits and evaluated the species’ performance in the tunnel and the resulting data quality, we narrowed down our selections to the red-winged blackbird and the common grackle.
Unfortunately, it was not simply a case of releasing unwanted birds and keeping the good ones. The common grackles, for example, became conscious of the net and carefully avoided a wide swath of suspicious areas. Following a month of trials, we never caught a grackle again. The red-winged blackbird, however, was not so cautious, and we continued to catch them regularly. While their numerous population and medium-small size made them reasonable candidates, they became our primary bird as we had no other viable choice.
3.2.2 Marking the bird
The wild bird was gently grasped in the hand while retroreflective markers were glued to the feathers with a mild adhesive. This process was time-limited by the temperament of the bird; an excitable one could become more stressed the longer it was held, to the point of refusing to fly in the tunnel. An important step that we omitted was careful measurement of bone, body, and selected feathers for
37

determining actual scale and error in the reconstructed marker trajectories. Instead we relied solely on calibrated digitization as our data source of bird features. This issue and its impact on the results will be discussed in Section 6.2.6.
The distribution of the markers on the wing was sparse. Figure 3.6 illustrates the template used: a series of four markers on the arm joints (6-9) and five on the primary and secondary feathers (1-5). For capturing downstroke portions of the wing beat, we placed markers on the dorsal surface of the wing, directly opposite the markers on the ventral surface. The shoulder joint was problematic due to its larger girth and high density of feathers where the wing merges with the body. Markers attached to the rump and thigh were intended for tracking changes in the orientation of the body.
Not all of the markers placed on the bird were ultimately useful. Although additional markers can contribute to a more detailed reconstruction, they also become harder to individually track and are prone to occlusion by some moving body part or off-angle viewing. In cases where the marker signals were too weak, there was scarcely any definition of the wing configuration other than the silhouette.
3.2.3 Flight and retrieval
At first we used the tunnel’s dropbox as the release point for the bird, but we found that the subject did not always fly consistently within the cameras’ fields of view. In the final trials, we improved our framing of the bird by unloosing it from the hand at a level with the cameras. The bird flew to the end of the tunnel where it was recaptured and evaluated for signs of stress. If the bird appeared upset or fatigued, the session was ended. Most birds were flown twice and none more than four times. At the end of an experimental trial, the bird was freed in an area close to its original place of capture.
38

5

3

8 9
76

4

1 2

2 1

6 9

5’ 3’
4’
1’ 2’

8’ 7’ 6’

1 tip of 9th primary 4 inner 4th primary 7 wrist joint

2 inner 9th primary 5 tip of 2nd secondary 8 elbow joint

3 tip of 4th primary 6 manus 9 shoulder joint

Figure 3.6: A marked red-winged blackbird (immature adult). Top: Retroreflective markers were affixed to arm joints and on selected tips and mid-lengths of primary and secondary feathers. Lower left: Two more markers were placed on the ankle and the hip. Lower right: Markers on the dorsal side of the wing are mirror duplicates of the ventral side (denoted by prime). The unlabeled markers were not digitized because of visibility problems.

We performed approximately twenty flight trials over three months with twelve yielding video data. The other trials were aborted due to a camera issue or the bird choosing a flight path outside of a camera’s field of view. Several video streams initially appeared to have a full set of markers visible but on closer inspection

39

we found many gaps in the data or incorrect camera calibration. These problems motivated us to upgrade the lighting system and introduce a new camera configuration. Even after these tunnel modifications, only our last capture session (July 31, 2008) with a red-winged blackbird had enough continuity in the marker trajectories to be suitable for reconstruction, and we analyze that data set in Chapter 5. The rest of the video footage has been archived—the slow motion movies are still useful for descriptive purposes.
40

CHAPTER 4 DATA PROCESSING
The task of reconstructing 3D data from two sets of 2D image sequences was extremely difficult with our existing experimental configuration. In order to obtain some useful, geometric flight data, we had to complete a number of pre-processing steps before extracting the 3D marker trajectories from the raw footage. These steps are as follows:
1. Calibrate the cameras to calculate the direct linear transformation (DLT) coefficients necessary to obtain 3D data.
2. Manually track a set of wing and body markers for a selected flight sequence. 3. Interpolate the gaps in the 2D space-time trajectories for each wing and body
marker using cubic splines. 4. Compute the 3D marker coordinates using the DLT coefficients and the
stitched trajectory data.
Each of the steps mentioned above has errors associated with the data acquisition hardware or the process used for computation. We were, however, able to approximate at least one complete data set which appears to be visually correct. Our pre-processing procedures are described in more detail below.
4.1 Camera calibration
A still frame from the footage was chosen such that the calibration object remained maximally visible in both viewports, avoiding self-occlusions. The landmarks on the calibration object were of known dimension so that the computer could solve for a transformation between the two camera views. Tyson Hedrick’s Digitizing Tools [Hed08], a group of MATLAB scripts, contain a DLT calibration module
41

which takes as input 3D coordinates of the calibration points and snapshots of the calibration object. The user specifies which points in one camera view correspond to the other. An example of a digitized cube (rear camera) is shown in Figure 4.1. After both views of the calibration object were processed, the DLT coefficients were calculated using a linear solver.
Figure 4.1: Left: The calibration program interface. A close-up view of a marker (white) is visible in the magnified region (square inset). Right: Our intrepid colleague Kim held the calibration “cube” while under the glare of several thousand watts of light. The intersections of the cube struts (red numbered dots) were marked in the same order as the file specifying their measured 3D coordinates.
Several factors can affect the quality of the camera calibration. Since the images of each marker covered multiple pixels, accurate digitization in the DLT calibration program was assisted by a zoom feature and automatic centroid detection for a group of similar-intensity pixels. Adding more cameras would introduce a helpful redundancy that can give a better estimate of the error. A well-aligned calibration structure and precise measurements of its features would also reduce the residual. Camera lenses can introduce nonlinear optical distortions that should
42

be corrected in a separate processing step. Lastly, the calibration object should fill the volume that is being filmed; the more markers distributed throughout the extent of the space, the better we can trust the data originating from the region that was calibrated. [Hed08]
An estimate of the DLT calibration residual identified problem points on the calibration cube and how much uncertainty in the marker trajectories during the flight trials would be due to calibration inaccuracies. The residuals in Figure 4.2 suggested that points 19, 32, and 43 were good candidates for exclusion. The most likely causes for these errant points were variations in the flexible cube structure and inaccurate digital marker labeling. Our calibration cube was composed of lightweight plastic tubing, advantageous for portability and ease of manipulation in cramped quarters, but susceptible to being bent out of shape—no doubt this contributed to some error in the calibration. We did not test for lens distortion, but it is likely that the first two causes were more detrimental to our accuracy.
4.2 Marker digitization
Once the two cameras were calibrated and the DLT coefficients were established, the markers on the bird were ready to be digitized with Hedrick’s marker tracking tool (DLT Data Viewer), which shares a similar interface with the calibration module. Unfortunately, the visibility and intensity of the light-reflecting markers were not consistent enough from frame-to-frame to take advantage of the autotracking feature, even after adjusting the brightness, contrast, and gamma of the videos. In Figure 4.3 the white markers on the feathers contrast nicely with the wing, but a number of visual flaws are apparent. First, the lack of dynamic range in the cameras introduces clipping for the intensity extremes: portions of the body and wings were indistinguishable from the black background in the rear camera.
43

DLT Residual Diagnostic
1.94

DLT residual with 1 calibration point removed

1.92

1.9

1.88

1.86
candidate points for exclusion 1.84

1.82 0

5 10 15 20 25 30 35 40 45
Calibration point

Figure 4.2: Each calibration point was removed one at a time, and the newly calculated DLT residual, improved or worsened, was updated. DLT residuals represent the mean-square error (measured in pixels) about a 3D point generated by the DLT algorithm [Hed08].

Second, the side camera was slightly unfocused. Third, we were unable to increase the shutter multiplier because of insufficient lighting, so motion blur was present in all data sets. Rather than appear elliptical, the markers traced out a “smeary” curve. We were consistent in choosing the center of the motion blur path as the location for the digitized point. If we desired higher temporal resolution, we could mark a point at the beginning, middle, and end of the motion trail, effectively increasing the frame rate by three.
Hedrick [Hed08] warns that the line of least residual, though helpful for estimating the location of a marker in another camera view, can be misleading; the DLT algorithm is sensitive to the calibration conditions noted earlier. Indeed, there were some frames where the markers on the bird were visible in both views and yet the least residual guide did not intersect either marker. A better algorithm would

44

use distance information from neighboring landmarks to limit the search space of possible point locations, but this method would require a densely marked wing, and we were not prepared to invest the inordinate time required to manually track so many points at this proof-of-concept stage.

SIDE CAMERA

REAR CAMERA

epipolar line

suggested marker
Figure 4.3: The tracking tools allow the user to manually specify the locations of a marker in each camera view. If the cameras are properly calibrated, clicking on a marker in one view will cause an epipolar line of least residual to appear in the other view(s), hopefully intersecting the same marker.

4.3 Interpolation of image-space marker trajectories
The red-winged blackbird flaps its wings at a rate of about 12.5–13 beats per second. No matter where the cameras are positioned, the bird’s opaque wings will occlude a subset of the markers due to the extreme deformation of the feathers or on-edge viewing angles. At fast film speeds (500 fps), even a brief occlusion can cause a substantial gap in the data, on the order of ten missing frames. In general, the more proximal the marker on the wing, the greater the likelihood for self-occlusion. Our best results suffered many such gaps, making interpolation difficult. In addition, the absence of marker information poses a serious problem for continuing the analysis in the 3D realm, as illustrated in Figure 4.4.
45

Incomplete Marker Data & Discontinuous 3D Trajectory
Camera 1 (2D) Camera 2 (2D)

Interpolated Marker Data & Continuous 3D Trajectory

DLT-computed trajectory (3D)

 time →

Figure 4.4: Only data common to both cameras can be used to compute the 3D trajectory (black). Left: Non-interpolated 2D marker trajectories (gray) yield a disjointed 3D trajectory. Interpolating the missing sections in the 3D path would be error-prone and clearly ignores the marker locations recorded in each camera alone. Right: Instead, the holes in the 2D trajectories must be joined beforehand so that the DLT calculation produces a continous 3D curve.

Incomplete marker trajectories were processed with MATLAB’s Spline Toolbox. The raw values associated with a marker include x and y screen coordinates for each camera, parameterized by time. Gaps in the data were interpolated with 4th order smoothing splines (Figure 4.5). The underlying assumptions justifying our use of interpolation include periodicity and at least 2nd order smoothness in marker position. The first premise is quite obvious: flapping motion is cyclic in nature, and thus many of the marker trajectories resembled sinusoidal functions. However, not all 2D marker trajectories had such clear trends; a few bore features of sigmoid curves or nondescript functions with sudden slope irregularities. A possible explanation is that some of the markers were affixed to flexible feathers that underwent sudden deformation, thereby thwarting assumptions of smoothness. We acknowledge that filling in the missing data, no matter how plausible the algorithm, is an approximation. This method allows us to proceed with the 3D visualization and model-fitting tasks, but severely limits what we can claim to be accurate motion. In the results chapter we will examine the validity of some of the joint markers based on assumptions of rigid bone attachment.

46

Interpolation of the Manus Marker Trajectory
(rear camera)
1100

1000

data spline

missing data

900

y-coordinate of marker trajectory (pixels)

800

700

600

500

400

300 20 40 60 80 100 120 140 160
frame # (500 fps)

Figure 4.5: MATLAB’s Spline Tool easily stitched together broken image-space data with plausible smoothing splines. This graph shows an interpolation result for the y-coordinate of the manus tip marker.

4.4 Reconstruction

The joined data were re-imported into Hedrick’s DLT Data Viewer for two additional steps. First, we briefly scanned the modified trajectories to verify that (a) the interpolated marker locations were at visually reasonable spots on the wing in each camera view, and (b) the DLT residuals for the interpolants were comparable to the errors in the non-interpolated regions. Second, the software automatically calculated all 3D marker trajectories based on the 2D paths and the DLT coefficients, and this reconstructed data was exported to separate text files for use in a 3D graphics package. The positional error of the reconstructed points was 2-5 mm, but only for those markers that were originally visible in both cameras and digitized within the calibration region. Ty Hedrick’s DLT package provides several tools for estimating errors, such as a calibration residual diagnostic and the option

47

to generate 95% confidence intervals for the reconstructed points [Hed08]. Due to time constraints, it was necessary to work with non-ideal data in order to
demonstrate other aspects of the overall motion capture method. If improved data collection procedures and additional cameras yielded data that were more complete from the outset, the need for interpolating large gaps in the 2D trajectories would have been greatly reduced.
48

CHAPTER 5 RESULTS
Our original goal was to obtain sufficient data to accurately describe the progression of geometric wing deformations of a medium-sized passerine in free flight. Despite the inaccuracies of our recording processes and the use of interpolated data which can be rationalized but is not scientifically accurate, we still were able to construct a useful motion description.
To verify the utility of this approach we created a triangulated wing shape consisting of seven triangles and eight vertices to approximate the wing surface. We virtually “glued” this polygonal mesh to the marker data, which directed the movement of the vertices in time and space, generating a flapping behavior. We then enhanced the display of the wing motion by fitting a rough anatomical boneand-feather model to the marker trajectories. We were able to visually ascertain this motion by superimposing our simulated wing mesh on the original videos. Although we recognized that there are many inaccuracies within this pilot study, the resulting motion of the 3D wing model was remarkably similar to what was obtained photographically.
5.1 Marker trajectories
The reconstructed trajectories were imported into Autodesk Maya 2008 for visualization. The discrete data points for each marker were connected with a color-coded spline and then rendered from several viewpoints as displayed in Figure 5.1. The complete data is shown, but a smaller timeframe in which 3D data existed for all markers was used for the wing model fit. This subset of data is bracketed by starting and ending spheres in Figure 5.1C, and its duration is slightly longer than one wing beat. Although the data is incomplete beyond the demarcated range, its
49

AB
1st primary (tip) 1st primary (inner) 4th primary (tip) 4th primary (inner) manus (tip) wrist 2nd secondary (tip) time time elbow shoulder
C
time
Figure 5.1: Marker trajectories. Partial data exists for the inner 2nd secondary marker, retrices (tail feathers), and beak but not enough for a consistent reconstruction and were thus omitted from these graphs. A. Posterio-lateral view (perspective). B. Posterior view (rear orthographic). C. Lateral view (side orthographic). The brackets denote the portion of the data used for fitting the wing over a time period of 92 ms.
periodic nature is evident, especially when viewed from the rear (Figure 5.1B). In order to view the flap cycle from a stationary frame of reference, the trans-
lation of the markers in the overall direction of travel was removed. The resulting data is similar to the kind that would be recorded in a wind tunnel1 where the bird is mostly stationary along the anterior-posterior axis of the body but is free to move vertically and laterally. In Figure 5.2, the trajectories have been frozen with respect to the forward motion of the shoulder. The swept curves of selected
1It is important to note that ornithologists believe that birds “flying” in a wind tunnel do not exhibit the same wing motions and deformation as in free flight.
50

9th primary tip
A 9th primary, halfway
manus

B

51

t = 0 ms

10 ms

20 ms

30 ms

40 ms

50 ms

60 ms

70 ms

80 ms

Figure 5.2: Trajectories of three markers on the leading edge of the wing with respect to the shoulder joint. Marker positions (black dots) for the tip of the 9th primary (wingtip), a marker halfway down the 9th primary, and the manus were
sampled every 10 ms and composited over the looped motion paths. A. Latero-ventral view. B. Posterio-ventral view.

9th primary tip
A 9th primary, halfway
manus

B

52

t = 0 ms

10 ms

20 ms

30 ms

40 ms

50 ms

60 ms

70 ms

80 ms

Figure 5.3: Snapshots at 10-ms intervals of the triangulated wing mesh layered over the same marker trajectories depicted in Figure 5.2. The vertices of the wing mesh consist of three leading edge markers (large black dots) and other wing markers (small black dots). The fully-labeled mesh is pictured in Figure 5.5. A. Latero-ventral view. B. Posterio-ventral view.

markers have been trimmed down to a single flap cycle, which start and end at the onset of the downstroke. A slight discontinuity in the motion paths of Figure 5.2A is due to variation in the bird’s wing beat. In Figure 5.2B, the wingtip flex is obvious judging from its kidney-shaped trajectory (outer path).
The looped trajectories of Figure 5.3 also feature a triangular mesh surface whose vertex positions are driven by the motion of the markers. The elbow marker data, however, was too inconsistent to include as a vertex in the wing mesh. Although an imperfect representation, the animated mesh gives a rough impression of a flapping wing, and allows some problems with the data to be identified. For example, the upstroke appears reasonable in Figure 5.3A (side view) but in 5.3B (rear view), the surface collapses unrealistically at t = 60 ms due to uncertainty in the shoulder and other proximal markers in those frames.
5.2 Wing visualization
Although the reconstructed marker trajectories are the most faithful representation of the available 3D data, they are not well-suited for visualizing the wing shape as it transforms from one phase of the wing beat to another. The triangulated mesh based on the markers is an improvement over motion paths, providing some concept of the deforming surface, but it does not inform the viewer of the underlying bone structure and lacks the true wing surface area defined by the feathers.
With these deficiencies in mind, we devised a virtual wing skeleton and feather set to fit to the marker data. The components of this simple model are as follows:
• Arbitrarily-spaced shoulder, elbow, and wrist joints controlled with an inverse kinematics system
• A curve with aim constraints to control feather spread, inspired by the interremigial ligament
53

bone length (cm)

Bone Lengths Based on Joint Marker Distances
5 4.5 ulna
4 3.5
3 humerus 2.5
2
manus
1.5 1
0.5 0 10 20 30 40 50 60 70 80 90 100 time (ms)
Figure 5.4: Bone length was estimated based on the distance between adjacent joint markers and then plotted over time. The size of the manus was relatively consistent (SD = 0.0916 cm) while the humerus and ulna varied greatly in length due to missing shoulder and elbow marker data and error in the interpolation.
• The use of influence objects to artistically correct pops and discontinuities in the data-driven animation
• Minimal bone and feather geometry for economic visualization
There are a number of assumptions for this model. First, feathers plucked from a separate spread-wing specimen were measured from the quill to the feather tip, and the lengths of each virtual feather were set to these distances (Appendix A). The feathers were modeled as stiff, cylindrical rods, despite the fact that real feathers flex with external air pressure during flight. Even at rest, their rachi are slightly curved.
Second, we assigned arbitrary model bone lengths based roughly on the statis-
54

tics of the joint marker locations. For example, the humerus can be defined by the vector from the shoulder to the elbow joint. In Figure 5.4, pseudo bone lengths are plotted for the duration of the recorded flight. Clearly the shoulder and elbow marker data are suspect since the humerus and ulna lengths fluctuate wildly over time. Besides occlusion problems, one explanation is that markers were not centered over the joints. The shoulder is an inaccessible joint embedded under muscle and skin that no single marker can pinpoint. The manus, however, remains fairly constant in size with a standard deviation of less than a millimeter, which suggests that the wrist and manus marker data are trustworthy. The model bone lengths were set to the following values: humerus = 2.9 cm | ulna = 4.5 cm | manus = 1.5 cm (Figure 5.5A). However, these values are not proportional to actual specimen bone dimensions detailed in Appendix A. On average, the real ulna and manus

ACTUAL

A manus 2.7 cm

humerus 2.6 cm

ulna 2.7 cm

1.5 cm

4.5 cm

2.9 cm

B
3 5

1 4
8 9

1 tip of 9th primary 2 inner 9th primary 3 tip of 4th primary 4 inner 4th primary 5 tip of 2nd secondary 6 manus 7 wrist joint 8 elbow joint 9 shoulder joint
arm bones

MODEL

SCALED

Figure 5.5: Elements of the simplified wing model. A. From top to bottom: Measured red-winged blackbird bone lengths (see Appendix A) followed by our model’s chosen dimensions. Even when our model is scaled down according to the actual humerus length, the ulna is 47% too long. The shortness of the model manus is due to marker #6 positioned too proximally on the bone instead of at the tip. B. Schematic of the wing mesh constructed from 8 markers. The elbow marker was not included in the mesh due to its inaccurate, erratic motion.

55

rachis

interremigial ligament

Figure 5.6: The interremigial ligament is an elastic tissue that connects the primaries and secondaries and is passively involved in the spreading and folding of the wing [KM85]. Figure 5.7B demonstrates similar functionality in our model with a parametric curve.

with digits were nearly the same length (2.7 cm) while the humerus was slightly shorter (2.6 cm). Attempting to fit anatomically accurate virtual bones to the data would be problematic and especially prone to joint lock with an inverse kinematics control system. Therefore, we chose discretionary values.
We only have feather tip trajectories of the 9th and 4th primaries and the 2nd secondary (Figure 5.5B), so angular interpolation is needed to distribute the remaining feathers between the marked ones. The method we chose has at least some anatomical basis: we mimicked the function of the interremigial ligament (Figure 5.6) with a NURBS curve, whose control points include the aforementioned feather tips and another control vertex (CV) near the shoulder. The last CV near the shoulder is a substitute for what would ideally be a location on a tertial or proximal secondary. Aim constraints were distributed evenly along a segment of the three-dimensional ligament curve using a MEL2 script, and the orientation of the feather rods were linked to these goal objects (Figure 5.7). As the elbow and wrist
2Maya Embedded Language

56

A B CD
pseudo interremigial ligament
Figure 5.7: Fitting the bones and the feathers to the marker data. A. The reconstructed markers. B. A simplified arm skeleton fit to the shoulder, elbow, wrist, and manus markers. To simulate the interremigial ligament, three feather tip markers and the shoulder served as control points for the NURBS curve (black). Aim constraints (white dots) for the feather rods were distributed along the curve. C. The 9 primary and 9 secondary remiges were oriented towards the aim constraints. D. Compared to the feathered model’s surface area, the wing mesh lacks coverage especially near the shoulder and the distal primaries.
joints extend, the curve length and the distance between adjacent aim constraints increases, causing the feather rods to spread. When the joints flex, neighboring feather rods become more aligned, constricting the space between them.
The inverse kinematics solution which controlled the wing bones was not immediately satisfactory. Using only the elbow marker to determine the twist of the shoulder-wrist pole resulted in impossible configurations for the armature. In particular, the accuracy of the upstroke data suffered because most of the actual wing joints were in flexion, causing the shoulder and elbow markers to be minimally visible. The wild motion of the virtual elbow joint was finally supressed by sharing constraints between a marker on the 2nd secondary feather tip and a manually animated corrector object. In the computer generated skeleton, the shoulder joint’s position was also slightly “tweaked” to prevent a noticeable pop in the elbow joint during the upstroke. The use of rigging controls or other animation techniques that layer new motion over the data prevents us from making strong scientific claims,
57

but it does allow us to visualize flapping motion that is more believable than the raw version.
Selected frames of the fitted wing model are shown from side, rear, and top views in Figure 5.8. The bird’s direction of travel is from left to right as depicted in the lateral and dorsal views, but for the posterior snapshots, the flight vector is perpendicular to the page. For a subjective verification of the wing model, we composited the animated geometry over the video sequences, and images from the rendered movies are shown in Figure 5.9. As expected, the wing model conforms reasonably well to the markers and the bird silhouette; after all, these were originally extracted from the camera views used for digitization. Although these static diagrams with two-dimensional images show the correlation between our estimated procedures and the actual photographs, the comparison is much more vivid when the results are seen in true motion. The reader is referred to the footnoted hyperlink3 in order to view the actual video results.
5.3 Kinematics
The bird flew an average speed of 4.4 meters per second at a downward slope of 13◦ across the camera fields of view. We assumed that the tunnel walls and closed entrance minimized the effect of outdoor breeze during the experiments. The single wing beat captured was 77 ms long, a frequency of 13 cycles per second. The downstroke, as defined by downward movement of the wrist, lasted 40 ms (52% of the wing beat) while the upstroke was slightly shorter at 37 ms (48%). The joint angle data in Figure 5.10 were extracted from the fitted bones, and schematics to the right of the graphs demonstrate the axes used to calculate the angles.
Throughout the downstroke the humerus descends at a nearly constant rate of
3http://www.graphics.cornell.edu/
58

59

A
time
B
C
Figure 5.8: Snapshots at 6-ms intervals of the wing model fit to the marker data over one flap cycle. A. Lateral view. B. Posterior view. C. Dorsal view.

60

A
(L)
B
Figure 5.9: The flapping right-wing model overlaying the original footage, sampled every 14 ms. A. Side (master) camera. Note: In the 3rd slide, most of the left wing is visible beneath the bird’s abdomen (L). In the 4th slide, two markers on the left wing (arrows) nearly coincide with right wing markers. B. Rear (slave) camera.

JOINT ANGLE (degrees)

Joint Kinematics

100 50 0 -50
250 200 150
150 100
50
200 150 100
50 0 0

10 20 30

HUMERUS

ELEVATION

HUMERUS

DEPRESSION PROTRACTION

ELBOW

RETRACTION EXTENSION

WRIST

FLEXION EXTENSION

FLEXION
40 50 60 70 80 90 100
TIME (ms)

HUMERUS tail
head
tail head
WRIST
ELBOW

Figure 5.10: The first two graphs depict the orientation of the humerus, followed by the joint angles of the elbow and wrist. Legend: The red arcs from the humerus to the world horizontal plane represent its elevation-depression and protractionretraction modes, respectively. The schematic of the arm demonstrates the flexionextension angles of the elbow and wrist.

2.6◦ per millisecond. In the first 10 ms the humerus protracts rapidly through 81◦, and the elbow joint undergoes a slight extension before flexing at a constant rate for the remainder of the downstroke. The wrist remains mostly extended for the first 40 ms. In the late downstroke the arm adducts and retracts. Although the humerus may have ceased its downward motion and the wrist will soon elevate, the feather tips of the primaries continue the downstroke in an adducting sweep, maintaining propulsion as the upstroke begins.
During the first 20 ms of the upstroke the humerus elevates, raising the stillfolding elbow and wrist joints. Nearly 10 ms into the upstroke the elbow begins

61

to extend, followed 10 ms later by a sharp unfolding of the wrist. A questionable movement occurs during the upstroke where the elbow unnaturally “pops” as the humerus retracts and protracts suddenly. This, and other perturbances in the humerus elevation graphs are likely due to gross inaccuracies in the elbow marker’s position, resulting in an imperfect inverse kinematics solution. By the end of the upstroke, the joint angles of the humerus, elbow, and wrist are restored to their original state for the next cycle.
Based solely on observations from the videos, it appears that the bird’s body undulates as it flies, much like a swimmer performing the butterfly stroke. As the wings depress, the posterior of the body arches, followed by the retrices which extend upwards. At peak upstroke the posterior descends, returning the body to a more linear posture. The head remains relatively stationary. For the portion of the flight where it was visible, the beak was digitized (data not shown) and its trajectory was nearly linear. Like many other passerines, the red-winged blackbird favors a flight style where the feet are tucked against the body. In the side camera, the legs extend slightly at the end of the upstroke. This secondary motion is likely due to the temporary downward momentum of the posterior end of the body.
62

CHAPTER 6 DISCUSSION
6.1 Data analysis
There are no prior kinematics studies on red-winged blackbirds nor works on free flight to which we can compare our results. The best comparisons we can make are to Dial et al. [DJJ91], who provide joint angle data gathered from European starlings. In Figure 6.1, we have reproduced our results from Section 5.3 sideby-side with those of Dial et al. In this part of the discussion we abbreviate the common bird names, red-winged blackbird (RWBB) and European starling (ES). Each set of graphs depict the time-dependent elevation-depression and protractionretraction modes of the humerus (spherical coordinates) and extension-flexion of the elbow and wrist joints. Radioulnar twist was not measured.
From a broad perspective, the two graph sets have similar trends for each of the arm joints. The RWBB’s wing cycle was 77 ms (90 ms total shown) while the ES segment lasted 71.75 ms. Each curve in the ES data has peaks, valleys, or plateaus for which analogous features appear in our RWBB data. In Section 5.3 we interpret some of these key events (e.g. transition of downstroke to upstroke, wing furling, etc.), as well as identify spurious motion resulting from imperfect data and a suboptimal model fit. Dial et al. give a very detailed account of the wing kinematics in their text consistent with our observations. However, one striking case is the extension-flexion curves of the wrist, which are particularly close in shape for both species (labeled F on the bottom graphs). Our RWBB exhibited a delayed but fast flexion during the upstroke, and this very sharp dip in value is mirrored in the ES data.
The blackbird and starling share some flight style characteristics, but the tim-
63

Comparison of Joint Kinematics: Red-winged Blackbird vs. European Starling

A
100 50 0 -50
250 200 150
150 100
50

HUMERUS HUMERUS
ELBOW

B

JOINT ANGLE (degrees)

64

200 WRIST
150
100
50 F
00 10 20 30 40 50 60 70 80 90 ms
Red-winged blackbird (Agelaius phoeniceus)

European Starling (Sturnus vulgaris)

Graph reproduced from Dial, Goslow, and Jenkins (1991).

Figure 6.1: The wing beat of A. phoeniceus is qualitatively similar to that of S. vulgaris. A. Kinematics of the virtual bones from our wing model. B. Perceived bone angles interpreted from cineradiography of a starling [DJJ91].

ings of their kinematic events differ. We inferred that the RWBB humerus, for example, reaches maximal depression 36 ms before it maximally retracts whereas the ES humerus achieves this coordinated motion in less than 12 ms. On average the RWBB had 25◦ more joint range than the ES; this greater freedom of movement could be attributed to physiological differences between the RWBB and ES and/or errors in our fitting process. An outlier case was the protraction-retraction of the RWBB humerus, which spanned angles of nearly 96◦ while the ES was restricted to 27◦. This large discrepancy is likely due to the uncertainty in the elbow and shoulder markers’ positions—substantial sections of these maker trajectories are a “best guess” interpolation, and have an unknown effect on all of the calculated joint angles. The initial 10-ms protraction of the RWBB humerus, absent in the ES data, may be the end of the previous upstroke cycle, though more data is needed to properly classify the onset of the RWBB downstroke.
There were some procedural differences between the two studies. Our RWBB data was obtained in free flight while the ES flew in a wind tunnel. In Section 2.2 we noted that ornithologists have observed that birds modify their flight in wind tunnels—obviously, we cannot demonstrate this notion when measuring separate species with different tools. In our data, the angles between the bones were calculated exactly with a script, but the underlying marker trajectories had frequent missing sections. Patching these relatively large gaps required assumptions of continuity in the marker paths in order to continue with reconstruction and visualization. Dial et al. determined joint angles by manipulating mounted bird skeletons and matching them by eye to their X-ray video frames. They achieved consistency with this method by repeating the manual kinematics determination on four birds.
Despite the superficial similarities and glaring contrasts between the RWBB
65

and ES results, it is difficult to draw conclusions. That the humeral RWBB and ES curves resemble each other is not surpising: even the most rudimentary flapping limb will elevate and depress with some sinusoidal-like function. One might expect that some differences would appear in the other RWBB joint angles, since two of our four joint markers had accuracy issues. Yet this is not entirely the case, as seen in the RWBB extension-flexion of the wrist, which compares agreeably with the ES. A cautious explanation is that the kinematic trends were not so sensitive to the positions of the shoulder and elbow, and that our manually corrected animation and the use of nearby markers on the secondary feathers were reasonable substitutes for those two problem markers.
Ty Hedrick’s DLT package provides several tools for estimating error, such as a calibration residual diagnostic and the option to generate 95% confidence intervals for the reconstructed points [Hed08], but we defer an in-depth error analysis until we procure better equipment and improve our data acquisition methods to a point of reproducibility. We can, however, summarize the propagation of error from start to finish. The least adulterated data are the original high speed videos of the RWBB. Every process thereafter introduces some form of error. Previously, we have discussed the residuals in the DLT calibration process, the human error in digitizating markers, the approximations when interpolating gaps in the data, the reconstruction error, and the anatomical inaccuracies of the fitted 3D wing model. We have shown that our model (1) presents believable 3D wing beat motion from multiple viewpoints (2) reasonably matches 2D camera projections when superimposed over the video footage, and (3) produces joint kinematics that roughly resemble those of another bird. Therefore, we assert that our motion capture method is sound but currently quantitatively inaccurate due to insufficient equipment and poor data quality.
66

6.2 Proposed improvements
Many flaws in the data have been identified, and most are attributable to our equipment choices and the limitations of our experimental setup. Attempting to fix issues with the motion capture technology was frustrating because changing one variable often affected others in unexpected, adverse ways. Although there will always be unknown factors, we are confident that many of these problems can be remedied. We are encouraged by the fact that numerous researchers such as Hedrick [HUB04] and Riskin [RWID+08] have demonstrated high quality motion capture footage and accurate reconstructions, even if their end goals were different than ours. After sifting through the best of their techniques and analyzing our own methods’ shortcomings, we now outline steps that could be taken to improve the experimental setup and procedure.
6.2.1 Setting
The choice of experimental environment is extremely important because it places constraints on other aspects of the setup such as cameras, lights, and markers. An ineffectual change of setting could render the other elements useless. Foremost, the flight tunnel provided a space to guide the path of a wild bird keen on escape. But a dark, narrow space is not conducive for strategically placing cameras and lights, especially lenses which may have poor focal ability, limited field of view, and a distorted image at close range—walls and ceilings are simply in the way, which leads to drastic measures such as cutting holes in the structure for optimal equipment positioning.
Unlike a wind tunnel, whose enclosure, turbine, and safety mechanisms are necessary for minimizing turbulence and maintaining air pressure, there is no inherent reason for a flight tunnel to be so confined by rigid material. A flexible solution
67

could entail a cordoned hallway or a narrow room that employs one solid wall of the building and another made of a curtain-like canvas with slits or windows for observation. For example, Hedrick et al. [HUB07] created a netted corridor for studying low speed maneuvering flight of rose-breasted cockatoos. Netting or nonopaque cloth allows for better lighting, adaptable placement of equipment, and simple disassembly.
While the current structure somewhat hampered our data-gathering ability, there is no need for over-designing alternatives. Much of the hassle associated with predicting the weather, capturing birds, setting up, testing, and breaking down equipment could be avoided in a dedicated indoor center with trained, captive birds.
6.2.2 Bird selection
Experimentation with vertebrates is heavily regulated. Obtaining a more flexible IACUC protocol opens the possibility of keeping captive birds and training them to fly in an indoors facility, not necessarily the flight tunnel. Birds kept for the long term would become more accustomed to handling and flying in a familiar space, and may yield more consistent results. Treats or other motivation could persuade the subject to fly from open perch-to-perch in full view of the cameras, as in Carruthers et al. [CTT07], thus obviating the need for a restrictive tunnel. On the other hand, there are basic resource costs for maintaining captive birds: a space, proper food, veterinary care, exercise, etc.
The red-winged blackbird was a fine choice for the purposes of this work. However, there are advantages to choosing other species. The rock dove (pigeon), European starling, and cockatiel are commonly used in flight studies; data collected from any of these model birds could be compared to previous results
68

Figure 6.2: The ventral surface of a hairy woodpecker’s right wing. Most of the primary and secondary feathers have 4-5 regularly spaced white dots and the dorsal side of the wing shares a similar pattern. Retroreflective markers were attached to the tips of every 5th feather and on the wing joints in this early test.
[DJJ91, HTB02, UHMB05]. The downy and hairy woodpeckers have a striking pattern of white dots distributed over dark flight feathers which, under ample lighting, could be used to track the deforming shape of the wing (Figure 6.2). Hummingbirds are another intriguing option. Their constant need for nourishment combined with a bold personality allows for easy filming as they hover near a nectar feeder.
6.2.3 Recording technology
Our Fastec TroubleShooter cameras had several advantages including portability, battery power, easy synchronization, and a relatively high resolution and frame rate compared to more expensive models. However, the dependability of our particular units was low; they reliably corrupted our data at least 20% of the time, either with vertical line artifacts, unrepairable frame translations, or power issues. We will never know the extent of potentially good data that was lost, and in
69

later experiments we do not wish to waste valuable time troubleshooting faulty equipment.
Future camera purchases should include improvements in at least two areas: color sensitivity and resolution. First, the current cameras are grayscale, and while sufficient for distinguishing a bright white marker from dark feathers, we may want to avoid using markers altogether and instead track identifiable patterns on the plumage. If the patterns are different hues but similar in intensity value, then color sensors will be required to determine how the feathers have moved from frame to frame. Second, higher resolution videography is necessary for filming free flight because the camera lens must be zoomed out to encompass a full wing beat while still retaining enough detail to distinguish the markers. The Fastec TroubleShooters recorded at 1280 × 1024 pixels, but the largest dimension of the bird was never more than 650 pixels and rarely greater than 400 pixels. While sufficient for a small number of markers, such low resolution may not be adequate for tracking small features in the plumage or following a densely marked wing. As a side note, filming in a wind tunnel clearly has advantages for maximizing available sensor resolution since the bird remains stationary relative to the camera.
One of the most obvious enhancements to our setup would be to increase the number of cameras. Hedrick et al. [HTB02] used 4 cameras, Park and Hodgins [PH06] used 12, and motion capture studios in the entertainment industry may use more than 20. The additional views would safeguard against missing data due to occlusions and decrease the error during reconstruction. More cameras carry added cost, but there will soon be affordable alternatives. Ever-improving technology has filtered down from the $10,000-plus digital cameras into cheaper but reasonably capable “prosumer” models. For example, the Casio EX-F11 can record 300-fps movies at 512 × 384 pixels (up to 1200 fps is possible, albeit at
1http://www.exilim.com/intl/ex_f1/spec.html
70

the tiny resolution of 336 × 96 pixels) but can be purchased for less than $900 as of February 2009. Arguably, a properly positioned array of ten of these cameras could produce much better data than two or three of our current cameras.
6.2.4 Markers
The retroreflective markers adhere immediately to bird feathers without being permanently attached, they are thin and light enough so as not to interfere with flight, and when observed from the correct angle and with sufficient lighting, they return a very bright signal. Our final test run had more markers visible than ever before—a total of nine were digitized, and two others were visible for a short portion of the flight. A drawback of the flat markers is that they become severely dimmed when viewed from sharp angles or covered by the slightest bit of feather. Three markers failed to appear at all in the trial, either because they fell afoul of the problems mentioned or they actually fell off the feathers. Using additional cameras and lights can somewhat mitigate the visibility problems, but it may be worth examining alternative markers.
Both the entertainment industry and biomechanics researchers employ markers consisting of spherical or hemispherical styrofoam balls covered in a reflective material [Sco03]. These are obviously viewable from a greater range of angles than a flat marker, and consistent visibility is one prerequisite for automated tracking, a time-saving process we were unable to exploit with our data. Smaller styrofoam beads coated in reflective paint are a conceivable option. For robust attachment the glue must be strong, fast-drying, and non-toxic. However, a binding adhesive may result in feather damage upon marker removal. Despite these challenging specifications, expending some effort to find a better marker may ultimately improve the fidelity of later results.
71

6.2.5 Marker layout
In order to properly fit a modeled skeleton to motion capture data, one must be able to estimate the joint locations. Soft biological tissues surround the functional joints in vertebrates; any external markers will reflect the thickness of the bone and cartilage protecting the joint, while bulging muscles and skin deformations can shift the markers to positions unrelated to the center of rotation. The shoulder joint, for instance, is particularly inaccessible, and no single marker can accurately provide its location. Our strategy of using joint markers to drive an inverse kinematics solution was at best a faulty approximation. A better approach is to put at least three markers on each rigid arm segment, sufficiently distant from the parent joint, and solve for the center of rotation by examining the trajectories of the bone markers.
An important simplifying assumption is that any point on the limb remains a constant distance from the joint; that is, all sets of markers for that limb move on the surfaces of concentric spheres, or in the case of a constrained joint such as the elbow, on a circle. Chang and Pollard developed a constrained least-squares algorithm for estimating the center of rotation of a spherical joint from marker trajectories [CP06] and an optimized plane-fitting method for finding the dominant axis of rotation in hinge joints [CP07]. Their work improved upon existing data fitting techniques, which suffer accuracy problems when the range of marker movement is diminished. However, the center of rotation estimation algorithms may not yield good results if the marker sits atop the joint, as in our experiments. In this case the error in measuring the marker movement would be indistinguishable from actual rotation about the joint, which is why markers should be placed further down the limbs, away from the joints.
The variation in the distribution of markers on the wing make comparisons
72

c a b

d

o p q

e fg
l ij
m k
n

h

Figure 6.3: Ventral snapshot of a fruit bat revealing the layout of the markers, 17 of which were digitized (a-q) after its flight. [RWID+08]
difficult. What is the best way to develop a standardized pattern, one that is efficient, repeatable, and transferrable to similar species, thereby enabling comparison of multiple data sets? In Section 2.2.1 we surveyed recent work by Riskin et al. [RWID+08] involving a detailed study of bat flight. They tracked a plethora of wing markers (Figure 6.3) and analyzed the trajectories with proper orthogonal decomposition (POD). This numerical method identified which markers captured the essential motion of the wing beat and determined functional groups of joints with correlated movement. Riskin et al. examined other bat kinematics studies and found that scientists’ marker templates were only able to yield intermediate levels of dimensional complexity. Our current marker template, one based on intuition about the deforming feather structures and key joint locations, undoubtedly failed to capture many important aspects of the wing movement. We could possibly remedy this defect by following their exhaustive procedure. One caveat of this method is its dependency on anatomical structure and placement of the markers; results from birds with substantially different wing morphology are unlikely comparable.

73

6.2.6 Measurements
We neglected to collect anatomical measurements of the red-winged blackbird subject, and in retrospect, this step would have provided useful information for validating the reconstructed wing model. For example, none of the estimated bone lengths were verifiable, although other factors such as occluded markers are partially to blame. The primary explanation for this procedural flaw was simply a lack of experience with motion capture post-processing at the time. There was also a sense of urgency while marking the bird—after it nearly escaped twice, we decided to proceed with the flight experiments. Finally, we were preoccupied with a far more serious problem that was hindering our data collection: an acceptable camera and lighting configuration was not yet known, and our trial-and-error strategy for rectifying the setup was to fly as many birds as possible and then adjust the equipment until we obtained good image quality. By the time we realized that we had a viable data set, the bird had already been released, and only later did it become apparent that measurements of the subject would have been useful.
As discussed in Section 6.2.2, using calm, captive birds that are familiar with being handled would facilitate the measuring process. Calipers or a simple metric ruler would be sufficient for determining wing parameters such as feather and bone lengths. For studies involving the tail, the retrices should be quantified as well. The mass of the subject is important for momentum and acceleration calculations—the lift forces produced by an aerodynamic, full-body model of the bird would need to surpass its weight.
6.2.7 Reconstruction
Due to gaps in the marker data, we resorted to interpolating the 2D trajectories in order to create complete 3D traces. This sacrifice in reconstruction accuracy
74

enabled us to visualize the essence of our data in the form of a geometric wing model. A thorough critique of the direct linear transformation (DLT) method is beyond the scope of this work. Rather, we will suggest a few complementary reconstruction and post-processing techniques.
Feature-based machine vision algorithms could be employed to track natural feather patterns and glean wing shapes, as it was impractical to affix large numbers of markers along the length of each feather. Boosting the illumination would allow a higher frame rate and better isolation of the bird from the surroundings. One class of algorithms for separating foreground and background elements includes segmentation, for which a graph-based method was developed by Felzenszwalb and Huttenlocher [FH04]. An a priori bird model with realistic anatomical constraints and parametric feathers could be used to estimate the pose of the test subject by matching it to the silhouettes or extracted shapes with global optimization.
Although we have previously stated that densely marked wings are difficult to prepare, time-consuming to digitize, and offer negligible benefits in terms of revealing dimensional complexity, there is a counterargument for abundantly marking the subject. Park and Hodgins’ [PH06] methods, summarized in Section 2.3.1, are capable of capturing minute shifts in skin deformation due to gross limb movement or local muscle contraction. Furthermore, their hole-filling algorithm can predict the location of occluded markers based on intact neighboring topology. Applying this technique to a sparsely labeled wing is plausible but may result in a larger reconstruction error.
75

6.3 Future work
6.3.1 Model improvements
The final product of this thesis is an animated wing model driven mostly by the reconstructed marker trajectories. Our wing model was by no means physiologically accurate—it was merely a visualization aid for displaying the motion capture data. The approximations were many: the virtual bone lengths were not proportional to the dimensions of the actual bones, inflexible cylindrical rods served as placeholder feathers, and there were no muscle bulges or skin surfaces. These deficiencies can be overlooked for the purposes of this initial study, but once we improve our data collection methods, we ought to design a model that can faithfully reproduce detailed aspects of wing beat motion.
The computer animation industry focuses on the external appearance of creatures; any internal representation of muscles and bones is artistically directed and usually for the sole purpose of rendering realistic skin deformations. If the goal is to heighten the visual realism of a computer-generated flapping bird, then one valid approach is to create a red-winged blackbird model at the level of detail of Jeffrey Wang’s ivory-billed woodpecker rig [Wan07]. Crafting an animator’s model does not preclude integrating results from relevant biological research. For instance, Vazquez [Vaz94] has explained the automated mechanisms that link flexion and extension in the avian elbow and wrist, and applying this information to a model could decrease the likelihood of unrealistic poses.
Realistic simulation of flight depends on physiologically accurate models, biological parameters, and governing equations of motion. For example, an investigation of joint freedom would benefit from an armature composed of bone density models generated from CT scans. With a collision detection algorithm or finite
76

element analysis, joints would no longer be reduced to point or axis simplifications; instead, one could simulate bone surfaces sliding across each other. A distant goal would be the creation of a total musculoskeletal model where virtual contracting musculotendons generate torques about the arm joints, causing the wing to deform. Lift, drag, and gravitational forces also act on the wing; thus, the problem becomes one of finding muscle activation timings that will produce the necessary forces to keep the bird aloft. Determining the rate and amplitude of these parameters is a difficult optimization problem, but previous experimental data can constrain the range of inputs. The timings of muscle activation and force generation for a starling’s wing beat have already been measured with electromyography [DJJ91] and strain gauges [BDJ92]. Aerodynamic forces, either simulated with computational fluid dynamics or measured in a wind tunnel, could be applied to the entire structure.
Future models should take into consideration the underlying anatomical structures responsible for flight. There will always be trade-offs among physical realism, complexity, and computational cost, but new levels of simulation detail can be layered on as the research criteria dictate.
6.3.2 Research questions and applications
Our collective scientific knowledge on how birds fly is still in the making, and many questions remain unanswered. From mapping muscle synergies to charting energy requirements to simulating the aerodynamics of flight for different speeds and styles, there is much to be discovered for thousands of avian species.
The video results and 3D marker trajectories are immediately applicable to my colleague’s work on the aerodynamics of bird flight. David Kaplan resized Jeffrey Wang’s ivory-billed woodpecker model to match the smaller dimensions of the red-
77

winged blackbird. Using the controls provided in the rig, he manually posed the wing bones to align with the corresponding joint markers and bent each of the feathers to match the video silhouettes of the bird in the camera views. With a rapid prototype 3D printer, he created a series of solid wing models for different phases of the stroke. These models will be mounted in a wind tunnel where he will study the steady-state pressure distribution on each wing pose and describe the airflow over the surface. [Kap09, Wan07]
Another avenue of research to explore are the material properties of feathers and how they bend during flight. Our data contains partial information on the flexing of the distal primaries, which is especially pronounced during the middownstroke and upstroke. The 9th and 4th primaries carried two markers each: one on the feather tip and one partway up the shaft. Using the manus markers to approximate the location of the quill, one could fit a low-order spline from the bone through the two feather markers. In future experiments, adding markers along the length of the feather would provide the basis for a much better estimate of its curvature. Dynamic feather bending could be simulated with a spring system or a cantilever obeying the material properties of keratin [BP95]. A difficult task will be coupling the effects of muscle-initiated forces with aerodynamic phenomena, as well as feather-on-feather collisions.
Other interesting questions arise in evolutionary comparative studies, education, and animation. For example, are there fundamental joint kinematics across similar species? While we have examined linear free flight, what wing configurations are used to accomplish diving, turning, or landing maneuvers? A general muscle-driven wing model with adjustable parameters would be useful as an educational tool and for visualizing hypothetical flight in rare or extinct species. From a data-centric view, one could envision a collaborative motion capture library filled
78

with flapping styles that could be blended together to create new bird flight animations. For the promising multitude of questions to investigate, it may be apt to say, “The sky is the limit.”
79

APPENDIX A ANATOMICAL MEASUREMENTS

We took supplemental measurements of preserved red-winged blackbird specimens in order to build a more accurate wing model for fitting the motion capture data and analyzing results. The Cornell Laboratory of Ornithology has a wealth of stuffed birds, spread wings, and skeleton collections available for research or official loan. Two samples we measured are shown in Figure A.1.
Using calipers, we measured the right side wing bones of several blackbird skeletons (Table A.1). We chose labeled specimens that would have been close to the age and dimension of the immature individual we flew in the motion capture experiments. Unfortunately, the sexes of blackbirds at that developmental stage are indistinguishable to the eye, so both male and female samples were measured.

Table A.1: Lengths (mm) of the right humerus, ulna, and manus (carpometacarpus and second digit) measured from seven A. phoeniceus breeding individuals.

Item # Sex Humerus Ulna Manus

CU 40187 F

23.86 24.67 24.44

CU 40188 F

23.27 24.12 24.18

CU 40189 M 28.18 30.03 28.99

CU 40193 M 27.54 29.53 29.53

CU 40195 M 24.52 25.77 25.31

CU 40197 M 28.73 30.80 29.73

CU 48946 F

23.62 24.81 —

Average

25.67 27.10 27.03

80

humerus

ulna

manus 1 cm

Figure A.1: Arm skeleton (radius and some digits omitted) and feathered wing (dorsal view) of an adult male red-winged blackbird. In the enlarged frame, the bones have been composited over the wing to illustrate their approximate scale and location with respect to the feathers. For comparison, see Figure 2.1, a diagram of the internal structure of the pigeon wing. Photographed and measured at the Cornell Laboratory of Ornithology collection.
81

The remiges were measured with a ruler from the tip of the calamus to the distal end of the rachis, ignoring any feather curvature. The last column of Table A.2 represents the feather lengths multiplied by a factor of 0.913 for use with the virtual wing, which was modeled at a slightly different scale.

Table A.2: Lengths of primaries (P) and secondaries (S) obtained from a plucked A. phoeniceus wing. The values were normalized to approximate the overall feather dimensions suggested by the reconstructed marker data.

Feather 9P 8P 7P 6P 5P 4P 3P 2P 1P 1S 2S 3S 4S 5S 6S 7S 8S 9S

length (cm) 8.20 8.50 8.65 8.85 8.50 7.90 7.70 7.55 7.35 7.20 7.00 6.95 6.75 6.70 6.50 6.40 5.45 4.20

scaled (cm) 7.483 7.757 7.894 8.076 7.757 7.209 7.027 6.890 6.707 6.571 6.388 6.342 6.160 6.114 5.932 5.840 4.974 3.833

82

BIBLIOGRAPHY
[AAK71] Y. I. Abdel-Aziz and H. M. Karara. Direct linear transformation from comparator coordinates into object space coordinates in closerange photogrammetry. Proceedings of the Symposium on CloseRange Photogrammetry, pages 1–18, 1971.
[BDJ92] Andrew A. Biewener, Kenneth P. Dial, and G. E. Goslow Jr. Pectoralis muscle force and power output during flight in the starling. The Journal of Experimental Biology, 164(1):1–18, 1992.
[BMIJ98] K. P. Brown, H. Moller, J. Innes, and P. Jansen. Identifying predators at nests of small birds in a New Zealand forest. The International Journal of Avian Science, 140(2):274–279, 1998.
[BP95] Richard H. C. Bonser and Peter P. Purslow. The Young’s modulus of feather keratin. The Journal of Experimental Biology, 198(4):1029– 1033, 1995.
[Bra07] Elizabeth L. Brainerd. CTX Technology Development Project, August 2007. Available from: http://brown.edu/Project/CTX/.
[CP06] Lillian Y. Chang and Nancy S. Pollard. Constrained least-squares optimization for robust estimation of center of rotation. Journal of Biomechanics, 40(6):1392–1400, 2006.
[CP07] Lillian Y. Chang and Nancy S. Pollard. Robust estimation of dominant axis of rotation. Journal of Biomechanics, 40(12):2707–2715, 2007.
[CTT07] Anna C. Carruthers, Adrian L. R. Thomas, and Graham K. Taylor. Automatic aeroelastic devices in the wings of a steppe eagle Aquila nipalensis. The Journal of Experimental Biology, 210(23):4136–4149, 2007.
[DJJ91] Kenneth P. Dial, G. E. Goslow Jr., and Farish A. Jenkins Jr. The functional anatomy of the shoulder in the European starling (Sturnus vulgaris). Journal of Morphology, 207(3):327–344, 1991.
[DJS08] Kenneth P. Dial, Brandon E. Jackson, and Paolo Segre. A fundamental avian wing-stroke provides a new perspective on the evolution of flight. Nature, 451(7181):985–989, 2008.
[FH04] Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient graphbased image segmentation. International Journal of Computer Vision, 59(2):167–181, 2004.
83

[FLMDL+05] John W. Fitzpatrick, Martjan Lammertink, Jr. M. David Luneau, Tim W. Gallagher, Bobby R. Harrison, Gene M. Sparling, Kenneth V. Rosenberg, Ronald W. Rohrbaugh, Elliott C. H. Swarthout, Peter H. Wrege, Sara Barker Swarthout, Marc S. Dantzker, Russell A. Charif, Timothy R. Barksdale, Jr. J. V. Remsen, Scott D. Simon, and Douglas Zollner. Ivory-billed woodpecker (Campephilus principalis) persists in continental North America. Science Express, 308(5727):1460–1462, 2005.
[GDJ04] S. M. Gatesy, K. P. Dial, and F. A. Jenkins Jr. An inside look at skeletal motion in flying birds. Journal of Vertebrate Paleontology, 24:63A, 2004.
[Hat88] H. Hatze. High-precision three-dimensional photogrammetric calibration and object space reconstruction using a modified DLTapproach. Journal of Biomechanics, 21(7):533–538, 1988.
[Hed08] Tyson L. Hedrick. Software techniques for two- and threedimensional kinematic measurements of biological and biomimetic systems. Bioinspiration & Biomimetics, 3:1–6, 2008. Available from: http://www.unc.edu/~thedrick/software1.html.
[HTB02] Tyson L. Hedrick, Bret W. Tobalske, and Andrew A. Biewener. Estimates of circulation and gait change based on a three-dimensional kinematic analysis of flight in cockatiels (Nymphicus hollandicus) and ringed turtle-doves (Streptopelia risoria). The Journal of Experimental Biology, 205:1389–1409, 2002.
[HUB04] Tyson L. Hedrick, James R. Usherwood, and Andrew A. Biewener. Wing inertia and whole-body acceleration: an analysis of instantaneous aerodynamic force production in cockatiels (Nymphicus hollandicus) flying across a range of speeds. The Journal of Experimental Biology, 207:1689–1702, 2004.
[HUB07] T. L. Hedrick, J. R. Usherwood, and A. A. Biewener. Low speed maneuvering flight of the rose-breasted cockatoo (Eolophus roseicapillus). II. Inertial and aerodynamic reorientation. The Journal of Experimental Biology, 210(11):1912–1924, 2007.
[HZ03] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2nd edition, 2003.
[Kap09] David M. Kaplan. Master’s thesis, Work in Progress, Cornell University, 2009.
[KM85] A. S. King and J. McLelland, editors. Form and Function in Birds, volume 3. Academic Press, Inc., 1985.
84

[Kwo98] Young-Hoo Kwon. DLT method, 1998. Available from: http://www. kwon3d.com/theory/dlt/dlt.html.
[Let04] Henry H. Letteron. Polyhedral hull online compositing system: Reconstruction and shadowing. Master’s thesis, Cornell University, 2004.
[MLC06] G. E. Marai, David H. Laidlaw, and Joseph J. Crisco. Superresolution registration using tissue-classified distance fields. IEEE Transactions on Medical Imaging, 25(2):177–187, 2006.
[PH06] Sang Il Park and Jessica K. Hodgins. Capturing and animating skin deformation in human motion. In SIGGRAPH ’06: ACM SIGGRAPH 2006 Papers, pages 881–889, New York, NY, USA, 2006. ACM.
[PL93] Noble S. Proctor and Patrick J. Lynch. Manual of Ornithology. Yale University Press, 1993.
[PV78] P. P. Purslow and J. F. V. Vincent. Mechanical properties of primary feathers from the pigeon. The Journal of Experimental Biology, 72:251–260, 1978.
[Ray94] J. M. V. Rayner. Aerodynamic corrections for the flight of birds and bats in wind tunnels. Journal of Zoology, 234:537–563, 1994.
[RW99] Balajee Ramakrishnananda and Kok Cheong Wong. Animating bird flight using aerodynamics. Visual Computer, 15(10):494–508, 1999.
[RWID+08] Daniel K. Riskin, David J. Willis, José Iriarte-Díaz, Tyson L. Hedrick, Mykhaylo Kostandov, Jian Chen, David H. Laidlaw, Kenneth S. Breuer, and Sharon M. Swartz. Quantifying the complexity of bat wing kinematics. Journal of Theoretical Biology, 254:604–615, 2008.
[SBPE06] David A. Sibley, Louis R. Bevier, Michael A. Patten, and Chris S. Elphick. Comment on "Ivory-billed woodpecker (Campephilus principalis) persists in continental North America". Science, 311(5767):1555a, 2006.
[Sco03] Remington Scott. Sparking life: Notes on the performance capture sessions for The Lord of the Rings: The Two Towers. SIGGRAPH Computer Graphics, 37(4):17–21, 2003.
[SNF05] Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw. Automatic determination of facial muscle activations from sparse motion capture marker data. In SIGGRAPH ’05: ACM SIGGRAPH 2005 Papers, pages 417–425, New York, NY, USA, 2005. ACM.
85

[Tob07] Bret W. Tobalske. Biomechanics of bird flight. The Journal of Experimental Biology, 210(18):3135–3146, 2007.
[UHB03] James R. Usherwood, Tyson L. Hedrick, and Andrew A. Biewener. The aerodynamics of avian take-off from direct pressure measurements in Canada geese (Branta canadensis). The Journal of Experimental Biology, 206(22):4051–4056, 2003.
[UHMB05] James R. Usherwood, Tyson L. Hedrick, Craig P. McGowan, and Andrew A. Biewener. Dynamic pressure maps for wings and tails of pigeons in slow, flapping flight, and their energetic implications. The Journal of Experimental Biology, 208(2):355–369, 2005.
[Vaz94] R. J. Vazquez. The automating skeletal and muscular mechanisms of the avian wing (Aves). Zoomorphology, 114(1):59–71, 1994.
[Vid05] John J. Videler. Avian Flight. Oxford University Press, 2005. [Wan07] Jeffrey M. Wang. Animating the ivory-billed woodpecker. Master’s
thesis, Cornell University, 2007. [WBL+08] Jeffrey M. Wang, Kimberly Bostwick, Martjan Lammertink, John
Fitzpatrick, Fola Akinola, Daniel Lepage, and Donald P. Greenberg. Pattern matching an ivory-billed woodpecker model to single view video footage. Work in Progress, Cornell University, 2008. [WP03] Jia-chi Wu and Zoran Popović. Realistic modeling of bird flight animations. In SIGGRAPH ’03: ACM SIGGRAPH 2003 Papers, pages 888–895, New York, NY, USA, 2003. ACM.
86