Transducers & Instrumentation
Module 2 — Kinematics and Kinetics
Movement is what nervous systems are for
Everything a living organism does to the world, it does by moving.
“We have a brain for one reason and one reason only, and that’s to produce adaptable and complex movements.”
“Movement is the only way you have of affecting the world around you.”
— Daniel Wolpert, The real reason for brains, TEDGlobal 2011
Polycarpa aurata, Komodo. Nick Hobgood, CC BY-SA 3.0, via Wikimedia Commons
Wolpert’s evidence: the sea squirt swims until it finds a rock, attaches permanently, and then resorbs most of its own nervous system. Movement over, brain no longer worth its metabolic cost.
A significant amount of anatomical and physiological resources are dedicated to movement production: joints, muscles, skin, eyes, vestibular system, and the brain.
Various neurological and musculoskeletal conditions can impair healthy movement.
Describing movements with numbers
Movement description beyond words
Movements are changes in the position of an object of interest over time.
Position is a vector quantity describing the spatial location of an object with respect to a chosen reference frame.
A reference frame is two independent choices: an origin \(O_{A}\) (where \(x=0\) sits) and a basis \(\v{x}_{A}\) (a direction and a length).
With respect to this reference frame \(A\), the positions of particles \(P_1\) and \(P_2\) are given as follows, \[ \v{r}_1^{A} = \m{3.2} \qquad \v{r}_2^{A} = \m{-1.5} \]
where \(\v{r}_{1}^{A}, \v{r}_2^{A} \in \mathbb{R}\). The superscript \(A\) says the position is expressed in reference frame \(A\); the subscripts \(1\) and \(2\) are the particle numbers. A subscript also labels the frame’s own parts, e.g. \(O_A\). The numbers are the coefficients of \(\v{r}\) in the basis of that frame, which here is just \(\v{x}_{A}\).
Who decides the reference frame?
Why did we choose one reference frame over another?
Anyone! We choose what is natural and convenient for the problem at hand.
Both the origin and the basis are fair game.
In this course, we will only choose an orthonormal basis — basis vectors have unit length and are mutually orthogonal.
Reference frame \(B\) is translated and rotated with respect to reference frame \(A\). Every point in the 1D space can be represented in either frame, but the numbers will be different.
Changing between reference frames
How do we change between reference frames?
Points \(P_1\) and \(P_2\) in the two reference frames.
\[\begin{matrix} \v{r}_1^{A} = \m{3.2} & \v{r}_2^{A} = \m{-1.5} \\ \v{r}_1^{B} = \m{2.8} & \v{r}_2^{B} = \m{7.5} \end{matrix} \]
How are these numbers related?
Requires us to do vector addition and scaling.
Changing between reference frames
How do we change between reference frames?
Vector Addition and Scaling in 1D:
Changing between reference frames
How do we change between reference frames?
Magnitude of a vector: Also referred to as the 2-norm, Euclidean norm, or length of a vector.
Changing between reference frames
How do we change between reference frames?
Standard Inner Product between two vectors.
Changing between reference frames
How do we change between reference frames?
Two pieces of information to change between reference frames:
Rotation of the basis of the target (\(B\)) w.r.t. the base frame (\(A\)). Represented by a rotation matrix \(\v{R}_{B}^{A}\).
Translation of the origin of the target frame (\(B\)) w.r.t. the base frame (\(A\)), represented in the base frame. Represented by a translation vector \(\v{o}_{B}^{A}\).
Changing between reference frames
Rotation Matrix
Rotation Matrix: \(\v{R}_{B}^{A} \in \mathbb{R}^{1 \times 1}\) — Represents what the basis vectors of the frame \(B\) (subscript) look like in the frame \(A\) (superscript).
\[ \v{R}_{B}^{A} = \m{\v{x}_{A}^\top \v{x}_{B}} \]
In the above example, \(\v{R}_{B}^{A} = \m{-1}\), because \(\Vert \v{x}_A \Vert = 1\), \(\Vert \v{x}_B \Vert = 1\) and \(\theta = 180^\circ\).
Changing between reference frames
Translation Vector
Translation Vector: \(\v{o}_{B}^{A} \in \mathbb{R}^{1 \times 1}\) — Represents the position of the origin of the frame \(B\) (subscript) in the frame \(A\) (superscript). In the above example, \(\v{o}_{B}^{A} = \m{6}\).
The full description of the reference frame \(B\) w.r.t. the reference frame \(A\) is given by the pair \(\left( \v{R}_{B}^{A}, \v{o}_{B}^{A} \right)\).
This pair can be used to transform any vector expressed in frame \(B\) to frame \(A\).
Changing between reference frames
Going back and forth between frames
Given the pair \(\left( \v{R}_{B}^{A}, \v{o}_{B}^{A} \right)\), we can transform any vector expressed in frame \(B\) to frame \(A\) using the following equation:
\[ \v{r}_{i}^{A} = \v{R}_{B}^{A} \, \v{r}_{i}^{B} + \v{o}_{B}^{A} \]
Product of a matrix with a vector results in a vector.
In 1D, this product is very simple. We simply multiply the individual elements of the matrix and the vector.
Changing between reference frames
Going back and forth between frames
Given the pair \(\left( \v{R}_{B}^{A}, \v{o}_{B}^{A} \right)\), how do we go the other way around? \(\v{r}_{i}^{A} \mapsto \v{r}_{i}^{B}\).
\[ \v{r}_{i}^{B} = {\v{R}_{B}^{A}}^{-1} \left( \v{r}_{i}^{A} - \v{o}_{B}^{A} \right) = \v{R}_{A}^{B} \v{r}_{i}^{A} + \v{o}_{A}^{B} \]
where, \({\v{R}_{B}^{A}}^{-1} = \v{R}_{A}^{B}\), and \(\v{o}_{A}^{B} = -{\v{R}_{B}^{A}}^{-1}\v{o}_{B}^{A}\).
We obtain \({\v{R}_{B}^{A}}^{-1}\) by inverting the single number in the matrix. So, for the above example, we have, \[ {\v{R}_{B}^{A}}^{-1} = \v{R}_{A}^{B} = \m{\frac{1}{-1}} = \m{-1} \qquad \v{o}_{A}^{B} = -{\v{R}_{B}^{A}}^{-1} \v{o}_{B}^{A} = -\m{-1}\m{6} = \m{6} \]
Changing between reference frames
Going back and forth between frames \(A\) and \(B\)
Given: \(\v{R}_{A}^{B} = \m{-1}\) and \(\v{o}_{A}^{B} = \m{6}\).
Going from \(A\) to \(B\).
\[ P_1: \m{3.2} = \m{-1} \m{2.8} + \m{6} = \m{-2.8} + \m{6} = \m{3.2} \] \[ P_2: \m{-1.5} = \m{-1} \m{7.5} + \m{6} = \m{-7.5} + \m{6} = \m{-1.5} \]
Going from \(B\) to \(A\).
\[ P_1: \m{2.8} = \m{-1} \m{3.2} + \m{6} = \m{-3.2} + \m{6} = \m{2.8} \] \[ P_2: \m{7.5} = \m{-1} \m{-1.5} + \m{6} = \m{1.5} + \m{6} = \m{7.5} \]
Changing between reference frames
Now a third reference frame!
Let’s try a problem by bringing in a third reference frame \(C\).
- Find the pair \(\left( \v{R}_{C}^{A}, \v{o}_{C}^{A} \right)\). \(\longrightarrow \m{1.0}, \m{-2}\)
- Find the pair \(\left( \v{R}_{A}^{C}, \v{o}_{A}^{C} \right)\). \(\longrightarrow \m{1.0}, \m{2}\)
- Find \(\v{r}_{3}^{B}\). \(\longrightarrow \m{-1.5}\)
- Find \(\v{r}_{3}^{C}\). \(\longrightarrow \m{9.5}\)
- Find the pair \(\left( \v{R}_{C}^{B}, \v{o}_{C}^{B} \right)\). \(\longrightarrow \m{-1.0}, \m{8}\)
- Find the pair \(\left( \v{R}_{B}^{C}, \v{o}_{B}^{C} \right)\). \(\longrightarrow \m{-1.0}, \m{8}\)
Describing movements with numbers in 2D
Movement description in 2D
In the 2D real plane, we can represent the position of individual points using two numbers after we choose a reference frame — origin and basis vectors.
Reference frame \(A\): origin \(\v{o}_{A}\) and orthonormal basis \(\{ \v{x}_A, \v{y}_A \}\), where \(\Vert \v{x}_{A} \Vert = \Vert \v{y}_{A} \Vert = 1\) and \(\v{x}_{A}^\top \v{y}_{A} = 0\).
Points in the plane are represented as column vectors \(\v{r}_i^{A} \in \mathbb{R}^2\) w.r.t. the reference frame \(A\). \[ \v{r}_{1}^{A} = \m{ r_{11}^A \\ r_{12}^A} = \m{ \v{x}_A^\top \v{r}_1^{A} \\ \v{y}_A^\top \v{r}_1^{A}} = \m{ \Vert \v{x}_A \Vert \Vert \v{r}_1^{A} \Vert \cos \left( \theta_1 \right) \\ \Vert \v{y}_A \Vert \Vert \v{r}_1^{A} \Vert \sin \left( \theta_1 \right)} = \m{ \Vert \v{r}_1^{A} \Vert \cos \left( \theta_1 \right) \\ \Vert \v{r}_1^{A} \Vert \sin \left( \theta_1 \right)} \]
Describing movements with numbers in 2D
Vector addition in 2D
Vector addition in \(\mathbb{R}^2\)
Describing movements with numbers in 2D
Vector scaling in 2D
Vector scaling in \(\mathbb{R}^2\)
Describing movements with numbers in 2D
Movement description in 2D
The representation of \(P_1\) and \(P_2\) in reference frame \(A\) is \(\v{r}_{1}^{A} = \m{3.2 \\ 2.1}\) and \(\v{r}_{2}^{A} = \m{-1.5 \\ 1.2}\), respectively.
The following is what these numbers mean:
\[ \v{r}_{1}^{A} = 3.2 \v{x}_A + 2.1 \v{y}_A \qquad \qquad \v{r}_{2}^{A} = -1.5 \v{x}_A + 1.2 \v{y}_A\]
Different reference frames in 2D
Reference frame sharing the origin
Consider a second reference frame \(B\) shown in the figure. Both these reference frames share the same origin, but are rotated with respect to each other.
Note: \(\Vert \v{x}_{B} \Vert = \Vert \v{y}_{B} \Vert = 1\) and \(\v{x}_{B}^\top\v{y}_{B} = 0\).
The rotation of frame \(B\) with respect to frame \(A\) is given by a \(2 \times 2\) real matrix \(\v{R}_{B}^{A} \in \mathbb{R}^{2 \times 2}\).
\[ \v{R}_{B}^{A} = \m{\v{x}_{B}^{A} & \v{y}_{B}^{A}} = \m{\v{x}_{A}^\top \v{x}_{B} & \v{x}_{A}^\top \v{y}_{B} \\ \v{y}_{A}^\top \v{x}_{B} & \v{y}_{A}^\top \v{y}_{B}}\]
\[ \v{R}_{B}^{A} = \m{ \cos\left( \theta \right) & -\sin\left( \theta \right) \\ \sin\left( \theta \right) & \cos\left( \theta \right) }\]
Representation of \(P_1\) in reference frames \(A\) and \(B\) are \(\v{r}_{1}^{A}\) and \(\v{r}_{1}^{B}\), respectively. \[ \v{r}_{1}^{A} = \m{r_{11}^{A} \\ r_{12}^{A}} \qquad \v{r}_{1}^{B} = \m{r_{11}^{B} \\ r_{12}^{B}} \qquad \v{r}_{1}^{A} = \v{R}_{B}^{A} \, \v{r}_1^{B} \]
Different reference frames in 2D
Let’s do an example
Find the following.
- \(\v{r}_{P}^{A}\). \(\longrightarrow \m{\sqrt{2} \\ \sqrt{2}}\)
- \(\v{R}_{B}^{A}\). \(\longrightarrow \frac{1}{2}\m{\sqrt{3} & -1 \\ 1 & \sqrt{3}}\)
- \(\v{R}_{A}^{B}\). \(\longrightarrow \frac{1}{2}\m{\sqrt{3} & 1 \\ -1 & \sqrt{3}}\)
- \(\v{R}_{C}^{A}\). \(\longrightarrow \frac{1}{\sqrt{2}}\m{-1 & -1 \\ 1 & -1}\)
- \(\v{R}_{A}^{C}\). \(\longrightarrow \frac{1}{\sqrt{2}}\m{-1 & 1 \\ -1 & -1}\)
- \(\v{r}_{P}^{B}\). \(\longrightarrow \frac{1}{2}\m{\sqrt{6}+\sqrt{2} \\ \sqrt{6}-\sqrt{2}}\)
- \(\v{r}_{P}^{C}\). \(\longrightarrow \m{0 \\ -2}\)
- \(\v{R}_{B}^{C}\). \(\longrightarrow \frac{1}{4}\m{\sqrt{2}-\sqrt{6} & \sqrt{6}+\sqrt{2} \\ -\sqrt{6}-\sqrt{2} & \sqrt{2}-\sqrt{6}}\)
- \(\v{R}_{C}^{B}\). \(\longrightarrow \frac{1}{4}\m{\sqrt{2}-\sqrt{6} & -\sqrt{6}-\sqrt{2} \\ \sqrt{6}+\sqrt{2} & \sqrt{2}-\sqrt{6}}\)
Different reference frames in 2D
Let’s now shift the origin
What is \(\v{R}_{B}^{A}\)? \(\longrightarrow \m{1 & 0 \\ 0 & 1} = \v{I}\)
What is \(\v{r}_{P}^{B}\)? \(\longrightarrow \m{1 \\ 4}\)
What is \(\v{R}_{C}^{B}\)? \(\longrightarrow \m{\frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \frac{1}{2} & \frac{\sqrt{3}}{2}}\)
How do we change between reference frames \(A\) and \(B\)? We have rotation and the translation. \(\left( \v{R}_{B}^{A}, \v{o}_{B}^{A}\right)\).
\[ \v{r}_{P}^{A} = \v{R}_{B}^{A} \v{r}_{P}^{B} + \v{o}_{B}^{A} \] \[ \v{r}_{P}^{B} = {\v{R}_{B}^{A}}^{-1} \left( \v{r}_{P}^{A} - \v{o}_{B}^{A} \right) \]
\[\implies \v{R}_{A}^{B} = {\v{R}_{B}^{A}}^{-1} = {\v{R}_{B}^{A}}^{\top} \qquad \v{o}_{A}^{B} = -{\v{R}_{B}^{A}}^{\top}\v{o}_{B}^{A}\]
Different reference frames in 2D
Let’s now shift the origin
What is \(\v{r}_{P}^{C}\) if we know \(\v{r}_P^{B}\)?
We need \(\left( \v{R}_{C}^{B}, \v{o}_{C}^{B}\right) \longrightarrow \v{r}_{P}^{C} = \v{R}_{B}^{C} \left(\v{r}_{P}^{B} - \v{o}_{C}^{B} \right)\)
\[\v{r}_{P}^{C} = \m{\frac{\sqrt{3}}{2} + 2 \\ -\frac{1}{2} + 2\sqrt{3}}\]
What is \(\v{r}_{P}^{C}\) if we know \(\v{r}_P^{A}\)?
We need \(\left( \v{R}_{C}^{A}, \v{o}_{C}^{A}\right)\)
\[\v{R}_{C}^{A} = \v{R}_{B}^{A}\v{R}_{C}^{B} \qquad \v{o}_{C}^{A} = \v{o}_{B}^{A} + \v{R}_{B}^{A} \v{o}_{C}^{B}\]
\[\v{r}_{P}^{C} = \v{R}_{A}^{C}\left(\v{r}_{P}^{A} - \v{o}_{C}^{A}\right)\]
What is \(\v{r}_{P}^{C}\), if \(\v{o}_{C}^{B} = \m{1 \\ 1}\)? \(\v{r}_{P}^{C} = \m{\frac{3}{2} \\ \frac{3\sqrt{3}}{2}}\)
The body as connected rigid bodies
Full body kinematics: Track one body segment and the joint angles of the connected segments.
Individual joint kinematics: Track only the segments connected at the joints.
How do we measure kinematics?
General approach: track all 6DOFs of each limb segment \(\longrightarrow\) derive the individual joint variables.
- Pros: Full characterisation; Suitable for complex movement behaviors (e.g. gait analysis, upper-limb reaching, object manipulation, etc.).
- Cons: Too much information for simple movements (e.g. knee flexion/extension, elbow range of motion); Complex, expensive tools required.
Joint level approach: track specific joint variables of interest.
- Pros: Simple, compact tools. Focused, relevant information for simple movements.
- Cons: Unsuitable for complex movements.
We will explore sensing tools for both approaches.
- Camera-based motion tracking with markers (standard & fiducial).
- Inertial measurement units
- Optical and magnetic encoders
Camera-based movement tracking
Marker-based motion-capture suit. Image: cgicoffee.com
- Each marker is treated as a dot or point in space.
- \((x_i, y_i, z_i)\) position of the \(i^{th}\) marker estimated from the cameras in a lab-fixed reference frame.
\[ \left(x_i, y_i, z_i \right)_{i=1}^{M} \longrightarrow \left(x_j, y_j, z_j, \alpha_j, \beta_j, \gamma_j \right)_{j=1}^{L} \longrightarrow \left(\theta_k, \phi_k, \chi_k \right)_{k=1}^{J}\]
Camera-based movement tracking
A camera captures the spatial distribution of light arriving from the environment — it records the direction from which light arrives, not the distance it has travelled.
Camera optics produce an image of the world, captured by an image sensor (CCD or CMOS) in digital cameras, or by film in traditional cameras.
An image \(I\left(x, y\right)\) is a 2D distribution of brightness of light. For a camera, the 2D surface is a rectangular region \(\Omega\). \[ I: \Omega \subset \mathbb{R}^2 \mapsto \mathbb{R}_{+}; \quad \Omega \ni \left(x, y\right) \mapsto I\left(x, y\right) \in \mathbb{R}_{+} \]
Image intensity \(I\left(x, y\right)\) at any location \(\left(x, y\right)\) is determined by the amount of light entering the camera through its optical system that directs the light to the image sensor.
Geometry of the optics determines the mapping of the spatial location of a source of light in the world \(\left(x^W, y^W, z^W\right)\) to a spatial location of the image \(\left(x^I, y^I\right)\).
We will primarily focus on the geometry of the optics for the purpose of kinematics tracking using camera image.
Camera-based movement tracking
Physics of a Pinhole Camera
A useful and simplified model of the geometry of a camera’s optics is provided by a pinhole camera.
A box with an infinitesimally small hole that lets only a single ray from each point enter the box.
Produces an inverted image of the world on the back wall.
Camera-based movement tracking
Physics of a Pinhole Camera
\(P\) in space has coordinates \(\v{r}_P^C = \m{x_P^C & y_P^C & z_P^C}^\top \in \mathbb{R}^3\) w.r.t. the camera reference frame.
\(p\) is the corresponding point in the image with coordinates \(\v{r}_p^I = \m{x_p^I & y_p^I}^\top \in \mathbb{R}^2\) w.r.t. the image reference frame.
\(\mathbf{o}_I\) is principal point of the image, i.e. the point where the optical axis intersects the image plane.
Camera-based movement tracking
Physics of a Pinhole Camera
Assume the image plane to be in front of the camera center, then we have,
\[x_p^I = \frac{f}{z_P^C} x_P^C \qquad y_p^I = \frac{f}{z_P^C} y_P^C \implies z_P^C \m{x_p^I \\ y_p^I \\ 1} = \m{f & 0 & 0 & 0\\ 0 & f & 0 & 0\\ 0 & 0 & 1 & 0}\m{x_P^C \\ y_P^C \\ z_P^C \\ 1}\]
Camera-based movement tracking
Physics of a Pinhole Camera
With digital cameras, positions on an image are reported in pixels. Each pixel corresponds to a sensing area on the image plane.
We need to convert from \(\v{r}_p^I\) (e.g. units of mm) to sensor coordinates \(\v{r}_p^S\) in pixels.
The origin and the coordinate axes for the sensor frame \(S\) can be different.
How do we convert from \(\v{r}_p^I\) to \(\v{r}_p^S\)?
Camera-based movement tracking
Physics of a Pinhole Camera
How do we convert from \(\v{r}_p^I\) to \(\v{r}_p^S\)?
\[\v{r}_p^S = \m{s_x & s_{\theta} & 0 \\ 0 & s_y & 0 \\ 0 & 0 & 1} \m{1 & 0 & u \\ 0 & -1 & v \\ 0 & 0 & 1} \v{r}_p^I\]
\[\begin{align} x_p^S &= s_x\left(x_p^I + u\right) + s_\theta \left(-y_p^I + v\right) \\ y_p^S &= s_y\left(-y_p^I + v\right) \end{align}\]
\[\v{r}_p^S = \m{s_x & s_{\theta} & 0 \\ 0 & s_y & 0 \\ 0 & 0 & 1} \m{1 & 0 & u \\ 0 & -1 & v \\ 0 & 0 & 1} \m{f & 0 & 0\\ 0 & f & 0\\ 0 & 0 & 1} \m{\v{R}_{W}^{C} & \v{o}_{W}^{C}} \v{r}_P^W = \v{K}_f \m{\v{R}_{W}^{C} & \v{o}_{W}^{C}} \v{r}_P^W\]
Looks like there are additional 6 parameters associated with the camera itself – \(\left(f, s_x, s_y, s_\theta, u, v\right)\). But there are only 5!
\[\v{K}_f = \m{s_x f & -s_{\theta} f & s_x u + s_\theta v \\ 0 & -s_y f & s_y v \\ 0 & 0 & 1}\]
Camera-based movement tracking
Camera calibration — Zhang’s method
The method every calibration tool uses. It needs only a flat printed target — no machined 3D rig.
You provide
- A checkered board of known square size.
- Many photographs of it — 15 to 20 — held at different tilts, distances, and positions in the frame.
The software returns
- The 5 intrinsic parameters.
- The distortion parameters \(k_1, k_2, k_3, p_1, p_2\).
- Where the board was in each photograph.
How it decides:
- Positions of the corners on the checkered board are known.
- Detects the corners for the given size; this provides the correspondence between the 3D points and the image points.
- Searches for the set of numbers that best predicts where every corner actually appeared
- Leftover mismatch, in pixels, is the reprojection error.
Camera-based movement tracking
Human movement tracking setup
When tracking full body kinematics typically interested in:
- Movement of the whole-body centre of mass.
- Relative movement between adjacent bones or joint kinematics.
Tracking kinematics is acheived by placing markers on the individual body segments, tracked using a multi-camera setup.
Each rigid body has 3-4 markers to track its pose in 3D.
Camera-based movement tracking
Human movement tracking setup
Each rigid body has 3-4 markers to track its pose in 3D.
Consider three markers \(m_1, m_2\) and \(m_3\) on a rigid body \(B\).
The cameras give \(\v{r}_{m_1}^{A}, \v{r}_{m_2}^{A}, \v{r}_{m_3}^{A}\).
Two vectors in the body \[\v{d}_2 = \v{r}_{m_2}^{A} - \v{r}_{m_1}^{A} \qquad \v{d}_3 = \v{r}_{m_3}^{A} - \v{r}_{m_1}^{A}\]
First axis \(\quad \v{x}_{B}^{A} = \dfrac{\v{d}_2}{\Vert \v{d}_2 \Vert}, \qquad \v{o}_{B}^{A} = \v{r}_{m_1}^{A}\)
Normal to the marker plane \(\quad \v{z}_{B}^{A} = \dfrac{\v{x}_{B}^{A} \times \v{d}_3}{\Vert \v{x}_{B}^{A} \times \v{d}_3 \Vert}\)
Close the frame \(\quad \v{y}_{B}^{A} = \v{z}_{B}^{A} \times \v{x}_{B}^{A}\) \[\v{R}_{B}^{A} = \m{\v{x}_{B}^{A} & \v{y}_{B}^{A} & \v{z}_{B}^{A}}\]
\[\v{o}_{B}^{A} = \v{r}_{m_{1}}^{A}\]
Camera-based movement tracking
Human movement tracking setup
\[\v{R}_{B_d}^{B_p} = {\v{R}_{B_p}^{A}}^\top\v{R}_{B_d}^{A} \longrightarrow \left( \alpha_{pd}, \beta_{pd}, \gamma_{pd}\right)\]
There are various conventions for describing these joint angles, but an anatomical one is preferred for easy clinical interpretation, e.g. flexion/extension, abduction/adduction, and internal/external rotation.
Anatomical joint angles require appropriate marker placement on the proximal and distal limb segments.
Two sets of markers, or reference frames, are typically employed for this purpose:
- Anatomical reference frame: built from markers on palpable bony landmarks, so its axes have anatomical meaning — but those markers are prone to occlusion during movement.
- Tracking reference frame: built from a cluster of markers placed conveniently on the limb segment; it has no anatomical meaning of its own, so a static calibration trial records where the anatomical frame sits in it.
After Uchida & Delp, Biomechanics of Movement (2021), Ch. 7.
Camera-based movement tracking
Human movement tracking setup
Static calibration is done to find the relationship between the two reference frames:
\[\v{R}_{anatomical}^{tracking}\]
The anatomically relevant joint angles can then be estimated from the tracking reference frames.
\[\v{R}_{tibia}^{femur} = \v{R}_{thigh}^{femur}\v{R}_{lab}^{thigh}\v{R}_{shank}^{lab}\v{R}_{tibia}^{shank}\]
\(\v{R}_{tibia}^{femur}\) can be decomposed into anatomically relevant joint angles.
This process of obtaining the joint angles from the marker measurements, with no other assumptions, is called unconstrained inverse kinematics.
Camera-based movement tracking
Human movement tracking setup
Constrained inverse kinematics
Measured markers (blue) and the model’s markers (orange).
Start from a model of the skeleton: rigid segments of fixed length, joined by joints with a chosen number of DOFs (hinge knee, ball-and-socket hip…). Its configuration is one vector of generalised coordinates \(\v{q}\) — the joint angles themselves.
Markers are placed on the model at the same anatomical spots as on the subject. For any \(\v{q}\), the model predicts where every marker should be: \(\hat{\v{r}}_{m_i}^{A}(\v{q})\).
At each instant, find the pose that puts the model’s markers closest to the measured ones, all at once: \[\v{q}^{*} = \arg\min_{\v{q}} \sum_{i} w_i \left\Vert \v{r}_{m_i}^{A} - \hat{\v{r}}_{m_i}^{A}(\v{q}) \right\Vert^2\] A weighted least-squares fit; the joint constraints are built into the model, so they can never be violated.
What it buys: joint angles come out directly; uncorrelated marker noise is averaged across all markers rather than passed through; a lost marker is tolerated; segments never come apart.
What it costs: the model must first be scaled to the subject; any joint the model gets wrong (a knee that is not a hinge, a lax joint) is forced to behave as the model says — and the residual marker error is the only warning you get; soft-tissue artefact moves markers together, so it is not averaged away — the constraints can only push it into the joint angles the model allows.
Camera-based movement tracking
Tracking with fiducial markers — what a tag is
A fiducial is a printed pattern with two parts, and each has one job.
The square shape — finds the tag. Four straight edges of a black border, picked out even when small, oblique or partly covered. But every dark square-ish thing passes this test: windows, tiles, the paper it is printed on.
The bits inside — say what it is. Read the 36 cells and check them against the code book: is this a tag at all? (reject the windows), which tag? (thousands of IDs), and which way up? — a bare square is the same turned by 90°, the code is not, so the code says which corner is which.
Olson, E. (2011). AprilTag: A robust and flexible visual fiducial system. Proc. IEEE ICRA, 3400–3407.
Camera-based movement tracking
Tracking with fiducial markers — how the detector finds a tag
Wang, J. & Olson, E. (2016). AprilTag 2: Efficient and robust fiducial detection. Proc. IEEE/RSJ IROS. Photo: NASA Kennedy, Swarmathon, CC BY-SA 2.0, from the test set of the AprilTag library. Stage images are the library’s own debug output (tag36h11); quads, ids and the sample grid are drawn from its results.
0. The photograph. A robot holding a tagged cube: two tag faces, and some decoys — the round sensor housings, a cable, a tyre. The detector works on the grey-scale image only.
1. Adaptive threshold. Each pixel is called black or white against the darkest and brightest pixels in its own neighbourhood; uneven lighting does not matter. Where the two are too close, flat asphalt, the pixel is left undecided.
2. Connected components. Every blob of black and every blob of white gets a label, one colour each. The tag’s border is one black blob; the quiet zone around it one white blob.
3. Boundaries. The pixels where a black blob meets a white blob, grouped by which pair of blobs they separate. Each closed outline is one candidate for a border.
4. Quads. Each outline is tested for four corners and fitted with four lines. 14 candidates here — the two tag faces, and every sensor housing and cable end that is square-ish enough. The shape alone passes them all.
5. Decode. The homography from each quad’s corners samples the 8 × 8 cells; the 36 bits are matched to the code book. Two accepted, both id 0, with the corner order fixed by the code; the other twelve rejected.
Why this pipeline. The 2011 detector found edges from image gradients and chained line segments into quads. The 2016 redesign replaced that with the threshold-and-components stages above, for speed and better detection of small tags; the tag design and the decode are unchanged.
Camera-based movement tracking
What limits tag tracking — I. Distance
Effect of marker distance on pose estimation
Tag size in image \(s = f d / z\) pixels with pixel error \(\sigma\) pixels.
Example: \(d = 0.10\) m, \(f = 800\) px \(\implies s = 80/z\): 100 px @ 0.8 m, 40 px @ 2 m. \(\implies\) Tag’s \(\text{SNR} \propto \frac{1}{z}\)
Depth is computed from the tag’s image size \[z = \frac{f d}{s} \;\Rightarrow\; \delta z = -\frac{f d}{s^2}\,\delta s = -\frac{z^2}{f d}\,\delta s\] \[|\delta z| \approx \frac{\sigma\, z^2}{f d}\]
Double the distance, four times the depth error.
Projected corners with 0.5 px of Gaussian noise, pose by the closed form and refinement, median and interquartile range of 500 runs.
Camera-based movement tracking
What limits tag tracking — I. Distance
How a tilt shows up in the image
Foreshortening: a length seen at an angle looks shorter.
- Tag’s height \(d\) becomes \(d\cos\theta\) and its width at the centre is still \(s = fd/z\).
- The tilt is read from the ratio of two pixel lengths, \(h/s = \cos\theta\).
- The top and bottom edges sit at depths \(z \mp \tfrac{d}{2}\sin\theta\). The difference between the near and far edges is \(\approx s\,(d/z)\sin\theta\).
Camera-based movement tracking
What limits tag tracking — I. Distance
Effect of marker distance on pose estimation
Same tag, same camera, same noise: \(s = 80/z\) px and \(\sigma = 0.5\) px.
Orientation comes from foreshortening: the tilt is read from the ratio \(h/s = \cos\theta\) of two pixel lengths, each off by about \(\sigma\): \[\delta(\cos\theta) \approx \frac{\sigma}{s} \;\Rightarrow\; \delta\theta \approx \frac{\sigma}{s\,\sin\theta} \;\propto\; \frac{\sigma\, z}{f d}\] Double the distance, twice the orientation error — gentler than depth. At \(30^\circ\) and 100 px: \(2 \times 0.5/100\) rad \(\approx 0.6^\circ\); the simulation gives \(0.85^\circ\).
Projected corners with 0.5 px of Gaussian noise, pose by the closed form and refinement, median and interquartile range of 500 runs.
Camera-based movement tracking
What limits tag tracking — II. Face-on
Change the tilt by 5°. How much does the picture change?
The same 100 px tag, drawn from the camera model at six tilts about its horizontal axis.
At tag face-on: \(\cos\theta = h/s\). Tag’s true height 100 px, measured 99.5 px because of the noise. \(\cos\theta = 0.995 \implies \theta = 5.7°\).
At an oblique angle:: At 40°, true height 76.7 px, measured 76.2 px: \(\theta = 40.4°\).
This is Module 1’s rule. A sensor gives \(y = \mathcal{F}(x_m) + e_n\) and the noise referred back to the measurand is \(e_n \big/ \frac{\partial \mathcal{F}}{\partial x_m}\). Tag height \(h = s\cos\theta\), the measurand is \(\theta\), the sensitivity is \(\partial h/\partial\theta = -s\sin\theta\) and the noise is \(\sigma\), so \(\delta\theta \approx \sigma/(s\sin\theta)\) A face-on tag has zero sensitivity.
Camera-based movement tracking
What limits tag tracking — II. Face-on
The simulation agrees. Orientation error against tilt for the same tag, with the same corner noise as before.
Error \(\approx\) noise \(\div\) sensitivity: \[\delta\theta \;\approx\; \frac{\sigma}{s\,\sin\theta}\]
The dashed line is this law; it tells us to expect the error to go up as the tag becomes more face-on.
Same model as the distance slides, tag fixed at 0.8 m.
Camera-based movement tracking
What limits tag tracking — III. Edge-on
Turn the tag towards edge-on. What happens to the cells?
The real tag pattern, projected through the camera model at five tilts, with the grid of 1 px squares drawn over it. A small tag, 4 px per cell face-on, so that the pixels can be seen.
- Both directions change, but not alike. The columns converge by a small amount 4 px to 3.9 px. The rows collapse by \(\cos\theta\): 4 px, 3 px, 2 px, 1 px, 0.3 px.
- Once a row is squeezed to very few pixels, lens blur will average neighbouring black and white cells to grey, impacting payload detection.
Camera-based movement tracking
What limits tag tracking — III. Edge-on
With increasing tilt, reading of the payload bits worsens.
The height of the far row, for three sizes of tag in the image (tags are at different distances).
Usable tilt depends on the tag’s image size: about 80° for 100 px, 70° for 50 px, 50° for 25 px. Olson’s 70° is this number for his camera and tag.
Geometry only; the 2 px threshold is roughly twice a typical lens blur. Figure source: limit-oblique.tex.
Camera-based movement tracking
What limits tag tracking — IV. Motion blur
Move the tag during the exposure. What happens to the cells?
The same tag as before, tilted 30°, sliding sideways while the shutter is open. Each pixel records the average of everything that passed over it.
With exposure \(T_{\mathrm{exp}}\) a tag moving at \(v\) across the view travels \(v\,T_{\mathrm{exp}}\) metres, which is \(b = v\,T_{\mathrm{exp}}\,f/z\) pixels in the image. Every edge becomes a ramp \(b\) pixels long.
Tag corners may still be found, but will introduce pose estimation errors due to errors in corner detection.
Payload detection errors occur because the payload cells get averaged out.
Inertial measurement units
What an IMU is
An inertial measurement unit (IMU) is a chip carrying three-axis accelerometers and three-axis gyroscopes, one of each along every axis of the chip’s own frame \(S\) \(\longrightarrow\) a 6DOf sensor!
Most sensors also carry a three-axis magnetometer \(\longrightarrow\) a 9 DOF sensor! Its often referred to as a MARG (magnetic, angular rate and gravity) sensor.
Reports six numbers, typically a few hundred times a second:
- \(\v{a}_S^S(t)\): accelerometers output in the sensor frame.
- \(\boldsymbol{\omega}_S^S(t)\): gyroscope’s output in the sensor frame.
No reference outside the chip.
- A camera measures a marker against its fixed camera frame.
- An encoder measures a shaft against its housing.
- An IMU measures a proof mass against its own case, and expresses it in the case’s own axes.
Inertial measurement units
Accelerometer — the mechanism
The sensor output is \(y = x_S - x_M\), the thing the plates measure, with \(a_S = \ddot{x}_S\) the acceleration of the case: \[M\ddot{y} + B\dot{y} + K y = M\,a_S \qquad\Longrightarrow\qquad \frac{Y(s)}{A_S(s)} = \frac{M}{Ms^2 + Bs + K}\]
Inertial measurement units
Accelerometer — what the model says
Two useful numbers from the transfer function: \[\frac{Y(s)}{A_S(s)} = \frac{M}{Ms^2 + Bs + K}\]
- Static sensitivity \(= M/K\).
- Natural frequency \(\omega_n = \sqrt{K/M}\).
We ideally want both to be high, which we cannot because the two are one number!
Sensitivity and bandwidth are not a trade-off, they are an identity. Static sensitivity is \(M/K\) and \(\omega_n^2 = K/M\), so
\[\text{sensitivity} = \frac{1}{\omega_n^2}\]
for any mass and any spring. Choosing the bandwidth fixes the gap, and no choice of \(M\) escapes it.
MEMS parts choose bandwidth: \(\omega_n\) of a few kilohertz, flat over anything a limb does, and therefore a displacement of tens of nanometres per \(g\).
Module 1 again: we only need to do static calibration to find out the static sensitivity; the sensor’s dynamics can be ignored for human movements.
Inertial measurement units
Accelerometer — gravity is in the signal
The spring cannot tell the difference between the case accelerating upward and the case sitting still while gravity pulls the mass down. So the accelerometer reports, in its own frame, \[\v{a}_S^S(t) = \v{R}_E^S(t)\left(\v{a}_S^E(t) - \v{g}^E\right)\]
Two consequences:
- At rest it does not read zero. It reads \(+1\,g\) pointing up, along whichever of its axes happens to be up.
- The reading depends on the orientation \(\v{R}_E^S\) of the chip, which the chip does not know.
Both problems are addressed if \(\v{R}_E^S(t)\) is known. \[ \v{a}_{S}^{E}(t) = \left(\v{R}_{E}^{S}(t)\right)^\top \v{a}_{S}^{S}(t) + \v{g}_{}^{E}\]
Undo the rotation, remove the effect of gravity to obtain the linear acceleration.
Inertial measurement units
Accelerometer — using gravity to estimate orientation
Orientation estimation with 2-axis Accelerometer
Sensor 1: \(\v{R}_E^S = \v{I}\) \[\v{a}_S^S = -\v{g}^E = \m{0 \\ 9.81}\]
Sensor 2: \(y_S\) points down \[\v{a}_S^S = \m{0 \\ -9.81}\]
Sensor 3: turned by \(\theta\) \[\v{a}_S^S = 9.81\m{\sin\theta \\ \cos\theta}\]
Sensor 3 is the interesting one: the reading contains \(\theta\). Turn it around and a still accelerometer is a tilt sensor.
Inertial measurement units
Accelerometer — tilt from a still sensor
Invert the reading of Sensor 3. Two ways:
- One axis: \(\theta = \arcsin(a_x / g)\). Sensitivity \(\partial a_x/\partial\theta = g\cos\theta\), which goes to zero as the axis approaches vertical. The same slope problem as the face-on tag: near \(90^\circ\) a small error in \(a_x\) is a large error in \(\theta\).
- Both axes: \(\theta = \operatorname{atan2}(a_x, a_y)\). Sensitivity \(g\) everywhere, and \(g\) itself drops out of the ratio. Always do this.