Preface:  I’m not a mathematician, just a mere user of mathematics.  All the terms tend to confuse me, so here’s my personal glossary containing links to Wikipedia and MathWorld, a formal definition from one of those two places (which are usually overcomplicated if you’re just trying to use them), and my understanding of what each term means from a practical standpoint and/or an example of it that illustrates the main idea of the term.  This list should not be considered to be authoritative or complete, as I’m simply going to add terms as I run across them.  Feel free to correct me if you see any problems.

Linear Algebra

1. Affine space – a point set with a faithful freely transitive vector space action for the vector space.  Essentially, it’s a normal vector space without an origin point, meaning we can treat any point we want as an origin during analysis (so long as we’re aware of the fact that we have no origin and will have to translate our “coordinates” if someone decides to move the origin on us).  (Wikipedia) (MathWorld)
2. Eigenvalue – (no formal definition from either place except for in mathematical symbols I can’t copy) the amount by which an eigenvector changes when multiplied by its associated matrix. So, formally, if I have a square matrix , eigenvector , and scalar , is an eigenvalue if:
   

   

   (Wikipedia) (MathWorld)

3. Eigenvector – (no formal definition from either place except for in mathematical symbols I can’t copy) a vector that, when multiplied by a specific square matrix, changes only in magnitude, not in direction. So, formally, if I have a square matrix , vector , and scalar (formally known as an eigenvalue), is an eigenvector if:
   

   

   (Wikipedia) (MathWorld)

4. Vector space – a mathematical structure formed by a collection of vectors with an single point of origin.  An example would be normal Euclidean space. (Wikipedia) (MathWorld)

Statistical Basics

1. Covariance – how much two variables change together.   (Wikipedia) (MathWorld)
2. Cross-covariance (matrix) – sometimes used to refer to the covariance cov(X, Y) between two random vectors X and Y, in order to distinguish that concept from the “covariance” of a random vector X. So the key is that cross-variance refers to the covariance between a set of vectors, not the covariance within a single vector.  (Wikipedia) (MathWorld)
3. Expected value (expectation or expectation value) – the integral of the random variable with respect to its probability measure.  Basically, it’s the value you expect to get out of  a function.  (Wikipedia) (MathWorld)
4. Moment – a quantitative measure of the shape of a set of points.   (E.g., the “width” (the second moment) or “height” of the set of points, or the mean (the first moment) of the set of points.)  (Wikipedia) (MathWorld)
5. Variance – a special case of covariance when the two variables are identical.  (I.e., when there is only one variable you’re really looking at.)  This measures how far values of the variable are from the mean of the variable.   (Wikipedia) (MathWorld)

Statistical Analysis Techniques

1. Canonical correlation analysis (CCA–be careful, there are two of these) — a method of analysis that enables us to find linear combinations of two sets of correlated variables which have maximum correlation with each other.   This can be repeated up to n times, where n is the size of the smallest set.  (Wikipedia) (MathWorld)
2. Principal component analysis (PCA) – a mathematical procedure that transforms a number of possibly correlated variables into a smaller number of uncorrelated variables called principal components.  You can think of this as taking a set of data and finding the best perspective to look at it, where the “best” perspective is the perspective that shows the most variation.  (Wikipedia) (MathWorld)

Topology

1. Bijection (bijective map) – a function f from a set X to a set Y with the property that, for every y in Y, there is exactly one x in X such that f(x) = y.  Alternatively, f is bijective if it is a one-to-one correspondence between those sets. Basically, you have two sets, X and Y, and a function f that maps every value in X to a single value in Y.  That function f is the bijection.  (Wikipedia) (MathWorld — contains a great illustration if this is confusing)
2. Diffeomorphic (diffeomorphism)-  an isomorphism in the category of smooth manifolds. It is an invertible function that maps one differentiable manifold to another, such that both the function and its inverse are smooth. (Wikipedia) (MathWorld)
3. Embedding – one instance of some mathematical structure (call it X) contained within another instance (call it Y) where X maps to Y via an injective and structure-preserving map f.  Examples include the natural numbers (X) within the integers (Y), or the integers (X) within the rational numbers (Y).  This is similar to, but distinct from, the idea of a subset.  (Embedding seems to be more general.)  (Wikipedia) (MathWorld)
4. Homomorphism –  a structure-preserving map between two algebraic structures (such as groups, rings, or vector spaces). (Wikipedia) (MathWorld)
5. Injective – a function that preserves distinctness: it never maps distinct elements of its domain to the same element of its codomain.  (Wikipedia) (MathWorld)
6. Isomorphism – a bijective map f such that both f and its inverse f ⁻¹ are homomorphisms. (Wikipedia) (MathWorld)
7. Manifold – a mathematical space that on a small enough scale resembles the Euclidean space of a specific dimension.  A line is a one-dimensional manifold since if you look at a small area around any given point of a line, that area resembles one-dimensional space.  Likewise, a sphere (specifically the surface, not the volume), is a tw0-dimensional manifold since that surface can be represented by a set of two-dimensional maps, according to Wikipedia, though to me that doesn’t quite make sense.  You can indeed represent a sphere as two-dimensional maps (check out your handy road map if you don’t believe me), but not without distorting the map somehow, usually via some sort of projection or a non-rectangular drawing.  (Wikipedia) (MathWorld)
8. Morphism – an abstraction derived from structure-preserving mappings between two mathematical structures. (Wikipedia) (MathWorld)