Complexity
Time complexity of an operation is just the number of multiplications required (we assume addition is free).
The dot product of two vectors of length n is O(n).
Computing X'X requires p^2 dot products, each requiring n multiplications, so is O(p^2 n). But note that is X is sparse, this can effectively be cheaper.
Exactly inverting a p*p matrix requires O(p^3) operations. (We'll later learn much faster approximate methods that find pseudo-inverses using SVD).
Note that for "big O" notation, we just drop the smaller terms, so O(n^2 + n) = O(n^2).
Note that O(n + p) cannot be simplified unless you know that n<< p or p << n.
For more details, see
https://www.cs.cmu.edu/~adamchik/15-121/lectures/Algorithmic%20Complexity/complexity.html
or for even more details ( more than you need for this class), see: http://www.cs.dartmouth.edu/~ac/Teach/CS19-Winter06/SlidesAndNotes/CLRS-3.1.pdf