(Taken from http://latticeqcd.org/pythonorg/static/images/antigravity.png, adapted from http://xkcd.com/353/)
Python is the basic programming languages if one wants to work on data nowadays. Its popularity comes with its intuitive syntax, its support of several programming paradigms, and the package numpy (Numerical Python). Yes, if you asked which package is a “must-have” outside the standard Python packages, I would certainly name numpy.
Let me list some useful packages that I have found useful:
- numpy: Numerical Python. Its basic data type is ndarray, which acts like a vector with vectorized calculation support. It makes Python to perform matrix calculation efficiently like MATLAB and Octave. It supports a lot of commonly used linear algebraic algorithms, such as eigenvalue problems, SVD etc. It is the basic of a lot of other Python packages that perform heavy numerical computation. It is such an important package that, in some operating systems, numpy comes with Python as well.
- scipy: Scientific Python. It needs numpy, but it supports also sparse matrices, special functions, statistics, numerical integration…
- matplotlib: Graph plotting.
- scikit-learn: machine learning library. It contains a number of supervised and unsupervised learning algorithms.
- nltk: natural language processing. It provides not only basic tools like stemmers, lemmatizers, but also some algorithms like maximum entropy, tf-idf vectorizer etc. It provides a few corpuses, and supports WordNet dictionary.
- gensim: another useful natural language processing package with an emphasis on topic modeling. It mainly supports Word2Vec, latent semantic indexing (LSI), and latent Dirichlet allocation (LDA). It is convenient to construct term-document matrices, and convert them to matrices in numpy or scipy.
- networkx: a package that supports both undirected and directed graphs. It provides basic algorithms used in graphs.
- sympy: Symbolic Python. I am not good at this package, but I know mathics and SageMath are both based on it.
- pandas: it supports data frame handling like R. (I have not used this package as I am a heavy R user.)
Of course, if you are a numerical developer, to save you a good life, install Anaconda.
There are some other packages that are useful, such as PyCluster (clustering), xlrd (Excel files read/write), PyGame (writing games)… But since I have not used them, I would rather mention it in this last paragraph, not to endorse but avoid devaluing it.
Don’t forget to type in your IPython Notebook:
- Diane Mueller, Must-Have Python Packages in Finance
- Python Scientific Lecture Notes
- Natural Language Processing in Python
- Radim Rehurek, Practical Data Science in Python