I wouldn't say that it sucks - that's way too strong. I am one of the authors of Circuitscape (http://www.circuitscape.org), which uses python+numpy+scipy. The community loves it, the fact that its python, open source, embeddable in other tools, etc.
However, much of the code is written in a vectorized style for performance reasons, as is the case with many high level scientific computing languages. This leads to unnatural code in some cases, and also uses too much memory. First I thought IronPython was the way, but have been looking forward to pypy+numpy+scipy eagerly.
If I were to use julia, the code would be a lot more natural, because type inference and all the compiler goodies make it possible to simply write loops over arrays when I need them. This was one of the reasons we started working on julia, because everything else just seemed to fall just a little bit short.
OK, perhaps I should have qualified that. It's a good 90% solution for a lot of use cases, although obviously more useful for implementing algorithms in terms of numerical building blocks, than ultra-performance-critical code where you want to write a lot of your own tight inner loops. Is that what you meant?
The parallelisation story isn't great at the moment either, although seems like it has the potential to improve. Still 'sucks' is pretty harsh. For me, looking at how far it's come since I first played with it, I'm impressed. For machine learning, the majority of the building blocks one needs are there, and you get to sit back and put them together using a nice, clean, widely-adopted general purpose programming language. And unlike MATLAB still maintain a decent amount of control over things like memory usage and which BLAS routines it's calling.
Adding bindings for new libraries is more of a pain than it should be though on the occasions where you do really need some fortran or C++ library that doesn't have bindings yet. A language which bridges the gap between high and low levels (not C++!) and has great interop would be very interesting. I guess I'm just hopeful that this kind of thing can be achieved in a general-purpose language (new or existing) which like Python is adopted across the wider software engineering community. Perhaps that's my unreasonable demand to add to their list :)
Seriously? I mean, I know you are very excited about your work on PyPy, but why do you have to go around saying baseless stuff like this? There are many, many thousands of people who write lots of Python code every day that rely on Numpy, Scipy, and Matplotlib to get their work done, and who seem to be quite pleased with it. There is a lot of work left to do on all parts of that stack, but that's a far cry from "it sucks".
I guess I should clarify that - performance sucks. There are obviously various ways around it, but you just can't write a lot of performance critical python that way and I guess this is one of the reasons why julia exists in the first place.
But if you look at the actual cutting edge research work in the HPC and scientific space, they are working on languages that allow domain experts to express computation using higher level primitives, not on fancy compiler techniques to make general purpose imperative languages like C or Python "automagically" run faster on single cores.
The general consensus, if you look at languages like Chapel and Fortress and X10, seems to be that most scientific codes shouldn't be written using for-loops. That is the low-level control flow construct dating from the age of assembler. Instead, what scientists generally want to say is, "Apply this kernel across this domain, with these windowing conditions", or "Reduce values from this computation along these keys in my dataset". As software developers, our job is to provide the language runtime to allow them to do that; such a runtime will be the most robust, correct, maintainable, and performant.
Right. I think we actually violently agree with each other on that :) Note that in general it's ok to not write much python and live very happily just using it for non-performance critical parts. No doubt we both know a lot of people who are quite happy with that.
The problem with numpy's performance is twofold:
* Numpy expressions might not be fast enough. I believe you guys at continuum are trying to address that one way or another. In general the kernel expressed using high-level constructs in python should not be slower than an equivalent loop in C.
* Sometimes you actually want to write a for loop, because you don't care, because it's faster, because it's a single run, because the data is manageable etc. You should not be punished for doing that with 100x performance drop. You can still be punished for that with 2x performance drop.
