Sunday, December 30, 2007

More about Crunchy running under Python3.0a1

Just in time for the new year, and after a major reorganization of Crunchy's code to make it easier to write more unit tests for it, I did manage to get all of Crunchy's core functionality working under Python 3.0a1. Crunchy can now import html files that are not conforming to strict xml convention (i.e. not having closing tags for some elements) and thus create problems for ElementTree. The way it does that is to launch an external process under Python 2.5 (or whatever default version that is invoked via typing "python" at a terminal - but it has to be a 2.x version for it to work). This process imports the file, "cleans" it (using a combination of BeautifulSoup, ElementTree and Crunchy's security module), and saves it locally. After a delay, Crunchy loads up the cleaned up file and display it. In this process, most of the styling is unfortunately removed. :-(

However, the good news is that I was able to load up the official 3.0 Python tutorial (work in progress) and try it out using Crunchy. I did find out one limitation of using Crunchy to do so. Crunchy encodes the Python output using utf-8 before forwarding to the browser. So, instead of having things like b'Andr\xc3\xa9' appearing on the screen, it would be converted to André. Thus, Crunchy is not a very good platform to teach about encoding/decoding of strings. For other aspects though, it is an ideal tool (if I may say so) for going through the Python tutorial: there is no need to switch back and forth between the browser and a separate Python environment to try things out. I still hope to have the time and energy to go through the entire 3.0 tutorial (something I have never done before, for 2.x) and see if I can find any bugs or come up with useful suggestions.

All I have left to do for the next release is to write up some documentation/tutorial on the new Turtle module and on launching Crunchy under Python3.0a1. With a bit of luck, this will all be finished before the end of the year.

In addition to the code reorganization mentioned above, I did fix a few bugs and made an improvement on Crunchy's Borg interpreters. For those that aren't familiar with it, Crunchy allows to embed a number of interpreters (html input box communicating with the Python backend) within a single page. These interpreters can either be isolated one from another (meaning that a variable defined in one interpreter is only known by that interpreter) or can share a common environment (aka Borg interpreters). Normally, in a single user mode, using a single open tab in Firefox, every time a new page is displayed, the Borg interpreters are effectively reset (the old ones are garbage collected and the new ones are created from an empty slate).

Previously, if one were to have multiple users (or multiple tabs open from the same user) on the same Crunchy server, all Borg interpreters ended up sharing the exact same state. This is not very convenient if two users are trying to use the same variable names! I had planned to address this "feature" at some point after the 1.0 release but was forced into it earlier due to the 3.0a1 work. The reason is the following.

To create "Borg interpreters", I was using the Borg idiom invented by Alex Martelli. It goes as follows:

class Borg(object):
'''Borg Idiom, from the Python Cookbook, 2nd Edition, p:273

Derive a class form this; all instances of that class will share the
same state, provided that they don't override __new__; otherwise,
remember to use Borg.__new__ within the overriden class.
'''
_shared_state = {}
def __new__(cls, *a, **k):
obj = object.__new__(cls, *a, **k)
obj.__dict__ = cls._shared_state
return obj

When using this idiom in a standard program under 3.0a1/2, a deprecation warning is raised about object.__new__() not taking any argument. When I was trying to make use of this idiom in Crunchy running under 3.0a1/2, the deprecation warning was actually replaced by an exception. Rather than trying to silence this exception, I decided to take a different approach and used instead the following:

class BorgGroups(object):
'''Inspired by the Borg Idiom, from the Python Cookbook, 2nd Edition, p:273
to deal with multiple Borg groups (one per crunchy page)
while being compatible with Python 3.0a1/2.
Derived class must use a super() call to work with this properly.
'''
_shared_states = {}
def __init__(self, group="Borg"):
if group not in self._shared_states:
self._shared_states[group] = {}
self.__dict__ = self._shared_states[group]

# The following BorgConsole class is defined such that all instances
# of an interpreter on a same html page share the same environment.

class BorgConsole(BorgGroups, SingleConsole):
'''Every BorgConsole share a common state'''
def __init__(self, locals={}, filename="Crunchy console", group="Borg"):
super(BorgConsole, self).__init__(group=group)
SingleConsole.__init__(self, locals, filename=filename)
The "group" variable is taken to be a unique id generated by Crunchy each time it processes a given html page. Thus, each page loaded by a different user (or the same user, at a different time) from the same Crunchy server will result in a unique set of Borg interpreters.

To be fair, I must admit that I did not come up with this solution totally on my own. A while ago, I asked for something like this on comp.lang.python. (Those interested in the details should search for "Borg rebellion".) I just derived the above solution from some suggestions made at that time.

Finally, in addition to all this, I found out a bug in code.py for Python 3.0a1/2. I tried to send an email to the python-3000 mailing list about it, but it was held up, waiting for a moderator approval for a few days. So, I canceled it and filed a bug report instead (which I should have done in the first place) on the bug tracker. I still haven't seen any follow up - perhaps due to the title I gave it. The bug is actually very easy to fix - three lines of code need to be replaced by a single one. The solution is related to my only other "official" contribution to Python to date. Hopefully, by this time next year, I'll have learned enough to contribute more to Python.

Thursday, December 27, 2007

Crunchy and Python 3.0a2

Continuing with my experiment of adapting Crunchy to Python 3.0, I managed to get Crunchy to start with Python 3.0a2 and get some code running from the editor - but not from the interpreter, nor the doctest. Most of the problems I have are dealing with bytes-to-string conversion and string-to-bytes. As mentioned by Guido van Rossum last June
  • We're switching to a model known from Java: (immutable) text strings are Unicode, and binary data is represented by a separate mutable "bytes" data type. In addition, the parser will be more Unicode-friendly: the default source encoding will be UTF-8, and non-ASCII letters can be used in identifiers
Later on, in a comment from that post, we find:
  • > In your presentation last night you had one slide which
    > talked about the "str" vs "bytes" types in Python 3000. On
    > the bottom of that slide was something like:
    >
    > str(b"asdf") == "b'asdf'"
    >
    > However, in discussing this slide (very briefly) you said
    > that a type constructors like "str" could be used to do
    > conversion. It seems like "str" is behaving more like
    > "repr" in this case, which seems unusual and less useful
    > to me. Was this a typo, or is this actually the way it's
    > supposed to work? What's the rationale?

    To be honest, this is an open issue. The slide was wrong compared to the current implementation; but the implementation currently defaults to utf8 (so str(b'a') == 'a'), which is not right either. The problem is that there are conflicting requirements: str() of any object should ideally always return something, but we don't want str() to assume a specific default encoding.

    To be continued...
This change seems innocuous enough...

As a web server, Crunchy sends to and receives information from the browser as "binary data" or "bytes". As a generalized Python interpreter, Crunchy manipulates the information as "strings". It appears that the "bytes" implementation is done much more completely in Python 3.0a2 than it was in Python 3.0a1. And this is the source of many problems.

For example, Crunchy sends from the browser some information about the path to which a Python file should be saved and its content as follows:

'/Users/andre/.crunchy/temp.py_::EOF::_from Tkinter import *\nroot = Tk()\nw = Label(root, text="Crunchy!")\nw.pack()\nroot.mainloop()'

This is sent as a binary stream which needs to be converted to the string written above. This conversion is done via str(...). Using Python 3.0a1 (and 2.4 and 2.5), the result was as above; splitting the string gave the following:

['/Users/andre/.crunchy/temp.py', 'from Tkinter import *\nroot = Tk()\nw = Label(root, text="Crunchy!")\nw.pack()\nroot.mainloop()']

Now, with Python 3.0a2, it gets slightly more complicated. The first string acquires a "b" prefix upon conversion (as mentioned in the comment from Guido's blog mentioned before). After splitting, the result is

["b'/Users/andre/.crunchy/temp.py", 'from Tkinter import *\\nroot = Tk()\\nw = Label(root, text="Crunchy!")\\nw.pack()\\nroot.mainloop()\'']

So, we now have a first string with a "b'" prefix embedded in it, and a second one without. It seems that each case will have to be handled carefully on its own. And I suspect more problems will show up as we get closer to the final 3.0 release.

I know, I know, I'm really not following the "recommended" practice, as quoted on Guido's blog. I should probably wait first for Python 2.6 to come out. Then, I should have a complete unit test coverage and use the conversion tool to create a Python 3.0 version .... However, I am not convinced that the conversion tool will be smart enough to know when a function (that I write) expect a "str" object and when it expect a "byte" one. Furthermore, the few unit tests I had worked fine under both Python 2.5 and 3.0 ... but some functions that I had written with the expectation that they would receive some string arguments did not work in "production code", as they were getting some bytes arguments. And this failed completely silently...

If I had to give some advice to someone about creating Python programs that can work with both Python 2.x and Python 3.x, I would say like Guido: don't. :-) Unless of course you are like me and are doing this for fun and to get to learn about the differences between Python 2.x and 3.x along the way. But then, "be prepared for the unexpected" like the following: turning on a few print statements (via a "debug flag") can result in breaking some code; turn them off and the code works again... Yes, it did happen to me - I still have to figure out how...

Crunchy and Python 3.0a1

crunchy running under Python 3.0a1

It is often said that a picture is worth a thousand words...

I have managed to make Crunchy run under Python 3.0a1. Some of the features are not working but the interpreter and the editor work. The turtle module I have been working on also works "nicely" (read: as slow as before) with this new Python version. Unfortunately, when it is run under Python 3.0a1, Crunchy can not load most pages - including those of the official Python 3.0 tutorial. The reason is that is uses ElementTree to parse pages and it is unforgiving when it comes to having unclosed tags (as in <link> and <meta...> for example); it also seems to not be able to handle the <script>s that are included on the page. I have not yet found a way to reliably "clean" the pages before parsing them with ElementTree. While I believe that I should be able to do so with a bit more work, there is a bigger problem...

Unfortunately, Crunchy does not run under Python 3.oa2, and the error messages I get have not been too helpful in figuring out the error. However, perhaps this is due to a faulty installation. What makes me think so is that when I start a 3.0a2 session at a terminal, I get an error message when I use exit(). This is most unexpected.

In any even, the next release should include the new crunchy turtle module and be usable with 3.0a1. Perhaps Johannes, or some curious user, will be able to figure out how to make it run under 3.0a2 as well.

Tuesday, December 25, 2007

Slow turtle ... in time for Xmas

One of the task assigned in Google's HOP contest was to design a simple turtle graphics module for Crunchy.  This was done successfully by a student as a prototype.  This prototype had some unfortunate limitations in terms of number of turtles and simultaneous graphics canvas existing on the same page, but it did give me the impetus to use the student code as a proof-of-concept and implement a more complete turtle module for Crunchy.

Playing with turtles, and trying to draw fairly complex shapes, made me realize that the combination of using an html canvas and the Crunchy comet communication makes for an extremely slow turtle. It would be really nice to  find a better (faster) way.

The next Crunchy release should include that turtle module ... and an additional bonus: Crunchy can now be launched successfully using either Python 2.5 (or 2.4) and Python 3.0a1.  And the turtle module works with both.

At the moment, not all of Crunchy's features are supported when using Python 3.0.  However, this should no longer be the case by the time version 1.0 comes out.

And, for those that might be tempted to point out Guido's blog entry about not making programs compatible with both Python 2.x and 3.x, please don't bother.  I realize that it is not wise in general to try to do so.  However, given Crunchy's design philosophy to make it as easy for students/teachers/tutorial writers to use - it just does make sense: download, unzip, double-click; nothing else should be needed to start having fun with Python - no matter what new Python version gets installed.


Tuesday, December 18, 2007

(NOT) Bitten by PEP 3113

UPDATE: The comments left on this post (1 and 3) in particular corrected my misreading of PEP 3113. There is no such wart as I describe in Python 3.0. I should have known better than to question GvR and friends. :-) I'm leaving this post as a reference.

In trying to make Crunchy useful & interesting for beginning programmers to learn Python, I designed a small graphics library following some "natural" notation. As an aside, Johannes Woolard is the one who made sure that this library could be easily used interactively within Crunchy. I mention his name since too many people seem to assume that I am the only one involved in Crunchy's design. Anyway, back to the library...

In that library, the function used to draw a line between two points uses the syntax

line((x1, y1), (x2, y2))

for example: line((100, 100), (200, 200))


which should be familiar to everyone. Unfortunately, following the implementation of PEP 3113 in Python 3.0, this syntax is no longer allowed. This is ... annoying! There are two alternatives I can use:

line(x1, y1, x2, y2)

for example: line(100, 100, 200, 200)


or

line(point_1, point_2)

where point_a = (x_a, y_a). Update: with this second definition, it will be possible to invoke the function as
line((100, 100), (200, 200))

Of course, either of these two option is easy to implement (and is going to be backward compatible with Python 2k). However, I don't find either one of them particularly clear for beginners (who might be familiar with the normal mathematical notation) and do not consider this a (small) wart of Python 3k.

reStructuredText files and Crunchy

Crunchy can now handle reStructuredText (.rst) files in the same way it can process plain html ones! This requires the user to have docutils installed - which is normally the case for anyone that writes .rst files.

The test coverage for Crunchy is slowly improving. Currently, 10 modules are mostly covered by doctest-based unit tests, out of approximately 40. Since I make use of .rst files to keep the unit tests, these can now be browsed "pleasantly" using Crunchy itself.

Furthermore ... all the unit tests written so far work under Python 2.4, Python 2.5, and ... Python 3.0a1! This required some tedious rewriting of some parts of the code but the end result is well worth it - if only to really learn about differences between Python 2.5 and Python 3.0.

One thing that I found, which will be no surprise to TDD aficionados, is that code written without testing in mind can be quite tricky to write comprehensive tests for. Add to this the extra complication of making that code run under two incompatible Python versions, and you are on your way to major headaches. It's a good thing I am doing this only for fun!

Saturday, December 08, 2007

Launching Python 3.0 program from Crunchy running under Python 2.5

As part of Google's Highly Open Participation contest, Michele Mazzoni completed the task of creating a new option for Crunchy: one can now launch (starting with the next release of Crunchy - 0.9.8.5) a program using a different version of Python than the one used by Crunchy itself. While I had suggested that the alternate Python version could be set via the configuration options for Crunchy (usually accessible from a Python interpreter), Michele had the brilliant idea to add a simple input box where one can specify the path (or 'alias') of the Python version used right on the page where the program is launched from. This makes it extremely easy to change the interpreter version used to launch a user written program.

Michele has prepared a screencast demonstrating this, which should appear on ShowMeDo hopefully soon.

Thank you Michele - and thank you Google!

Tuesday, December 04, 2007

More results from GHOP

Google's Highly Open Participation (GHOP) contest is attracting a lot of attention from the right people: pre-university students. The PSF is one of ten organizations mentoring students working on Python-related projects. Since I submitted tasks suggestions early on and volunteered to help following a call for volunteers from Titus Brown, Crunchy has benefited from many students contributions. Crunchy's messages have been translated in Estonian, Macedonian, Polish and Italian with, hopefully, more translations to come. Some new unit tests have been added with more to come. There may be a couple of nice surprises coming out soon too :-)

While other projects have also benefited from GHOP's students contributions, there could be more. If you have some good ideas for mini-projects (doable in 3-5 days, at a couple of hours per day with perhaps one full day), your suggestions would most likely be most welcome. Just check out the GHOP Python discussion group. And, if you would like to join the (too small) ranks of Python mentors, please do; we need all the help we can get.

Friday, November 30, 2007

First Crunchy screencast

I finally got around to produce a first Crunchy screencast. Since Crunchy is an interactive program, I thought I should do screencasts that reproduced the interactive feel: that is to say, I would record a live, more or less improvised session. This means that the screen cast is not as polished as it could be - an even has a few minor mistakes in it. Still, I think it gives a good (superficial) overview of what Crunchy is capable of.

What do you think?

Wednesday, November 28, 2007

First result from GHOP: Crunchy Estonian

By now everyone has probably heard of Google Highly Open Participation contest, and the participation of the PSF. And, initially through serendipity but later with more active involvement, a few Crunchy-related tasks were created including some involving translations. And, just a day after the contest started, a first translation (in Estonian!) has been made for Crunchy by a student named Tanel.

The tasks assigned are usually not very big but they provide nice stepping stones for getting people involved in open source software. Thank you Google for this wonderful idea.

Tuesday, November 27, 2007

A failed experiment

On a whim, I decided to try a fun experiment: have Crunchy run correctly when invoked either from Python 2.4 or Python 2.5 ... or from Python 3.0! I thought that if I could make it to work, it would be a great tool to go through the 3.0 docs and tutorial, and finding out if there were any mistakes. (I was planning to do this, hoping to help the developing team in my own limited way.)

In the end, I had to give up, as Crunchy uses third-party modules (ElementTree, HTMLTreeBuilder, ElementSoup, BeautifulSoup, etc.) that I could not make compatible with both Python 2.x and 3.0 without essentially rewriting them in two separate modules each. Actually, this was already done for ElementTree (included in the stdlib under a different name) but not for the others.

The main stumbling block was string handling/encoding... This should not be a surprise to anyone who has followed the Py3k development.

Still, I've learned a fair bit from that experiment. One thing that I sort of knew already related to using print statements for debugging purpose. I often sprinkle my code with "if DEBUG: print ...", having multiple print statements appearing. Often I find that I want to have a few debug flags in the same file and think to myself

Shouldn't I replace these print statements by a debug() function with a variable setting the debug level ...

If I would have done something like this, I could have had just a few print statements located in a debug.py module. As it was, with the change in Py3k for print becoming a function, I had to do a lot of changes by hand. Yes, it might have been possible to use the automated 2to3 tool ... but I wanted to maintain compatibility with Python 2.x and, more importantly, get a feel for what was involved in making the code Py3k compatible.

If or when BeautifulSoup, ElementSoup and HTMLBuilder become Py3k compatible, I'll have to give it a try again...

Tuesday, October 16, 2007

Choosing a CMS ... Will Python make it

This post is not related to programming per se; feel free to skip it ...

"We" are in the process of revamping our web site, and use a CMS. We have just signed off on the first part of the project (new look and wireframe) and are looking at the second (and "final") stage. As we are too small to afford a commercial solution, we are looking at open source CMS. So far, the preferred choice of our director of technology is Alfresco. The choice recommended by our usual technology provider is Joomla! My favourite would likely be Plone - given it is written in Python ;-)

Normally, we would proceed with a public tender - but given that there are few local providers, we may just proceed with the same firm that provided services in phase one. However ... If some of you reading this have work experience in this area, and would be interested in such a project, have a look at our current web site, and give me an informal time/cost estimate of migrating this website to use a [Python based? ;-)] CMS. If we do proceed to tender, you would most likely be included in our list of firms contacted to submit an offer.

Friday, August 31, 2007

pyglet and pygame: which one?

Pygame is great. Pete Shinners has done some fantastic work in creating it. I have played a bit with it in the past and was planning to learn to use it better, and probably write some Crunchy-based tutorials for it. Unfortunately, this will not happen. Since I moved from a PC to a Mac, and installed Python 2.5, I can not use Pygame, as there is no ready-made version for my setup. Yes, I imagine I could find out how to compile it from the source ... but there is an easier alternative: pyglet. pyglet is a pure python library, with no external dependency. I downloaded it, tried it ... and it just worked. Not only that, but when Python 2.6 will come out, I know it will still work.

I do realise that pyglet is just in alpha stage ... but it's a very impressive alpha. It is not as fully featured as Pygame is ... but it works for me, thanks to Alex Holkner.

If you're currently using pygame, there's probably not any reason to switch. But if, like me, you find that you can not use pygame with your current setup, check out pyglet. You might be pleasantly surprised.

Python 3.0a1 has been released!

Congratulations to all Python developers on the release of Python 3.0a1!

My favourite language is on its way to get even better!

Tuesday, August 07, 2007

Crunchy-like sites for javascript and ruby

While I have been aware of the Try Ruby site for quite a while, recently I came across this interactive javascript tutorial. While none of these are nearly as flexible as Crunchy, they are nice resources that are currently available for anyone to try without needing to install anything. It would be nice if I could have a dedicated server running Crunchy in a sandbox so that people could try it out without having to download it.

flat is better than nested?

In the process of making Crunchy more secure, I had extra attention to css files that contained import statements, since these can be used to insert javascript code. I was rather surprised when I looked at www.python.org. First, I noticed the line
  <link rel="stylesheet" type="text/css" media="screen"
id="screen-switcher-stylesheet" href="/styles/screen-switcher-default.css" />
which, in itself, appears rather ordinary. Upon following the link, I found that the content of the css file consists of the single line:
@import url(../styles/styles.css);

In its default security mode, Crunchy just ignores any style information that contains "url(" in it. As a result, when viewed using the default security mode for Crunchy, the python.org site's formatting is lost.

I wonder why the link redirection is used...

Saturday, July 28, 2007

Crunchy Security Advisory

A security hole has been uncovered in Crunchy (version 0.9.1.1 and earlier).

Anyone using Crunchy to browse web tutorials should only visit sites that are trustworthy.

We are working hard at fixing the hole; a new release addressing the problems that have been found should be forthcoming shortly.

-----

The security problem is as follows:

In theory, a web page could contain some javascript code (or link to such code) that would bypass Crunchy's filter to be executed by the browser. If that is the case, the javascript code could be designed to send some Python code directly to the Python backend (i.e. without the Crunchy user pressing a button, or having the chance to view the code to be executed) so that it is executed. Such code could result in deleting the entire files or installing some virus on the user's machine.

At the moment, the risk is pretty low. Crunchy already removes all obvious (and most non-obvious) javascript code, links to such code, etc. The holes found require the use of some uncommon combination of html and css code, with a particular knowledge of Firefox.

(Note that browsers other than Firefox are likely to be even more vulnerable).

Furthermore, Crunchy is not that well known that it is likely to be a target by a cracker that would 1) write a "tutorial" interesting enough to lure current Crunchy users (who, at this point, are likely to include only advanced Python users) and 2) write some fairly involved javascript code to bypass the second security layer (where the commands enabling communication between the browser and crunchy are made up of random string generated uniquely at each new Crunchy session).

If anyone is interested in security issues related to Crunchy, feel free to contact me directly.

Tuesday, July 10, 2007

Rur-ple 1.0rc2 : bug fix for wxPython 2.8

Three changes; a grand total of 3 lines of code. That's all I needed to change to make rur-ple work with wxPython 2.8 (and probably 2.7). It only took me 8 months to get around to fix it due to a combination of lack of time and, when I had time, my obsession with Crunchy.

Speaking of Crunchy, I found a bug today: any Python output that has something like
< ..... >
in it will be such that the stuff between angle brackets will not be shown. This is because the browser thinks it is a valid tag. The first time I noticed it was when I tried to do help(cmp). Then, I was going through "how to think like a computer scientist" (before I point my son to it) and noticed that the exercise with type(...) did not work. So, it is time to do a bug fix release - but no official announcement of this minor release. I'm curious to see if I'll hear from someone about this bug prior to the next "major" release.

Monday, July 09, 2007

New Crunchy (0.9) is out

By now, the official announcement should have made it to many lists. After a mad rush of coding and writing documentation over the past week or so, and implementing lots of new cool features, the new Crunchy is officially taking over from the old one.

I believe that this new version can do much to promote Python usage. If you do agree, please mention it!

Here's a copy of the announcement I sent to various lists:

Crunchy 0.9 has been released. It is available at http://code.google.com/p/crunchy

What is Crunchy?

Crunchy is a an application that transforms html Python tutorials into interactive session viewed within a browser. We are not aware of any other application (in any language) similar to Crunchy. Currently Crunchy has only been fully tested with Firefox; we know that some browsers simply don't work with it. Crunchy should work with all operating systems - it has been tested fairly extensively on Linux, Windows and Mac OS.

What is new in this release?

Crunchy has been rewritten from scratch from the previous version (0.8.2), to use a custom plugin architecture. This makes easier to extend and add new functionality. Rather than list the differences with the old release, it is easier to list the essential features of this new version.

1. Crunchy can work best with specially marked-up html tutorials. However, it can now work with any html tutorials - including the official Python tutorial on the python.org site. Html pages can be loaded locally or remotely from anywhere on the Internet. Crunchy uses a combination of Elementtree and BeautifulSoup to process html pages. Non W3C-compliant pages can be handled, but the visual appearance is not guaranteed to reproduce that normally seen using a browser.

2. Crunchy can insert a number of Python interpreters inside a web page. In the default mode, it does that whenever it encounters an html [pre] element which is assumed to contain some Python code. These interpreters can either share a common environment (e.g. modules imported in one of them are known in the other) or be isolated one from another.

3. Crunchy adds automatic css styling to the Python code - you can look at the official Python tutorial using your browser (all Python code in blue) and compare with what Crunchy displays to give you a good idea.

4. Instead of inserting an interpreter, Crunchy can insert a code editor that can be used to modify the Python code on the page and execute it. The editor can be toggled to become a fairly decent syntax aware editor that can save and load files.

5. Crunchy has a "doctest" feature where the code inside the [pre] is taken to be the result of an interpreter session and the user has to write the code so as to make the interpreter session valid; this is useful in a teaching environment. Messages from the Crunchy's doctest are "friendlier" for Python beginners than the usual tracebacks.

6. Crunchy has a small graphics library that can be imported, either inside an editor or an interpreter, to produce simple graphics (even animations!) inside the browser.

7. For the user that needs better quality graphics, Crunchy supports programs (such as matplotlib) that can create image files; by executing the code, the image produced is loaded inside the browser window. In this capacity, Crunchy could be used as a front end for libraries such as matplotlib.

8. Crunchy supports code execution of files as separate processes, making it suitable to launch gui based application from the browser window.

9. Crunchy's interpreter has an interactive "help" feature like many python-aware IDEs.

10. Crunchy includes a fairly comprehensive tutorial on its own use, as well as a reference for tutorial writers that want to make their tutorials "crunchy-friendlier".

11. As a security feature, crunchy strips all pre-existing javascript code from an html page before displaying it inside the browser window.

Bug reports, comments and suggestions are always welcome.

André Roberge, for the Crunchy team.

Sunday, June 24, 2007

Fun with the new Crunchy

Since the new Crunchy can work with files that have no Crunchy-specific additional markup, some neat experiments are possible, in addition to playing with the official Python tutorial with a Python interpreter embedded on the web page that I mentioned in my last post. For the curious among you, here's something else to try.

Assuming you have the latest (version 0.2) alpha release of the new Crunchy, edit the file vlam.py, replacing "interpreter" by "editor" at lines 108 and 109. Then, launch Crunchy and click on the "tests" link. On the following page, click on the "Loading arbitrary tutorials" link. Then, in the box for loading remote tutorials, enter the address of the Python cookbook (http://aspn.activestate.com/ASPN/Python/Cookbook/).
The formatting will be off, but you can select your favorite recipe. When you do this, you can edit the code and execute it right on the page. Often, all you need to do is to replace __name__ == "__main__" by True and you are ready to try the examples, modifying them at will.

In the near future, Crunchy will provide an easy way to select which interactive element (Python interpreter or editor) is to be inserted by default, without having to edit the code by hand.

From vlam to nam: Using Crunchy to interact with the official Python tutorial

Breaking news: you can now use Crunchy to browse and interact with the official Python tutorial. More below...

Since my last post that described briefly some features of the new Crunchy engine, I have not had much time to work on Crunchy. Johannes too has been fairly busy so that Crunchy development pretty much stopped, until the beginning of Google Summer of Code. Two students joined the Crunchy team: Edin Salković and Bryan Psimas. Unfortunately, due to other commitments, Edin was not able to continue with his SoC project - but he did manage to write the prototype of a new plugin for Crunchy that I will describe below.

The new Crunchy engine now works extremely well. As I wrote in my previous post, Crunchy is now more responsive; it supports simultaneous execution of multiple code sample, proper input (stdin) handling from the webpage browser and the api (for writing tutorials) has been significantly simplified as there is no longer be any need to have an embedded "canvas" call in a tutorial: the user is now able to load a graphics dynamically anywhere on a page. These graphics can even include simple animations using pure Python code (for the end user - javascript behind the scene for us, unfortunately).

There is more. For example:
  • it is now possible to style <code> elements;
  • Python code sample (either inside <pre> or <code>) that contain pre-existing html markup can now be handled by Crunchy without generating an error (blank page!) as before;
  • it is now possible to specify a starting number different from 1 when requesting that line numbers be added to a code sample;
  • when it styles the code, Crunchy now automatically detects if a given code sample represents an interpreter session or a "normal" code sample;
  • it is now possible to use input() and raw_input() with an interpreter, and the result appears inside the page displayed by the browser.
In addition, thanks to Edin's work, a new type of interaction with Python code has been added: it is possible to save the result of a computation in a graphics file and display the result in the browser. This has been used successfully with matplotlib. There is a "small" bug in that the Python code needs to be executed twice due to synchronization issues with the browser loading the file and Python saving it.

Quite a few features from the "old Crunchy" need to be implemented (e.g. menus including a way to properly quit the application, translations, update:the ability to load an arbitrary tutorials given a URL, etc.) in the new version but, overall, it is working very well.

One of the "problems" with the old Crunchy is that it did require two things from a tutorial writer:
  1. That the html code written be W3C compliant (with no warning, no unclosed tags, etc.).
  2. That Crunchy specific markup (vlam = very little additional markup) be added to instruct Crunchy to add interactive elements.
Unfortunately, the official Python tutorial fails on both accounts. However, with the use of a version of BeautifulSoup adapted by effbot (thanks!) to be used with ElementTree, the new Crunchy can now deal with non W3C compliant code.

And ... Crunchy has now been written so that it automatically inserts a Python interpreter whenever it sees a bare <pre> (i.e. with nam = no additional markup). As a result, one can now browse the official Python tutorial with Crunchy and interact with it.

This automatic insertion of an interpreter sometimes yields too many interpreter inserted on a page than really needed; it is better to hand craft a tutorial. However, it should make Crunchy a lot more useful to many more people.

For those interested in trying out the "new Crunchy", an alpha release is available. If you have never used Crunchy before, you should try version 0.8.2 first and go through the tutorial from the menu.

Tuesday, March 06, 2007

New Crunchy engine

At the end of my talk at Pycon 2007, I gave an extremely brief demo of the new (not in production yet) upcoming version of Crunchy. Thanks to the work of Johannes Woolard, Crunchy's javascript core engine has been completely rewritten. As a result, Crunchy will be more responsive; it will also support simultaneous execution of multiple code sample, proper input (stdin) handling from the webpage browser and the api (for writing tutorials) will be greatly simplified as there will no longer be any need to have an embedded "canvas" call in a tutorial: the user will be able to load a graphics dynamically anywhere on a page.

But that is not all...

Inspired by a side remark by Ivan Krstić (of OLPC fame) about finding the best way to design plugins in Python, I contacted Johannes to discuss the idea of using plugins to extend Crunchy. After a few back and forth emails, while I was on the road on business trips, Johannes just went ahead and implemented a first (then a second...) way of using plugins to extend Crunchy. In the end, it looks like we'll have to rewrite almost completely (like we did during the Summer of Code 2006) Crunchy's code as we move from one minor version (0.8) to the next (0.9). However, the result will be well worth it.

The unstated goal: to make it (almost) as easy for developpers to add new capabilities to Crunchy as it is for tutorial writer to use Crunchy to create interactive tutorials. Ok, this might be a slight exaggeration ... but not much of one ;-) And most of this will be the result of Johannes' great work. So, for all of those employers (and there were many at Pycon) looking for a brilliant Python programmer to hire over the summer, I know of an Oxford student who certainly fits the bill.

Monday, February 26, 2007

Pycon experience

Pycon 2007 is now officially over. It was very nice to be able to associate faces to so many familiar names from the Python community and get the opportunity to talk with a few of them.

My talk didn't go as smoothly as I had hoped. When I practiced it, it seemed to always take too long. So, when I had to do it in front of a fairly large crowd, I rushed through it a bit too quickly. Still, Crunchy did seem to generate a fair bit of interest and, in addition to the demo in my talk, I got the opportunity to demonstrate it "one-on-one" to about a dozen people who all seemed to be very keen.

Also, the idea of creating a special Python logo ("web badge") for identifying Crunchy-ready tutorials, which I had alluded to in a previous post, will likely be given the ok by the PSF. As soon as it is confirmed, I'll make an announcement here.

Since my last post, the PEP I wrote for keeping raw_input()'s functionality in Python 3000 [but renaming it "input()"] had been accepted. On Monday, I took part briefly in a Python 3000 sprint where I got to implement the required "raw_input(...) -> input(...)" and "input(...) -> eval(input(...))" conversion tool for the general tool that will be available to convert Python 2.x programs so that they work with Python 3k. I also implemented the corresponding simple unit tests as well a few missing tests for the "xrange() -> range" conversion. I would not have been able to do this if Guido van Rossum had not taken the time to explain a few things to me about the conversion framework. In the end, Guido checked in my changes himself. I get the feeling that he could have done the entire thing in about the same time it took him to explain to me the tricky bits. Nonetheless, I do get the satisfaction of having, with the PEP and the conversion tool, given back a bit to the Python community. Hopefully, with Crunchy, I (and Johannes, of course) will give back even more.

Monday, February 12, 2007

Switched to Mac OS

I'm a statistics... I just got a Macbook and will likely use it as my main development platform soon. So far, moving from Windows XP to Mac OS has been fairly painless. I have not done any Python programming but did try out Crunchy and Rur-ple. I found out that Crunchy requires Python 2.4+ (I had 2.3.5 installed on the Mac), that it does not work with Safari, and that there is a "bug" so that the "tooltip" does not work with Firefox on the Mac. I was planning to bring my Macbook to Pycon but will have to reevaluate as I'd like the Drunchy demo to go as smoothly as possible. There also does not appear to be a sound module on the Mac similar to winsound or ossaudiodev... I also found a new bug with rur-ple, but have no time to look into it before Pycon.

Not having taken side in the emacs/vim religious war, I have to find a decent programming environment. On my PC, I used SPE which I really liked but I have read a lot of good things about textmate and nothing negative (other than the fact that it is not free). Any advice from Mac/Python users?

Monday, February 05, 2007

Rur-ple segmentation fault on Ubuntu: solved?

Some kind readers have left comments on this blog regarding segmentation faults occurring when trying to run Rur-ple using Ubuntu linux. I finally got around to installing a vmware version of ubuntu so that I could check for myself and believe I found the source of the bug. I would appreciate if someone actually using Ubuntu could verify.

Near the bottom of rur_start.py, a splash screen is started with 100 (milliseconds) as a default. Changing that number to 1000 removed the segmentation fault when running rur-ple in a virtual ubuntu environment.

There is at least one other bug in rur-ple when using a version of wxPython greater than 2.6; it is in lightning.py. Changing "event.KeyCode()" to "event.GetKeyCode()" apparently solves the problem.

Of course, I would be interested in hearing about any other bugs that have been encountered so that I could do a final cleanup before finally releasing the 1.0 version.

Wednesday, January 31, 2007

Lightning talk?

Living in a small village in Nova Scotia, 300 km from the nearest city (Halifax, pop.: 380,000), the opportunities to discuss face to face with other pythonistas are non-existent. This is one reason I've been really looking forward to going to my first Pycon. In addition to my scheduled talk, I have been thinking about giving a lightning talk of a different kind, but still touching upon a subject dear to me, that is the promotion of Python and its use as a teaching language. While lightning talks are so short that they normally touch upon only one topic, I was thinking of actually covering three topics, under the common theme of "interactive tutorials". These three topics are, from the more serious to the semi-frivolous:

1. The fate of input() and raw_input() in Python.
2. Using a distinctive logo for promoting interactive tutorials using Python.
3. Creating an interactive tutorial repository.

According to this, both input() and raw_input() are supposed to disappear in Python 3.0. After thinking about it for quite some time, I posted a message on edu-sig about the proposed change last September, expressing my "opposition" to the proposed change, and seeking the opinion of others on this topic. After a while, a near-consensus emerged, supporting the idea of keeping at least something like raw_input() in Python 3.0. I then wrote a pre-PEP, which generated more positive comments and encouragements but otherwise no further action from other edu-sig subscribers more involved in the Python community than I am. Unfortunately, due to my work, I had to drop all my Python-related activities for a few months and was not able to follow up until late December. With the end of the year approaching, I found the time to incorporate the comments received on edu-sig and offered a revised pre-PEP to the Python 3000 list on December 22nd. This was a single posting from me as I don't subscribe to that list. A bit of discussion ensued after which Guido stated that he like the proposal better than any of the other alternatives mentioned and, later, suggested that someone should "clean up and check in the proto-PEP and start working on an implementation or patch."

Unfortunately, and perhaps because this happened just before the holidays, nothing further was done. So, I'm thinking about raising the issue again at Pycon. Having something like raw_input() is, I believe, nearly essential for using interactive tutorials of the kinds produced by crunchy in a teaching environment aimed at beginners.

Further on the topic of interactive tutorials, I have been thinking of an easy means to identify them. For those that have tried crunchy, you will have noticed that an interactive tutorial viewed through a regular browser without using crunchy appears to be nothing else than your average (W3C compliant) html document. Since such tutorials could be included on any web site, I was thinking it might be useful to give a visual clue to crunchy users. There already exist a Python logo to identify applications, namely:
I've been thinking of asking the Python software foundation for the permission to modify this logo, replacing the word "powered" by "interactive", and adding a small change to crunchy so that every tutorial created with it would automatically have the logo added at the top. I'm debating whether I should do this prior to Pycon, or wait until I get the chance to lobby for it in person.

Finally, if enough people start writing interactive tutorials using crunchy, I thought it might be useful to have a central repository somewhere. Moving away from the Monty Python references, I thought that the Snake PIT (for Python Interactive Tutorials) might be a catchy name.

So, for you, patient reader that has made it this far, does it sound like a worthwhile for a Lightning talk, especially if it is supplemented by a one-minute demo of an interactive tutorial? I'm pretty sure I could cover all three topics above in less than the scheduled 5 minutes for a lightning talk, as I would not have to go into much detail about the history; just the facts...

Saturday, January 27, 2007

Crunchy 0.8 is out

Version 0.8 of Crunchy has been released. It is available at its new home on code.google.com.

Crunchy, the Interactive Python Tutorial Maker, is an application that
transforms an ordinary html-based Python tutorial into an interactive
session within a web browser. Currently, only Firefox is supported.
Crunchy is developed and tested on Windows XP and Ubuntu Dapper Drake,
but should work on any suitable windows or UNIX system.

Three major improvements have been made since version 0.7 had been released.

1. New editor

Instead of a simple html textarea, Crunchy now gives the option of
using a "real" editor, namely EditArea. EditArea support syntax coloring
and allows loading and saving local Python files among other features.
Within Crunchy, it is set up so that the tab key is translated into 4
spaces.

2. Language support

Crunchy now supports English and French, through the use of ".po"
files. When running Python code, some error messages have been
adapted/translated. EditArea itself support more languages
(currently: Danish, Dutch, English, French, German, Italian, Japanese,
Polish, Portuguese).

3. Graphical tutorial converter.

Crunchy uses some supplementary markup to transform html files into
interactive tutorials. Whereas previous versions required a tutorial
maker to edit an html file "by hand", version 0.8 includes a tutorial
editor: with a few clicks, you can easily add to an html file the
chosen interactive elements and options for Crunchy.

In addition to the above major improvements, the code has been
refactored significantly and a number of small bug fixes have been
made. Crunchy will be demonstrated at the upcoming Pycon 2007.

The next release will likely have a new, simplified API for tutorial writers, but with more powerful features, thanks to the work of Johannes Woolard. Unfortunately, it is unlikely to be ready in time for Pycon. Anyone planning to go to Pycon, and who is interested in Crunchy should feel free to contact me with any questions/suggestions they may want to have me address during my presentation.

Wednesday, January 10, 2007

Pycon 2007: it's Crunch time - take 2

Well, it's done: I've registered for Pycon, booked a hotel room and bought plane tickets. Since November 1, when I first heard that my talk had been accepted, crunchy has changed quite a bit and my planned presentation will have to change. One demo I had planned to do, using codetch, and which I thought would take about 5 minutes will now be doable in less than 30 seconds without having to use anything else than the "new and improved" crunchy. Unfortunately, there has been so much changed since the last release that the documentation has to be re-organized significantly - this means that I can't really do a new release right now as it would likely be too confusing. So, I'm racing against the clock to put everything together to do a new release before Pycon. However, if some of you are interested in having a preview, drop me an email and I'll let you know where you can get the code from and give a quick description of how to use the new features.
Looking at the official statistics, I find the relatively small numbers of attendees compared with last year somewhat disappointing. Given that more talks are going to be presented this year, with a much larger rejection rate, I find this rather puzzling. I imagine the organizers are rather disappointed.

Wednesday, January 03, 2007

Unicode headaches ... and a solution

Work on Crunchy has restarted by both Johannes and I over the holidays, after a few months long hiatus. Hopefully, we'll be in a position to do a new release soon (just a few more features...). Among the changes, Crunchy now has a proper embedded editor, EditArea. See this example to get an idea of what EditArea can do. Actually, the Python support has since been improved by Christophe Dolivet, the creator of EditArea, prompted by a few suggestions of my own, but the new version has not been made publicly available yet. It is however already used in the development version of Crunchy, and, with a few additions of my own, will definitively be showcased during my Pycon presentation.

Other Crunchy changes include a proper handling of English and French translations. Now, since EditArea's tooltips are available in many more language, I thought I should use a more complete encoding (like utf-8) rather than the one I normally use (latin-1). After all, if rur-ple has been found useful enough to be adapted to 6 languages (with Italian and Chinese in the pipeline), I figured that Crunchy is likely to eventually see the same kind of adaptation.

However, I soon encountered a most puzzling bug. All strings translated were properly rendered by Firefox except for those coming out of an interpretor session (some tracebacks have been customized and translated in French as well as being available in English). When French was selected as the default language, whenever an accented letter, like à, was supposed to appear as a result of a Python output, a ? instead was displayed.

After trying all kind of encoding/decoding tricks over the course of a few hours, I remembered that I had set the default sytem encoding on my computer to latin-1. I decided to change it to utf-8 and, sure enough, everything was working as expected. Success at last!

However, this was only the beginning of my problems. My favourite editor, SPE, stopped working. I also tried to run rur-ple which failed miserably. [Idle, on the other hand, which I use very rarely, was still working perfectly.] Clearly, changing the site default encoding was not an appropriate solution: I could certainly not depend on having a crunchy user set his or her site customization to utf-8. A different approach was needed.

After reverting back to latin-1 as my default python site encoding (so that paths that included my name, André, were properly read by SPE) and poring over the code, I finally figured out a more general solution.

Whenever Crunchy executes some Python code written by a user (or provided by a tutorial writer), it starts a Python interpreter in a separate thread. This Python interpreter uses the default system encoding for all its computation. When the result needs to be sent back by the Crunchy server to Firefox, it needs to have its encoding changed as

result = result.decode(sys.getdefaultencoding()).encode('utf-8')

Of course, in retrospect, it all makes sense but it did stump me for quite a few hours; perhaps the information included here will save a few minutes or hours to someone else.