Jun 28 04:26 PM
corehackers: process work
Several suggestions/requests have
reached me regarding corehackers. First, #corehackers on
irc.perl.org now exists. Second, a task list is very high on my
priority list. The only thing higher is getting basic maintenance
planned and running. I need to get the mindless tasks sorted out so
that I can focus fully on corehackers. I think I've github
maintenance figured out. All going well, that will get much more
simple shortly. Next I need to get a non-github wiki set up
someplace. I know where I'll host it; I just haven't picked the
software. That said, I do need your help getting started. What do
you need to start working? I've heard "C Primer", "Intro To
The Core", and the task list. Send me email or a message on twitter
or identi.ca if you think of anything else.
Jun 27 12:23 PM
corehackers: help improve perl5
Once upon a time, a man named Jon
Orwant threw a mug against a wall and declared Perl to be stagnant,
perhaps dying. Let me quote
Larry Wall here:
Many projects were born out of that sentiment and that realization, including the great and wonderful perl6. In the beginning, we envisioned that perl6 would replace perl5 just as perl5 replaced perl4. Years have past and perl5 is still with us. However, for the most part, excitement in perl5 has faded.
The world has lost perl5 in all the buzz of perl6.
I assert the following:
To quote the best movie of all time, Short Circuit: "[perl5] is alive! No disassemble!" Every day, thousands of programmers write millions of lines of perl5. Every day, hundreds of modules are uploaded to CPAN. Yet every day, many perl5 programmers wonder how they're going to find another perl5 job. (warewolf on irc.perl.org's #perl wishes to take credit for the Short Circuit reference. Since he provided logs, I'm inclined to let him :)
Something needs to be done. We need excitement not just about perl5 as a language but about perl5 as an interpreter. Projects like Moose are helping to drive excitement about perl5 as a language. If you look under the hood, though, Moose has to do some scary shit to work its magic. To do Moose *right*, we need to extend the core.
At YAPC10, Chip Salzenberg announced a project called 'corehackers' designed to excite people who are capable of hacking the perl5 core. There are a lot of experiments to try and a lot of great ideas brewing out there. We need to improve the core, make it more accessible, and take some weight off the pumpkings. I volunteered to help Chip and I started pondering and planning my involvement.
As I talked to non-core people at YAPC, something became very clear to me. There are lots of people in the community who want to help but have little C ability and no experience with the core. p5p has probably never heard of them and they've certainly never seen a patch. These people want to help though. They want perl5 to become better than it is, not just through contributions to CPAN, but by improving the language itself.
How do we get these people involved? How do we rally this pool of excited talent? How do we keep from driving p5p insane with the influx of newbies asking questions and submitting patches that may not be right, may suck mightly and obviously require mentoring? I pondered and pondered and talked and talked. Then it hit me.
We need a project dedicated to the newbies. We need documentation and introductions to the core. We need a mentor program. We need someone else (or several someone elses) to read through and integrate the patches from the newbies. I stood up in front of YAPC and volunteered to be that person. Now I'm standing up in front of my 5 readers, Iron Man, Planet Linode, and whoever else, volunteering to be that person.
Here's how we're going to do it. My project is part of corehackers. Like corehackers, this is not a fork of perl5. corehackers runs parallel to p5p and perl5 core development, providing a service, hopefully a useful one, to p5p. Chip and I will be working with the volunteers to guide their work, integrate their work and submit it to p5p for inclusion. We'll be working on the core too. We'll be putting our money where our mouths are and building a better perl5 from the grass-roots up.
What do we need? We need you. We need you to take a few minutes out of your day to document a piece of the core. We need you to document what isn't documented. We need you to learn the core and fix bugs. We need your ideas and we need your help working on new core features. I'm not looking for a long term, or even medium term, committment. You decide your own level of involvement.
Now, how do you get involved? Go to github and use git to either clone or "fork" our repo. When your change is complete, either issue a pull request or send me a patch directly (sungo@sungo.us). If you want to chat, grab either chip or myself on irc.perl.org. (Warning: #perl is not a help channel. Think of a tavern from an old western film and replace the residents with wifi-enabled hackers. Just query us directly.)
As far as I'm concerned, I plan to commit more than talk. Chances are that you won't see me blogging much more about the philosophy of all of this. I'll dump out status updates and ask for thoughts on ideas. I'm sure there'll be twitter traffic too. Actions > words.
"[Orwant] keeps throwing coffee mugs against the other wall, and he says 'we are fucked unless we can come up with something that will excite the community, because everyone's getting bored and going off and doing other things'".
Many projects were born out of that sentiment and that realization, including the great and wonderful perl6. In the beginning, we envisioned that perl6 would replace perl5 just as perl5 replaced perl4. Years have past and perl5 is still with us. However, for the most part, excitement in perl5 has faded.
The world has lost perl5 in all the buzz of perl6.
I assert the following:
- perl6 will not replace perl5.
- perl5 will live on long after we've all retired thanks to the millions and millions of lines of code already in production. (Making squinty eyes at that? COBOL has a lesson to teach you.)
- Once again, perl5 is risking The Orwant Condition of boredom and irrelevancy.
- In fact, we're already at that point.
To quote the best movie of all time, Short Circuit: "[perl5] is alive! No disassemble!" Every day, thousands of programmers write millions of lines of perl5. Every day, hundreds of modules are uploaded to CPAN. Yet every day, many perl5 programmers wonder how they're going to find another perl5 job. (warewolf on irc.perl.org's #perl wishes to take credit for the Short Circuit reference. Since he provided logs, I'm inclined to let him :)
Something needs to be done. We need excitement not just about perl5 as a language but about perl5 as an interpreter. Projects like Moose are helping to drive excitement about perl5 as a language. If you look under the hood, though, Moose has to do some scary shit to work its magic. To do Moose *right*, we need to extend the core.
At YAPC10, Chip Salzenberg announced a project called 'corehackers' designed to excite people who are capable of hacking the perl5 core. There are a lot of experiments to try and a lot of great ideas brewing out there. We need to improve the core, make it more accessible, and take some weight off the pumpkings. I volunteered to help Chip and I started pondering and planning my involvement.
As I talked to non-core people at YAPC, something became very clear to me. There are lots of people in the community who want to help but have little C ability and no experience with the core. p5p has probably never heard of them and they've certainly never seen a patch. These people want to help though. They want perl5 to become better than it is, not just through contributions to CPAN, but by improving the language itself.
How do we get these people involved? How do we rally this pool of excited talent? How do we keep from driving p5p insane with the influx of newbies asking questions and submitting patches that may not be right, may suck mightly and obviously require mentoring? I pondered and pondered and talked and talked. Then it hit me.
We need a project dedicated to the newbies. We need documentation and introductions to the core. We need a mentor program. We need someone else (or several someone elses) to read through and integrate the patches from the newbies. I stood up in front of YAPC and volunteered to be that person. Now I'm standing up in front of my 5 readers, Iron Man, Planet Linode, and whoever else, volunteering to be that person.
Here's how we're going to do it. My project is part of corehackers. Like corehackers, this is not a fork of perl5. corehackers runs parallel to p5p and perl5 core development, providing a service, hopefully a useful one, to p5p. Chip and I will be working with the volunteers to guide their work, integrate their work and submit it to p5p for inclusion. We'll be working on the core too. We'll be putting our money where our mouths are and building a better perl5 from the grass-roots up.
What do we need? We need you. We need you to take a few minutes out of your day to document a piece of the core. We need you to document what isn't documented. We need you to learn the core and fix bugs. We need your ideas and we need your help working on new core features. I'm not looking for a long term, or even medium term, committment. You decide your own level of involvement.
Now, how do you get involved? Go to github and use git to either clone or "fork" our repo. When your change is complete, either issue a pull request or send me a patch directly (sungo@sungo.us). If you want to chat, grab either chip or myself on irc.perl.org. (Warning: #perl is not a help channel. Think of a tavern from an old western film and replace the residents with wifi-enabled hackers. Just query us directly.)
As far as I'm concerned, I plan to commit more than talk. Chances are that you won't see me blogging much more about the philosophy of all of this. I'll dump out status updates and ask for thoughts on ideas. I'm sure there'll be twitter traffic too. Actions > words.
May 12 05:10 PM
Ode To My Anger With Dropbox
Dropbox is a great idea. If you don't
look behind the curtains, it works really well too. For the
unfamiliar, Dropbox is an application that sits resident on your
machine, syncing files in a magic directory up to the server.
Changes are propagated to any other machine you've linked to your
account. The files live on your drive so if your network connection
drops, you still have your data.
Lately, I've been working on Net::Dropbox, a library to connect to a local dropboxd instance and get information about your files or the daemon's operation. Here's the fun part. The Dropbox folks have released an open source nautlius plugin to talk to the closed source dropboxd. Instead of real documentation, the staff points to this source code, and contributed community code on the wiki, as "documentation of the 'api'".
The "api" (in this case, commands accepted over two local unix sockets) changes in almost every release. The Dropbox team certainly has the right to do that but it makes the aforementioned "documentation" really problematic.
The wiki-provided code can immediately be dismissed. There are 10 or 15 scripts in every language imaginable on the wiki. For the most part, they were written months and months ago. The "api" has changed in ways that makes that code completely irrelevant and, with few exceptions, totally unusable.
As for the nautilus plugin source.... *sigh*
Let me get my "I'm a C programmer" rant out of the way with first. This code is terrible. Forget documentation. How about comments? We have comments like "oh god" and "not thread safe". Great, guys. You don't have much in the way of useful comments but you do have the "low bridge" and "deaf child area" signs up.
The formatting.... Shortest summary: "STOP MIXING WHITE SPACE AND HARD TABS IN INITIAL INDENTING. FIX YOUR EDITOR, DIPSHIT. YOU ARE A PYTHON SHOP. HOW DO YOU NOT UNDERSTAND THIS" *ahem*
Look. This is C. It's a language that allows macros. This code uses them a few times. When it comes to figuring out all the ways that a connection might fail, however, the "let's disconnect and return 'uh, we timed out'" code is duplicated time and time again. In send_command_to_db, I count 4 copy and paste moments. This happens a lot.
These folks fucking adore glib. From my read, some python folks got their hands on gcc and tried to do a literal translation. Everything is a glib hash, no matter how temporary the data. "Find some data. Shove it into a new glib hash. Look at the data. Free the hash." Some routines do this 5 or 6 times. *grumble*
Further, the code is completely tied to gnome/nautilus. When it builds the command block to as for status on a file, it takes a URI from nautilus, converts it, via another magic nautilus comand, to a better URI, utf8s that, and sends it along. Ok. What does the nautilus URI look like? Certainly the gnome folks have documented that. Nope. Their docs simply list that the relevant routine exists. I have to write some goddamn test code, or guess wildly, to figure out what that uri looks like internally. I am Jack's perforated colon.
To summarize: While I've certainly worked with worse code, I was getting paid lots of money to do so. Reading this code makes me want to charge someone a consultancy fee.
Now for my "I'd like to use your code as 'documentation' as you suggested" rant. Good documentation, or the myth of self-documenting code, has one major attribute: clarity. What goes in and what comes out? What commands can I use? "Clarity" is apparently Swahili for "I'd like to play hide and seek in a septic tank." As noted above, I have no love for this C code. The protocol elements for dropboxd are pretty simple text blocks. It took me about 20 minutes to figure that out as a C programmer. I can't imagine how long it would take someone who doesn't know any C. Anyway, generating output is this big "fun" glib hash run-through. Parsing the output is buried in a single function which I missed for a while. It calls out to *another* routine...
At this point I attempted suicide.
It turns out that generating commands and parsing the response is brain-dead easy. The perl code is tiny tiny. I spent about 5 minutes on it. After several hours of digging through C code and drinking myself into a stupor.
Question #2: what commands does the daemon accept? I've found 4 read-able commands and one write command. At this point, I have a routine that lets me make crap up and throw it against the daemon to see what happens. Most of the work I'm doing is figuring out new commands and their arguments.
WHY IS THIS SO DIFFICULT? Seriously, all this work could be avoided by a simple damn text file. It would be maybe 30 lines long.
I do have one thought. If you've made if this far, I have a question. Would you be interested in an open source replacement? At least then I could be angry at *my* crappy code...
Lately, I've been working on Net::Dropbox, a library to connect to a local dropboxd instance and get information about your files or the daemon's operation. Here's the fun part. The Dropbox folks have released an open source nautlius plugin to talk to the closed source dropboxd. Instead of real documentation, the staff points to this source code, and contributed community code on the wiki, as "documentation of the 'api'".
The "api" (in this case, commands accepted over two local unix sockets) changes in almost every release. The Dropbox team certainly has the right to do that but it makes the aforementioned "documentation" really problematic.
The wiki-provided code can immediately be dismissed. There are 10 or 15 scripts in every language imaginable on the wiki. For the most part, they were written months and months ago. The "api" has changed in ways that makes that code completely irrelevant and, with few exceptions, totally unusable.
As for the nautilus plugin source.... *sigh*
Let me get my "I'm a C programmer" rant out of the way with first. This code is terrible. Forget documentation. How about comments? We have comments like "oh god" and "not thread safe". Great, guys. You don't have much in the way of useful comments but you do have the "low bridge" and "deaf child area" signs up.
The formatting.... Shortest summary: "STOP MIXING WHITE SPACE AND HARD TABS IN INITIAL INDENTING. FIX YOUR EDITOR, DIPSHIT. YOU ARE A PYTHON SHOP. HOW DO YOU NOT UNDERSTAND THIS" *ahem*
Look. This is C. It's a language that allows macros. This code uses them a few times. When it comes to figuring out all the ways that a connection might fail, however, the "let's disconnect and return 'uh, we timed out'" code is duplicated time and time again. In send_command_to_db, I count 4 copy and paste moments. This happens a lot.
These folks fucking adore glib. From my read, some python folks got their hands on gcc and tried to do a literal translation. Everything is a glib hash, no matter how temporary the data. "Find some data. Shove it into a new glib hash. Look at the data. Free the hash." Some routines do this 5 or 6 times. *grumble*
Further, the code is completely tied to gnome/nautilus. When it builds the command block to as for status on a file, it takes a URI from nautilus, converts it, via another magic nautilus comand, to a better URI, utf8s that, and sends it along. Ok. What does the nautilus URI look like? Certainly the gnome folks have documented that. Nope. Their docs simply list that the relevant routine exists. I have to write some goddamn test code, or guess wildly, to figure out what that uri looks like internally. I am Jack's perforated colon.
To summarize: While I've certainly worked with worse code, I was getting paid lots of money to do so. Reading this code makes me want to charge someone a consultancy fee.
Now for my "I'd like to use your code as 'documentation' as you suggested" rant. Good documentation, or the myth of self-documenting code, has one major attribute: clarity. What goes in and what comes out? What commands can I use? "Clarity" is apparently Swahili for "I'd like to play hide and seek in a septic tank." As noted above, I have no love for this C code. The protocol elements for dropboxd are pretty simple text blocks. It took me about 20 minutes to figure that out as a C programmer. I can't imagine how long it would take someone who doesn't know any C. Anyway, generating output is this big "fun" glib hash run-through. Parsing the output is buried in a single function which I missed for a while. It calls out to *another* routine...
At this point I attempted suicide.
It turns out that generating commands and parsing the response is brain-dead easy. The perl code is tiny tiny. I spent about 5 minutes on it. After several hours of digging through C code and drinking myself into a stupor.
Question #2: what commands does the daemon accept? I've found 4 read-able commands and one write command. At this point, I have a routine that lets me make crap up and throw it against the daemon to see what happens. Most of the work I'm doing is figuring out new commands and their arguments.
WHY IS THIS SO DIFFICULT? Seriously, all this work could be avoided by a simple damn text file. It would be maybe 30 lines long.
I do have one thought. If you've made if this far, I have a question. Would you be interested in an open source replacement? At least then I could be angry at *my* crappy code...
May 07 12:27 AM
Begin the begin, again and again
Today is the first really productive
day I've had in a long time. As such, I figure maybe I should try
to resurrect my blog. Maybe I'll keep it up this time.
General Life Update: About 5 weeks ago, I got laid off from a job I should have quit eons before, and one that perhaps I shouldn't have taken in the first place. Anyway, my stress level is low. My blood pressure is back to normal and I'm losing weight. Job hunting really sucks and can be depressing as hell sometimes. I've got some decent leads though and things are looking up. However, if you'd like me to work for/with you, don't hesitate to hit that resume link and toss me an email. :)
Code: I have a problem. For the last 18 months, I've done little programming. A bit of perl and C here and there. Some objective-c. For the last 5 weeks, I've opened my editor quite a few times and sat starting at code, unsure of where to start. I have another problem too. I hang out on irc with rjbs, one of the most prolific coders I've ever known. (It's good code too!) He's starting to wear off on me. Better said, I'm very envious of his prolific coding which seems to have spurred me on to action.
A few days ago, he blogged about a deficiency or 12 with the Dropbox standalone daemon for linux. Specifically, dropboxd tells you precisely fuck-all about what it's doing. (The other day, mine kept exiting after a few minutes of running. I had to debug the thing with strace to find out that I needed to upgrade it due to some protocol change.)
"I see a Dropbox client Perl library in my future...", rjbs said. I looked in my src directory and realized that I'd started on one in December. I grabbed rjbs, got his thoughts and behold, Net::Dropbox was born. it is probably more accurate at this point to say that it was conceived. It'll be born when I release the little bastard.
Along the way, I heard nperez mumble about lack of proper documentation in CPAN::Uploader, an rjbs module/app I use often. So... I patched it. In that process, I found a bug in Getopt::Long::Descriptive. It's on the plate to patch tomorrow. Also, I've been learning Dist::Zilla, which is rjbs' hotness for simplifying module releases, and Moose, the new object model for perl5. Note: It seems that almost every module I use involves rjbs at some point. That's kinda creepy in a way. If he dies, I'm fucked.
You know what? This fucking feels great. I'm way behind the times in terms of modern perl but I'm coming up to speed fast. Will I keep this up? Time will tell. I certainly hope I will.
General Life Update: About 5 weeks ago, I got laid off from a job I should have quit eons before, and one that perhaps I shouldn't have taken in the first place. Anyway, my stress level is low. My blood pressure is back to normal and I'm losing weight. Job hunting really sucks and can be depressing as hell sometimes. I've got some decent leads though and things are looking up. However, if you'd like me to work for/with you, don't hesitate to hit that resume link and toss me an email. :)
Code: I have a problem. For the last 18 months, I've done little programming. A bit of perl and C here and there. Some objective-c. For the last 5 weeks, I've opened my editor quite a few times and sat starting at code, unsure of where to start. I have another problem too. I hang out on irc with rjbs, one of the most prolific coders I've ever known. (It's good code too!) He's starting to wear off on me. Better said, I'm very envious of his prolific coding which seems to have spurred me on to action.
A few days ago, he blogged about a deficiency or 12 with the Dropbox standalone daemon for linux. Specifically, dropboxd tells you precisely fuck-all about what it's doing. (The other day, mine kept exiting after a few minutes of running. I had to debug the thing with strace to find out that I needed to upgrade it due to some protocol change.)
"I see a Dropbox client Perl library in my future...", rjbs said. I looked in my src directory and realized that I'd started on one in December. I grabbed rjbs, got his thoughts and behold, Net::Dropbox was born. it is probably more accurate at this point to say that it was conceived. It'll be born when I release the little bastard.
Along the way, I heard nperez mumble about lack of proper documentation in CPAN::Uploader, an rjbs module/app I use often. So... I patched it. In that process, I found a bug in Getopt::Long::Descriptive. It's on the plate to patch tomorrow. Also, I've been learning Dist::Zilla, which is rjbs' hotness for simplifying module releases, and Moose, the new object model for perl5. Note: It seems that almost every module I use involves rjbs at some point. That's kinda creepy in a way. If he dies, I'm fucked.
You know what? This fucking feels great. I'm way behind the times in terms of modern perl but I'm coming up to speed fast. Will I keep this up? Time will tell. I certainly hope I will.
