Red Green Software

We take stopped projects and get them going again.

Browsing Posts in perl

YAPC::NA::2010 – Perl is alive!

This was my first YAPC, and I wasn't really sure what to expect.

What I experienced was absolutely amazing!

There is a perl community that is alive and growing.  Perl5 releases may have been troublesome in the past, but are now being brought under control.  They are organized, and quite predictable now.  Just look at the Perl5.10 and Perl5.12 releases.

The history of perl was shown.  I never realized that perl was so old!  Yet perl isn't old and crufty.  It is alive and growing.  Thanks to ideas developed in perl6, we can enjoy modern OO with Moose and Web development frameworks like Catalyst.

The mantra of “There is more than one way to do it” is indeed perl.  That's why there are so many modules in CPAN.  It allows perl to try different solutions to problems before committing to a single solution.

I attended many of the introduction to sessions, which gave a quick overviews of Moose, DBIx::Class, and Catalyst.  All of which are modern perl, built upon the Moose OO system.  I'm sorry, trying to give an overview of just one of these wouldn't do them justice, nevermind all three of them, and this was just part of Day 1!

The talk on Modern Perl peaked my interest.  It must be about perl6, right?  Clearly the old perl5 couldn't be considered modern, could it?  Well, they are both considered to be “Modern Perl”.  Why?  Because modern perl5 programs are based upon Moose.  And Moose was based upon ideas in perl6!  The perl community is alive and growing, and enjoys borrowing good ideas.  With Moose, it has allowed the perl community to experiment, and come up with new ideas of how to solve problems in OOP.

I must interject here – since this was my first year, I was pulled aside and given a VIP ribbon for my badge.  I'm a VIP because I'm new to YAPC.  They wanted to know what I was doing with perl, and if I had any questions about anything with perl.  I asked about roles in Moose, I didn't quite understand where they fit into things.  I've worked with OO before in C++, but roles didn't seem to be anything that fit into anything I knew about OOP.  Well, after a few, “it is like Java interfaces, but it isn't” comments, which had me scratching my head, I suddenly understood roles.  Roles are similar to inheritance (the object gets those attributes defined in the role), but they aren't inherited.  Here's the example given to me.  You have objects that are modeled after a room.  Some objects you can sit on, some you can't.  You can define this in a role, and apply it to what you want.  Inheritance is vertical, roles are horizontal.  In other words, there are objects that won't ever fit correctly into an inheritance tree, but roles allow you to apply common attributes to objects.  And this is something that is solved in perl today.  Wow!

I'm  using perl to interface to websites on the web.  Some have been painful due to excessive use of javascript to implement navigation.  (sigh)  I was told one word:  “Selenium”.  Ok, I had heard of it before, but wasn't sure exactly what it was.  Why, yet another gem in CPAN that I didn't know about!  It allows you complete control over a web browser.  Who cares about how much ajax/javascript is on the page!  Let the browser handle all of that, and you tell the browser what to do.  And yes, it can handle popup authentication windows, yes, it can handle ajax.  If it works in a browser, you can script it with Selenium!  Why have I been wasting so much time trying to use WWW::Mechanize against javascript based sites?  No more of that nonsense with Selenium!

Next is Lightning talks.  Quick, short talks about a given topic.  This was the highlight of the day for me.  Quick talks that covered about everything!  One talk was singing a silly tune.  Another was about what do to with the closet that's full of YAPC conference T-shirts.  Another, why Python is better then perl.  Hey, can you do that at YAPC?  Well, yes, you can, and to be fair, he spent the last half of his talk going over what perl does better then Python!

Then there were modules mentioned.  Some of which I had heard of, but never used.  Others I had no idea about.  But many of them, I just said to myself, “Why haven't I ever heard of these modules before now?”.  I almost wish there was a newsletter or electronic magazine that would allow module authors to give a public announcement about their modules and what they can do for you.  Or, like in the case of MooseX::POE, where the author tells you not to use this module because it sucks, why it sucks, and what you should use instead.  This type of information would really be useful to know when searching through CPAN!

Modules in my notes that I must experiment with:

  • local::lib
  • CPAN::Mini
  • perlbrew
  • autodie
  • Modern::Perl
  • Regexp::Grammars
  • IO::Prompter
  • Try::Tiny
  • DBIx::Class::TimeStamp
  • KiokuDB

There was so much information about what new modules are available, I really felt like I didn't know anything at all about perl, or what was going on in the community!  But I'm actively using perl everyday. Really!  How can I be so out of touch with everything that is going on?

This type of information is the most valuable that I gained from attending YAPC.  I've learned about even more time saving modules, then I knew before.

I'm not ready to give a talk, yet, but I'm definitely ready for YAPC::NA::2011 in Asheville, North Carolina!  Teach me more about perl!

Building your own version of perl is indeed the best way to know what is in your perl, but it also keeps you safe from system perl updates.  But what about when your perl breaks in strange ways?

From my apache error_log:

Can't load '/opt/perl510/lib/5.10.1/i686-linux-64int/auto/B/B.so' for module B: /opt/perl510/lib/5.10.1/i686-linux-64int/auto/B/B.so: failed to map segment from shared object: Permission denied at /opt/perl510/lib/5.10.1/i686-linux-64int/XSLoader.pm line 70.

Can't load '/opt/perl510/lib/5.10.1/i686-linux-64int/auto/IO/IO.so' for module IO: /opt/perl510/lib/5.10.1/i686-linux-64int/auto/IO/IO.so: failed to map segment from shared object: Permission denied at /opt/perl510/lib/5.10.1/i686-linux-64int/XSLoader.pm line 70.

Can't load '/opt/perl510/lib/5.10.1/i686-linux-64int/auto/Fcntl/Fcntl.so' for module Fcntl: /opt/perl510/lib/5.10.1/i686-linux-64int/auto/Fcntl/Fcntl.so: failed to map segment from shared object: Permission denied at /opt/perl510/lib/5.10.1/i686-linux-64int/XSLoader.pm line 70.

The script that I was writing works correctly when run from the shell.  But when called under apache, forget it!

The clues to what is really happening here can be found in the setroubleshoot logs:

setroubleshoot: SELinux is preventing the send_alert.pl (httpd_sys_script_t) from executing /opt/perl510/lib/5.10.1/i686-linux-64int/auto/Data/Dumper/Dumper.so. For complete SELinux messages. run sealert -l 15f88a0f-9133-49de-8946-24d241a0219c

SE Linux is here to ruin your day!  So turn it off already.  That seems to be the solution that everyone else uses when they confront SE Linux issues…

sealert explains:

sealert -l 15f88a0f-9133-49de-8946-24d241a0219c

Summary:

SELinux is preventing the send_alert.pl (httpd_sys_script_t) from executing /opt/perl510/lib/5.10.1/i686-linux-64int/auto/Data/Dumper/Dumper.so.

Detailed Description:

SELinux has denied the send_alert.pl from executing /opt/perl510/lib/5.10.1/i686-linux-64int/auto/Data/Dumper/Dumper.so. If send_alert.pl is supposed to be able to execute /opt/perl510/lib/5.10.1/i686-linux-64int/auto/Data/Dumper/Dumper.so, this could be a labeling problem. Most confined domains are allowed to execute files labeled bin_t. So you could change the labeling on this file to bin_t and retry the application. If this send_alert.pl is not supposed to execute /opt/perl510/lib/5.10.1/i686-linux-64int/auto/Data/Dumper/Dumper.so, this could signal a intrusion attempt.

Allowing Access:

If you want to allow send_alert.pl to execute /opt/perl510/lib/5.10.1/i686-linux-64int/auto/Data/Dumper/Dumper.so: chcon -t bin_t '/opt/perl510/lib/5.10.1/i686-linux-64int/auto/Data/Dumper/Dumper.so' If this fix works, please update the file context on disk, with the following
command: semanage fcontext -a -t bin_t '/opt/perl510/lib/5.10.1/i686-linux-64int/auto/Data/Dumper/Dumper.so' Please specify the full path to the executable, Please file a bug report (http://bugzilla.redhat.com/bugzilla/enter_bug.cgi) against this selinux-policy to make sure this becomes the default labeling.

While SE Linux can be frustrating at first, once you know where to look and what to for in the setroubleshoot logs, you'll soon find that it tells you what you need to do in order to get things working correctly.  And when someone tells you to turn off SE Linux as 'the fix' to the SE Linux problem, let them know that isn't a solution.

If you see something like this when installing MIME::Tools:

t/Smtpsend.t ......... accept failed: Connection timed out at t/Smtpsend.t line 46.

# Looks like your test exited with 110 before it could output anything.

t/Smtpsend.t ......... Dubious, test returned 110 (wstat 28160, 0x6e00)

Then, I bet you have a dual-core or dual-processor system!

Easy enough to fix, using your editor, edit t/Smtpsend.t at line 46:

vi +46 t/Smtpsend.t

Add a line so that the section looks like this:

sleep 1;

# In the parent
my $s = $sock->accept();
if (!$s) {
    kill(9, $pid);
    die("accept failed: $!");
}

Then, make test, and make install.  The problem is the $sock->accept() gets called before the connection.

I will be attending YAPC::NA 2010. Yet Another Perl Conference at the Ohio State University.

I wish I could afford to stay for the Moose and Catalyst training classes, but I think this year it will be enough to just attend the conference.

Having been an avid perl user for quite some time now, I understand a lot more about why perl people hate RHEL, RedHat Enterprise Linux (as well as CentOS), and why some people hate perl.  RedHat has a bad, but well earned reputation for breaking perl.  They like to upgrade perl versions, and include some older perl modules, and other things that just break perl badly.  So what does a perl loving perlmonk do?

Well, you could whine about RHEL in #perl on the freenode servers.  But whining isn't productive or helpful, it just annoys people that need to get things done, and doesn't help anyone get the most out of their perl experience.

So what do you do?  Do what most power perl programmers do — Build your own perl.  That is probably the best kept secret in the perl developer community, unfortunately!

When you do that, you can safely ignore the perl your Linux distribution includes.  You can keep your Linux distribution updated, and you won't have to worry about breaking your production perl programs.  And, when there is a new version of perl, you can build it, and safely test your code with the new version!

Building perl

First, download the perl source.  You will end up with the file latest.tar.bz2. 

Extract with tar -xjf latest.tar.bz2 then change into the source directory.  Today, that is perl-5.10.1.

Now comes the questions:  Do you want 64 bit integer support?  Do you want threads, or do you want your perl to run faster?  There is some overhead in making variables thread-safe.  Some of these options are discussed at "When perl is not fast enough".

Here are the defaults that were recommended to me by Khisanth in #perl:

sh Configure -Dprefix=$HOMEperl5100 -Duseshrplib -Dusemultiplicity -Duse64bitint -Duseperlio -Duselargefiles -des

Ok, that's not what I use.  I don't need a shared perl library file (which run slower), because I won't be embedding perl anywhere.  I haven't found anything on usemultiplicity, so I don't use that either.  But, everything else is good.  This is what I have been using:

sh Configure -Dprefix=/opt/perl510 -Duse64bitint -Duseperlio -Duselargefiles -des

Note:  Using $HOME doesn't actually work, I think he meant not for the shell to replace $HOME with your home directory, but rather it to be replaced by the person typing.  Also, I am using the /opt directory, because I want my built perl to be used system-wide.  For local use, or testing, -Dprefix=/home/stevet/perl510 is what I use.

Next, comes the build process: make  Then, comes the test process: make test  If everything passes: make install

Congratulations, in your prefix directory, you now have a bin directory that contains: perl, perldoc, and cpan.  Try it out, /opt/perl510/bin/perl -v should display the version information:

This is perl, v5.10.1 (*) built for i686-linux-64int

Testing new versions of perl

To test new versions of perl, use the above directions, or run perl -V on your current production perl.  (This will show what build options you used.)  Grab the source to the new version of perl, and do everything the same except the -Dprefix.  Install the new perl into another directory.  Then, use the new perl with your code.  If it works, wonderful!  If not, continue to use the original perl binary until you get the new version worked out.

Most of the time, it is simply a matter of installing needed modules into the new version.  Make sure you run the new perl's cpan!

When cpan goes bad

This is why I am writing this.  Things went horribly wrong when I tried this.  Strange failures installing XML::Parser, and then again installing MIME::Entities.  Consult the guide and don't panic.

First, make sure that you have read the "Life with CPAN" guide.  Some of the things it talks about applies to having your own perl build.  But that only gets you so far.  Debugging cpan install failures is quite a challenge sometimes.  But, it does get easier with time and practice.  So let's get started with some CPAN debug tips, and then I'll finish up with my personal experiences with XML::Parser and MIME::Entities.

Debugging CPAN installs

Ok, you are at the cpan prompt, you type in: install DBD::mysql and suddenly pages of errors go flying by.  Now what?  Start with the very first line that gives an error.  But that was many screens ago, how do you find that?  Use the *nix tools available for just such things:  screen or script.  In screen, type ctrl-a and H.  That turns on screen logging, and you'll see it starts logging everything to a file called screenlog.0.  With script, you would type something like: script -c cpan and the output will be in a file called typescript.

Ok, so the first line that gives an error.  What type of error is it?  Does it look something like this:

In file included from dbdimp.c:20:
dbdimp.h:22:49: error: mysql.h: No such file or directory
dbdimp.h:23:45: error: mysqld_error.h: No such file or directory
dbdimp.h:24:49: error: errmsg.h: No such file or directory

This is from the c compiler (in this case gcc).  It is saying that it needs some header files.

Missing header files

This means that you don't have the development version of the program or library installed on your system.  Using your Linux distribution tools, install the development version. 

In the above case of mysql, it would be something like this for RHEL/CentOS: yum install mysql-devel.

Missing modules

When cpan goes to make test, does it error out with: Can't locate Test/Pod.pm in @INC (@INC contains: ...).  Ok, this is probably a minor bug in the module.  The module is supposed to list all of the modules that it has dependencies on, but maybe the author forgot one.  At the cpan prompt try: install Test::Pod, or whatever the module name was that it couldn't find.  If the missing module installs, retry installing the module that was giving you problems.  Odds are good that it will work fine now.

Failing tests, or what to try next

Probably the best part of perl, and of cpan, is the testing modules.  These allow the developer to design tests to exercise their code, and the libraries they depend upon to make sure that they function correctly.  But, what when those tests fail?

First stop is search.cpan.org.  Search for the module that you are having problems with.  Select the link to the module, and then on the right hand side click, View bug reports.  Look through the reports.  Are other people having the same error that you are?  Or, even better, are there patches available that fix the problem?

If you don't see your particular error listed, you might want to report it to the developer.  Or, even better, figure out what caused the error and report that!  Writing code that works well on many platforms is hard, and the developer might not have access to a machine like what you are using.

As a last resort you can also go to http://www.cpantesters.org and search for the troublesome module there.  While you won't find any answers there, you will see if anyone else is having problems installing the module, and what error messages they are getting.  Here are cpan testers' results of installing XML::Parser for example.

Google the error message

Just use google on error messages.  Don't use google when working with normal perl, because there is a lot of old perl code out there on the Internet.  There are still tutorials, which, when they were written 10 years ago were cutting edge.  But now, they show you the wrong way to write modern perl code.  If you want modern perl help use perlmonks.

Ok, so maybe you have found something, like what I found on my failed test installing XML::Parser.

XML::Parser

First, I got:

Expat.xs:12:19: error: expat.h: No such file or directory

Ok, so install the expat-devel, using yum and you're set, right?  No, this one gets much worse.  The module builds, but now it fails tests.  Searching around, I find out that debian is aware of this bug, thanks to failing perl tests.  What happened was a bug was found in the expat library that caused a security issue.  The library was quickly patched, however, the patch actually broke the expat library.  It wasn't detected by the expat developers, but it was detected by XML::Parser. 

Remember those module tests that are so great?  Yes, they are great at finding problems, but sometimes it is slow getting the fixes.  And RHEL, well, they have the broken security patch for expat.  They haven't gotten the latest correct patch into their system.  So, just like building your own perl, now you build your own expat.

Downloaded the expat 2.0.1 library from http://expat.sourceforge.net/ and installed it into /opt/expat.  Finally, a working, bug free, xml parsing library is installed.

Configuring XML::Parser to use the working expat library was a challenge, but by doing look XML::Parser in cpan, I was able to build using:

perl Makefile.PL EXPATLIBPATH=/opt/expat/lib EXPATINCPATH=/opt/expat/include

Then make (which compiles the module), make test (which now passed for me), and finally make install.

MIME::Entity

This on fails like this:

t/Smtpsend.t ......... accept failed: Connection timed out at t/Smtpsend.t line 46.
# Looks like your test exited with 110 before it could output anything.

If you are seeing this error, I would almost bet that you are using a dual-core or have multiple processors!  The bug is in the test script itself.  Use look MIME::Entity at the cpan prompt to get into the build directory.  Edit the t/Smtpsend.t with your favorite editor, and jump to line 41.  You should be right before the following lines:

# In the parent
my $s = $sock->accept();

Above these lines add: sleep 1

Ok, type: make test and it should pass just fine.  What I think happens is when the test script forks on a dual core system, the connection is accepted before the connection happens, so it fails.  But, with a 1 second sleep, the accept works correctly, and the test now passes.

Crypt::SSLeay

This fails on:

t/01-connect.t .. 1/8
#   Failed test 'Net::SSL->new'
#   at t/01-connect.t line 25.
# SSL negotiation failed:  at t/01-connect.t line 11
#  at t/01-connect.t line 11
# ; Interrupted system call at t/01-connect.t line 11
# ; Interrupted system call at t/01-connect.t line 11
# ; Interrupted system call at t/01-connect.t line 11
# Looks like you failed 1 test of 8.
t/01-connect.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/8 subtests
        (less 7 skipped subtests: 0 okay)

Again, the problem is with the test.  If you failed this test, it means you are installing this on a server that has https running on port 443.  Edit the t/01-connect.t script, and search for the two occurrences of 443 that look like:

        PeerPort => 443,

and

skip( "nothing listening on localhost:443", 7 )

Change them to some other port that isn't in use, like 4443.  Re-run make test.  If passing, now make install.

Wouldn't it be easier to just force the install?

No.  Forcing the install of perl modules that fail their tests is just asking for trouble.  Find out why it is failing.  If the test is bad, fix the test.  In the case of the expat bug, you would be opening your system up to exploits from malformed XML.  Really, don't force installs.

Perl Logfile

Comments off

Logging to a file

Using search.cpan.org there are many options to logging to a file.  Finding just one that does exactly what you need can be a challenge.

Here, I will talk just about one:  Log::LogLite

LogLite was almost exactly what I was looking for, but I found it to be lacking in one area:  The date output formatting was most useless, and not customizable. "YYYY-DD-MM"!?  Why not YYYY-MM-DD like most computer programmers like, since it alpha sorts correctly?

A simple change near line 69 like so:

my $line = $self->{TEMPLATE};
  $line =~ s!<date>!$self->date_string()!igoe;
  # changed to $self->date_string() so module can be customized by deriving from.

And now I can derive modules from Log::LogLite and customize the date formatting.

package MyLog;
use Log::LogLite;
 
@ISA = ("Log::LogLite");
 
sub date_string {
 my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
 return sprintf( "%02d/%02d/%04d %02d:%02d:%02d",
  $mon + 1, $mday, $year + 1900, $hour, $min, $sec );
} # of date_string

But since I have to modify the original module in order to have the flexibility to customize it, I just renamed the module LogLite.pm to MyLog.pm, made the above two changes and copy it around to where I need it. 

So much for code reuse.