Bring on the one liners!

I have been playing with Perl for a couple of years, off and on, not really that intensely. About 6 months ago I really picked up the pace when my last job released its online training courses. I ended up taking the 3 main Perl courses and learned A LOT, all at the companies expense. Almost 2 months ago I landed the job I am in now, as a Perl Developer, and watch as my learning curve took a sharp incline towards the ceiling. I love it when it does that and I really into what I have to learn.

Since getting this job, I have written a couple of scripts for work here that are a few hundred lines long and others that are only a couple hundred lines in length. Not too shabby and they work great.

Recently though (starting last week), I have been working on a Production issue. I did a fix for the issue but have since then been learning how the scripts work. The project is mainly shell scripts, but one of the scripts that is called is a ".sh" file that only contains... yup, you guessed it... a Perl one liner. This one liner is supposed to compare two files and return the lines that aren't duplicates.

Well, in my examination, I will be darned if I can find any where that the one liner is actually doing a comparison. To me, it looks like the script just simply outputs the first file and then appends the second file.

Anywho, that issue aside, during the whole research that I have been doing, I have had the opportunity to really delve into how to do one liners in Perl. It is really rather interesting. Granted, its not my favorite way to code Perl, but for the quick dirty job that takes seconds, it is definitely the way to go.

Heads up --> Opinion Ahead

A buddy from work and I were talking this morning regarding a project that they are wanting to build. He came right out and asked me about Perl's capabilities. I guess that the project that they have to build would be for system management and notifications to distribution lists. It would be databased and they would like a gui for it as well.

"Ok", I said, so what is the issue? I guess that one of his colleagues make the comment that Perl just isn't the way to go for such an application. And I quote: "The database interface is "eh" and the graphical capabilities are not that good either. Perl is really only good for text modifications."

I somewhat choked. Sure, text is what Perl was designed for, but to make a statement so completely confounded without first seeing what has changed with the language is just lame!

It seems that this guy hadn't touched ANY Perl in well over 10 years. Ok, 'nuff said. That means that:

#1. He really hasn't seen Perl lately
#2. He really hasn't seen Perl lately
#3. Perl 10+ years ago and Perl today are two completely different beasts.

Yes, I know #1 and #2 are the same, that was completely on purpose. I listened to the list of requirements for the application and they sound totally feasible for Perl. I cannot divulge them here as it is work related ( I am sure you understand), but believe me, Perl wouldn't have many, if any, issues. Granted, the coding would be a little intense, haveing to use TK, DBI and other assorted modules, but aside from that, it sounds like it would be a sweet application.

The big plus for Perl is that it is a language approved and used inside the organization, so no having to get approval for it. That should make them lean in the right direction.

An Enigma machine on eBay....Oh My!

If you are a cryptography buff or even an aficionado, then I can imagine that your eyebrow raised at the mention of one of these beauties actually being auctioned off.

It seems that someone in Italy is trying to cash in on a bit of history by offering one of these machines to the public. If this turns out to be a true Enigma machine, then this is an incredible find. I can only imagine how lucky the winning bidder will be feeling.

Here is the link: http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=270146164488

Granted, in my opinion, something as monumental as the Enigma machine definitely belongs in a museum. My only hope is that, whomever buys it does not keep it locked away, but instead puts it on display for others to see and enjoy.

Of course, if you happen to be a hobby cryptographer, and a handy one at that, then you could always build your own Enigma machine in your workshop. That is definitely an interesting prospect for a project. Hmmm.

Anywho, enjoy the auction. It will be interesting to see how high the bidding goes.

Watch that punctuation

I'll tell you, there is nothing like coding all day and having a really good time doing so, and then, right around mid-afternoon you hit an error that just stops you. You can't figure it out for the life of you.

I am working on a script to automatedly download ALL of CPAN. ( I know.... WHY? Because I want to, that's why. Plus, my laptop isn't always able to get online and when I need a module, if I have it readily available, I can manually install it.)

Basically, the script fetches a copy of the CPAN modules list, and I am then parsing out of the HTML that was returned, the paths to the modules, including the module names. Here is the section of code that was bugging me:


if($line =~ m/\.gz/)
{
@elements = split(/"/, $line);
print PATHS ("$elements[1]\n");
}


I was really wanting this to work ( and the code above does), but when I IM'd Merlyn (Randal Schwartz), he noticed that I was missing a double quote in the print path. Man, after all day coding, I don't blame myself for missing that one. Thanks Merlyn for the second set of eyes on that one!

Once I have the code completed I will post it here.

[TAOREs] File or Directory from a listing

Welcome!! This is the first article in my new column on regex's, where I will cover regex's that I have written or found(obviously giving credit where credit is due) and also lessons on writing regex's. I know that I am starting this column without even a lesson on regex's, so if you are new to regex's and really don't know much or anything about them, then this will seem quite foreign and I appologize for that. Please know that the lessons will follow shortly. Without further ado, lets jump in feet first.

I was working on a script a couple of weeks ago where the script had to FTP into a server, grab a directory listing from a specific directory, and then send an email to a distribution list if a file was found in the directory.

Seems pretty straightforward, but I quickly realized that this was an opportunity for me to get a little practice writing regex's. The remote server was windows, so when the FTP connected and ran the "dir" command, the output had the usual FTP banter, but also a line similar to the following when a file was found:

drwxr-xr-x 7 Administrators group 0 Jun 04 22:07 myfile.txt

This is the typical output format for FTP on a Windows machine. Yes, it is the same format as the long listing provided by an "ls -l" on a unix machine. So, knowing that, I quickly set out to write a regular expression to match just the "myfile.txt" file name of that line.

Here is the regular expression that I came up with to match the file name in that line:


m/^.+\s+\d\s+\w+\s+\w+\s+\d+\s+\w+\s+\d+\s+\d{1,2}:\d+\s+(\w+\.\w+)$/


This is pretty straight forward (if you know something about regex's). Let me break it down with the regex formatted slightly differently:


m/ # start the match
^ # start from the beginning of the line
. # match any single character
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\d # match a digit
\s # match a space
+ # match preceding element one or more times
\w # match a word character
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\w # match a word character
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\d # match a digit
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\w # match a word character
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\d # match a digit
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\d{1,2} # match a digit, at least once, but up to twice
: # match a colon
\d # match a digit
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
( # Start subexpression group for capturing
\w # match a word character
+ # match preceding element one or more times
\. # match a period, yes, this needs to be escaped as the period is a regex character too
\w # match a word character
+ # match preceding element one or more times
) # End the subexpression group for capturing
$ # match the end of line
/ # end the regex


I know, it seems a bit messier, but this is one way of writing a regular expression so that you can annotate what is happening. Personally, I like the first version better.

Now, with this regex, if your file name (or directory name) was in a different format that something like "myfile.txt", then you would have to edit the regex to reflect that difference, or risk the code not working.

In the above though, the part of the regex that is enclosed in ( ) will be placed in the special variable $1 so that its value can be referenced elsewhere in the script.

The Art Of Regular Expressions

I have been playing with Perl now for the last couple of years, somewhat actively. My playing turned into a job when I accepted the position that I am presently in. Before I found this job though, my last job was kind enough to offer online, self-paced CBT type training that you could do in your own time. So, I took a gander at it and they had a number of Perl courses.

Now, I am ALWAYS learning and can usually be found with a computer book somewhere in my vicinity pretty much most of the day. I never miss a chance to learn something new if I can help it. That said though, I still will admit that there is nothing like some of the training you can find that gets right down to the nitty gritty without beating around the bush. This is especially true when you are trying to get a solid base in the basics of whatever you are learning.

The CBT's that my last job provided me access to did just that. Not only did they give me a really good understanding and grounding in Perl basics, but they also gave a really good lesson on Regular Expressions.

Regex's (as they are also affectionately referred to) can be some of the most elusive topics to get a grasp on in Perl. I am no expert by any stretch of the imagination, but the course that I took gave me an incredible head start into the world of regex's.

Before that course, I knew a fair amount about using pattern matching in Unix. I was quite adept at shell scripting and could write an awk script or two to do what I needed to. But, when it came to Perl's regex capabilities, I was lacking. ***BAM*** Then came that course. It truely opened my eyes to the overwhelming beauty and seductive power that Perl regex's have.

With a little "to-the-point" training and guidance, I think anyone can easily get a grounding in Perl regex's. In fact, I think I am going to add on to this blog and start a column regarding regex's. I will not only post some mini lessons on regex's, designed to further your knowledge in how to assemble regex's, but also examples of regex's that I have written as well, that work for the purpose they were created. Hopefully all of that will provide you a new route for reference when trying to use regex's.

I think I will call the new column "TAOREs" (for The Art Of Regular Expressions). I hope that you enjoy and also contribute any suggestions that you may have as well.

The Art of the Automated Download

How many of us have found a site that ended up having a whole bunch of files on the site, for download, that you really wanted? Well, I found a site just like that. Too many sites use PHP or something else, to where it used ID's and other codes to refer to the documents, instead of just putting links into their web pages. Once in a while though, you do come across a site that DOES put the actual links to the files on the page.

In the case of the site that I am referring to, there are a little over 2000 files that I wanted to grab. The thing is, I didn't want to have to site there and "right-click-save as" for each and every one as that would have taken "days" to complete. So, noticing that all of the files had actual URLs that led right to them, I looked at the page source. There it hit me. Perl!

So, I copied the source and quickly drummed up a regular expression which grabs all the URLs of all of the pdf files on the page. After I grabbed the URLs, I put together some code which quickly went out and grabbed each and every one in turn and saved it to my hard drive.

This sounds all simple and stuff, but for someone like me who still considers him a novice in the Perl world, it did take a couple of hours of research. First I tried to use the "WWW::Mechanize" module and was able to retrieve a complete list of the pdf files and their paths, but not the actual files. I tried other packages and such, delving into LWP itself, but I could not for the life of me get this code to actually download the files.

I found "lwp-download" and gave it a shot. Wow! It looked like it was working, up until the 904th document, where it died. I couldn't figure it out nor understand what was going on. Why was this dying at the same point every time. Well, I did eventually figure it out and was able download ALL of the over 2000 files to my hard drive. I couldn't believe it, but the routine I used only took a few minutes to download all of the files (granted, I have 15 Mb FiOS for my internet, so please bear that in mind as well).

Just as an FYI (and you can get this looking at the code, I used the LWP::Simple::getstore() routine to download the files. It was a lot easier than going through the process of figuring out why my WWW::Mechanize wasn't working, believe me. I will figure that module out later, but for now, this did exactly what I wanted.

This is probably a bit much and others would more than likely have a better way to do it, but here it is. Here is the code I used to parse HTML code for its links and download them.


#!/usr/bin/perl

use strict;
use warnings;
use File::Basename;
use LWP::Simple;


#############################################
# Read the entire HTML file into an array, line by line so that we can
# parse out the information we need, one line at a time.
#############################################
my @code = `cat /home/jlk/development/perl/stampAlbums/code.txt`;

if (-e "/home/jlk/development/perl/stampAlbums/files.txt")
{
`rm /home/jlk/development/perl/stampAlbums/files.txt`;
`touch /home/jlk/development/perl/stampAlbums/files.txt`;
}


foreach my $line (@code)
{


############################################
# The following code takes the site's html file(in this case, it
# is the stampalbums.com download site) and parses out all of the
# downloads URL's.
############################################
if($line =~ m/^\s+ {
my @splitLine = split(/\"/, $line);

##############################################
# Now, open the file in which you will store all the URL's to
# the files and write each URL on a seperate line.
##############################################

open(FILES, ">>/home/jlk/development/perl/stampAlbums/files.txt");
print FILES ("$splitLine[1] \n");
close(FILES);
}

}

###########################################
# Open the file containing all of the URLs
###########################################
open(FILES, "< /home/jlk/development/perl/stampAlbums/files.txt");

###########################################
# Do the download of the files
###########################################
foreach()
{
my $localdir = "/home/jlk/development/perl/stampAlbums/albumPages/";
my $path = "$_";
my($filename, $directories, $suffix) = fileparse($path);

LWP::Simple::getstore($_, $filename);
}



Further IP Validation information

After yesterday's posting on one way (of many) to validate an IPv4 address, I have some more information for those interested.

I had posted a question at one point to thescripts.com Perl forum, regarding an issue I was having. One of the more knowledgeable users over there (he goes by Miller), enlightened me as to a CPAN module that does the same thing without re-inventing the wheel. Please don't think I regret writing my script. On the contrary, I am very happy I did as I learned a couple of lessons in doing so.

Anywho, the module mentioned above actually validates IP addresses for you to check validity just as my script from yesterday does, it just does so slightly differently.

I thank Miller for his gracious input and the link to the CPAN module!!

IPv4 Address validation

Well, in my quest to learn Perl and practice putting together regex's, I took on the task yesterday of putting together a script which validates if an IP address is valid or not. Now, when you say "is valid", what exactly does that mean?

Well, there are two criteria that IP addresses need to really meet:

1) They need to have 4 octets, each octet containing from 1 to 3 digits.
2) Each octet's digits must make up a number between 0 and 255.

With that in mind, I went directly after the first of the two, validating the format of the IP address entered. The regex that I initially came up with is listed in #1 and the other two regex's came from the "Mastering Regular Expressions" book:

#1 m/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/
#2 m/\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?/
#3 m/\d(\d\d?)?\.\d(\d\d?)?\.\d(\d\d?)?\.\d(\d\d?)?/

What is so great about Perl is TIMTOWTDI, which affords us being able to have 3 different regex's that come to exactly the same solution.

You may be asking yourself, what do I do with the regex(s) above? Well, you can take them and use them in your code to validate an entered IP address, or put it into a loop to test an entire file full of IP's to test their validity.

Here is an example of how to incorporate the above regex(s) into some code. Please know that this is only how I do it and my differ considerably from how you implement it:




#!/usr/bin/perl
 
use strict;
use warnings;
 
 
print("What is the IP Address you would like to validate: ");
my $ipaddr = ;
chomp($ipaddr);
 
 
if( $ipaddr =~ m/^(\d\d?\d?)\.(\d\d?\d?)\.(\d\d?\d?)\.(\d\d?\d?)/ )
{
    print("IP Address $ipaddr  -->  VALID FORMAT! \n");
    
    if($1 <= 255 && $2 <= 255 && $3 <= 255 && $4 <= 255)
    {
        print("IP address:  $1.$2.$3.$4  -->  All octets within range\n");
    }
    else
    {
        print("One of the octets is out of range.  All octets must contain a number between 0 and 255 \n");
    }
}
else
{
    print("IP Address $ipaddr  -->  NOT IN VALID FORMAT! \n");
}
 

Well, there you go, IP address validation. No, this script obviously does not take into account IPs that are reseverd such as the 192.168 or 10.246 addresses, but you can test for whatever you need to given the grouping in the regex that provides the $1, $2, $3 and $4 variables.

Happy Perl Coding!!!

Update on curing the hunger

You know, it is never a good idea to make rash judgments. They always tend to lead to something you didn't want or expect. That is one reason I said in my previous post that I would look into the Podcast idea.

Over the weekend, I listened to an older Perlcast that Josh McAdams made, where he interviewed Randal Schwartz. Whether or not that was made before or after Stonehenge started supporting Perlcast is for them to know. Either way, in the interview, Josh asked Randal his opinion on Perlcast and what he said was that he actually really like Perlcast and the idea of reporting the Perl Community news for all to listen to. He also mentioned that it wasn't trying to do Perl instruction over an audio feed, which Randal did not find any use in.

Hearing that I thought longer and harder about what I was leaning towards and realized that he is completely and totally correct. You have to see code to learn it, that is why there are so very many resources and documents on the web and also so very many books on the topic of Perl.

I am one of those people that when I come across something I want to know more about or if I somehow think of an idea for a pet project, I write it down somewhere. These scribblings are actually gathered into lists. Yes, I do eventually get to them, but sometimes it takes a while. Usually if I need something and its on a list or if I get bored I tend to reference my list(s) for something to do. I took a look this weekend at my list of pet projects and did see a couple on there that would afford me the opportunity to delve into some modules that I have been wanting to play with. So, that is where I will turn to to cure my hunger pangs, instead of trying to create a podcast that wouldn't be all that useful. (sure, a video podcast would do the trick, but I have neither the resources nor the hosting to hold the files, so that is definitely out of the question at the moment)

Any who, back to work at learning Perl. Happy Monday everyone!!!