[TAOREs] File or Directory from a listing

Welcome!! This is the first article in my new column on regex's, where I will cover regex's that I have written or found(obviously giving credit where credit is due) and also lessons on writing regex's. I know that I am starting this column without even a lesson on regex's, so if you are new to regex's and really don't know much or anything about them, then this will seem quite foreign and I appologize for that. Please know that the lessons will follow shortly. Without further ado, lets jump in feet first.

I was working on a script a couple of weeks ago where the script had to FTP into a server, grab a directory listing from a specific directory, and then send an email to a distribution list if a file was found in the directory.

Seems pretty straightforward, but I quickly realized that this was an opportunity for me to get a little practice writing regex's. The remote server was windows, so when the FTP connected and ran the "dir" command, the output had the usual FTP banter, but also a line similar to the following when a file was found:

drwxr-xr-x 7 Administrators group 0 Jun 04 22:07 myfile.txt

This is the typical output format for FTP on a Windows machine. Yes, it is the same format as the long listing provided by an "ls -l" on a unix machine. So, knowing that, I quickly set out to write a regular expression to match just the "myfile.txt" file name of that line.

Here is the regular expression that I came up with to match the file name in that line:


m/^.+\s+\d\s+\w+\s+\w+\s+\d+\s+\w+\s+\d+\s+\d{1,2}:\d+\s+(\w+\.\w+)$/


This is pretty straight forward (if you know something about regex's). Let me break it down with the regex formatted slightly differently:


m/ # start the match
^ # start from the beginning of the line
. # match any single character
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\d # match a digit
\s # match a space
+ # match preceding element one or more times
\w # match a word character
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\w # match a word character
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\d # match a digit
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\w # match a word character
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\d # match a digit
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
\d{1,2} # match a digit, at least once, but up to twice
: # match a colon
\d # match a digit
+ # match preceding element one or more times
\s # match a space
+ # match preceding element one or more times
( # Start subexpression group for capturing
\w # match a word character
+ # match preceding element one or more times
\. # match a period, yes, this needs to be escaped as the period is a regex character too
\w # match a word character
+ # match preceding element one or more times
) # End the subexpression group for capturing
$ # match the end of line
/ # end the regex


I know, it seems a bit messier, but this is one way of writing a regular expression so that you can annotate what is happening. Personally, I like the first version better.

Now, with this regex, if your file name (or directory name) was in a different format that something like "myfile.txt", then you would have to edit the regex to reflect that difference, or risk the code not working.

In the above though, the part of the regex that is enclosed in ( ) will be placed in the special variable $1 so that its value can be referenced elsewhere in the script.

The Art Of Regular Expressions

I have been playing with Perl now for the last couple of years, somewhat actively. My playing turned into a job when I accepted the position that I am presently in. Before I found this job though, my last job was kind enough to offer online, self-paced CBT type training that you could do in your own time. So, I took a gander at it and they had a number of Perl courses.

Now, I am ALWAYS learning and can usually be found with a computer book somewhere in my vicinity pretty much most of the day. I never miss a chance to learn something new if I can help it. That said though, I still will admit that there is nothing like some of the training you can find that gets right down to the nitty gritty without beating around the bush. This is especially true when you are trying to get a solid base in the basics of whatever you are learning.

The CBT's that my last job provided me access to did just that. Not only did they give me a really good understanding and grounding in Perl basics, but they also gave a really good lesson on Regular Expressions.

Regex's (as they are also affectionately referred to) can be some of the most elusive topics to get a grasp on in Perl. I am no expert by any stretch of the imagination, but the course that I took gave me an incredible head start into the world of regex's.

Before that course, I knew a fair amount about using pattern matching in Unix. I was quite adept at shell scripting and could write an awk script or two to do what I needed to. But, when it came to Perl's regex capabilities, I was lacking. ***BAM*** Then came that course. It truely opened my eyes to the overwhelming beauty and seductive power that Perl regex's have.

With a little "to-the-point" training and guidance, I think anyone can easily get a grounding in Perl regex's. In fact, I think I am going to add on to this blog and start a column regarding regex's. I will not only post some mini lessons on regex's, designed to further your knowledge in how to assemble regex's, but also examples of regex's that I have written as well, that work for the purpose they were created. Hopefully all of that will provide you a new route for reference when trying to use regex's.

I think I will call the new column "TAOREs" (for The Art Of Regular Expressions). I hope that you enjoy and also contribute any suggestions that you may have as well.

The Art of the Automated Download

How many of us have found a site that ended up having a whole bunch of files on the site, for download, that you really wanted? Well, I found a site just like that. Too many sites use PHP or something else, to where it used ID's and other codes to refer to the documents, instead of just putting links into their web pages. Once in a while though, you do come across a site that DOES put the actual links to the files on the page.

In the case of the site that I am referring to, there are a little over 2000 files that I wanted to grab. The thing is, I didn't want to have to site there and "right-click-save as" for each and every one as that would have taken "days" to complete. So, noticing that all of the files had actual URLs that led right to them, I looked at the page source. There it hit me. Perl!

So, I copied the source and quickly drummed up a regular expression which grabs all the URLs of all of the pdf files on the page. After I grabbed the URLs, I put together some code which quickly went out and grabbed each and every one in turn and saved it to my hard drive.

This sounds all simple and stuff, but for someone like me who still considers him a novice in the Perl world, it did take a couple of hours of research. First I tried to use the "WWW::Mechanize" module and was able to retrieve a complete list of the pdf files and their paths, but not the actual files. I tried other packages and such, delving into LWP itself, but I could not for the life of me get this code to actually download the files.

I found "lwp-download" and gave it a shot. Wow! It looked like it was working, up until the 904th document, where it died. I couldn't figure it out nor understand what was going on. Why was this dying at the same point every time. Well, I did eventually figure it out and was able download ALL of the over 2000 files to my hard drive. I couldn't believe it, but the routine I used only took a few minutes to download all of the files (granted, I have 15 Mb FiOS for my internet, so please bear that in mind as well).

Just as an FYI (and you can get this looking at the code, I used the LWP::Simple::getstore() routine to download the files. It was a lot easier than going through the process of figuring out why my WWW::Mechanize wasn't working, believe me. I will figure that module out later, but for now, this did exactly what I wanted.

This is probably a bit much and others would more than likely have a better way to do it, but here it is. Here is the code I used to parse HTML code for its links and download them.


#!/usr/bin/perl

use strict;
use warnings;
use File::Basename;
use LWP::Simple;


#############################################
# Read the entire HTML file into an array, line by line so that we can
# parse out the information we need, one line at a time.
#############################################
my @code = `cat /home/jlk/development/perl/stampAlbums/code.txt`;

if (-e "/home/jlk/development/perl/stampAlbums/files.txt")
{
`rm /home/jlk/development/perl/stampAlbums/files.txt`;
`touch /home/jlk/development/perl/stampAlbums/files.txt`;
}


foreach my $line (@code)
{


############################################
# The following code takes the site's html file(in this case, it
# is the stampalbums.com download site) and parses out all of the
# downloads URL's.
############################################
if($line =~ m/^\s+ {
my @splitLine = split(/\"/, $line);

##############################################
# Now, open the file in which you will store all the URL's to
# the files and write each URL on a seperate line.
##############################################

open(FILES, ">>/home/jlk/development/perl/stampAlbums/files.txt");
print FILES ("$splitLine[1] \n");
close(FILES);
}

}

###########################################
# Open the file containing all of the URLs
###########################################
open(FILES, "< /home/jlk/development/perl/stampAlbums/files.txt");

###########################################
# Do the download of the files
###########################################
foreach()
{
my $localdir = "/home/jlk/development/perl/stampAlbums/albumPages/";
my $path = "$_";
my($filename, $directories, $suffix) = fileparse($path);

LWP::Simple::getstore($_, $filename);
}



Further IP Validation information

After yesterday's posting on one way (of many) to validate an IPv4 address, I have some more information for those interested.

I had posted a question at one point to thescripts.com Perl forum, regarding an issue I was having. One of the more knowledgeable users over there (he goes by Miller), enlightened me as to a CPAN module that does the same thing without re-inventing the wheel. Please don't think I regret writing my script. On the contrary, I am very happy I did as I learned a couple of lessons in doing so.

Anywho, the module mentioned above actually validates IP addresses for you to check validity just as my script from yesterday does, it just does so slightly differently.

I thank Miller for his gracious input and the link to the CPAN module!!

IPv4 Address validation

Well, in my quest to learn Perl and practice putting together regex's, I took on the task yesterday of putting together a script which validates if an IP address is valid or not. Now, when you say "is valid", what exactly does that mean?

Well, there are two criteria that IP addresses need to really meet:

1) They need to have 4 octets, each octet containing from 1 to 3 digits.
2) Each octet's digits must make up a number between 0 and 255.

With that in mind, I went directly after the first of the two, validating the format of the IP address entered. The regex that I initially came up with is listed in #1 and the other two regex's came from the "Mastering Regular Expressions" book:

#1 m/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/
#2 m/\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?/
#3 m/\d(\d\d?)?\.\d(\d\d?)?\.\d(\d\d?)?\.\d(\d\d?)?/

What is so great about Perl is TIMTOWTDI, which affords us being able to have 3 different regex's that come to exactly the same solution.

You may be asking yourself, what do I do with the regex(s) above? Well, you can take them and use them in your code to validate an entered IP address, or put it into a loop to test an entire file full of IP's to test their validity.

Here is an example of how to incorporate the above regex(s) into some code. Please know that this is only how I do it and my differ considerably from how you implement it:




#!/usr/bin/perl
 
use strict;
use warnings;
 
 
print("What is the IP Address you would like to validate: ");
my $ipaddr = ;
chomp($ipaddr);
 
 
if( $ipaddr =~ m/^(\d\d?\d?)\.(\d\d?\d?)\.(\d\d?\d?)\.(\d\d?\d?)/ )
{
    print("IP Address $ipaddr  -->  VALID FORMAT! \n");
    
    if($1 <= 255 && $2 <= 255 && $3 <= 255 && $4 <= 255)
    {
        print("IP address:  $1.$2.$3.$4  -->  All octets within range\n");
    }
    else
    {
        print("One of the octets is out of range.  All octets must contain a number between 0 and 255 \n");
    }
}
else
{
    print("IP Address $ipaddr  -->  NOT IN VALID FORMAT! \n");
}
 

Well, there you go, IP address validation. No, this script obviously does not take into account IPs that are reseverd such as the 192.168 or 10.246 addresses, but you can test for whatever you need to given the grouping in the regex that provides the $1, $2, $3 and $4 variables.

Happy Perl Coding!!!

Update on curing the hunger

You know, it is never a good idea to make rash judgments. They always tend to lead to something you didn't want or expect. That is one reason I said in my previous post that I would look into the Podcast idea.

Over the weekend, I listened to an older Perlcast that Josh McAdams made, where he interviewed Randal Schwartz. Whether or not that was made before or after Stonehenge started supporting Perlcast is for them to know. Either way, in the interview, Josh asked Randal his opinion on Perlcast and what he said was that he actually really like Perlcast and the idea of reporting the Perl Community news for all to listen to. He also mentioned that it wasn't trying to do Perl instruction over an audio feed, which Randal did not find any use in.

Hearing that I thought longer and harder about what I was leaning towards and realized that he is completely and totally correct. You have to see code to learn it, that is why there are so very many resources and documents on the web and also so very many books on the topic of Perl.

I am one of those people that when I come across something I want to know more about or if I somehow think of an idea for a pet project, I write it down somewhere. These scribblings are actually gathered into lists. Yes, I do eventually get to them, but sometimes it takes a while. Usually if I need something and its on a list or if I get bored I tend to reference my list(s) for something to do. I took a look this weekend at my list of pet projects and did see a couple on there that would afford me the opportunity to delve into some modules that I have been wanting to play with. So, that is where I will turn to to cure my hunger pangs, instead of trying to create a podcast that wouldn't be all that useful. (sure, a video podcast would do the trick, but I have neither the resources nor the hosting to hold the files, so that is definitely out of the question at the moment)

Any who, back to work at learning Perl. Happy Monday everyone!!!

The hunger and thirst for everything Perl

I have been playing with perl for about 2 years. I wrote a Perl script at my last job that just set up some directories and created some log files, nothing extravagant about it. Unfortunately, my last job really left little time for coding or learning to code Perl for that fact, given the insane and overwhelming workload that we were faced with. Even attempting to pick up some Perl while on lunch proved to be a daunting and sometimes impossible task. The only time I could really focus and learn some Perl was while at home after the kids were in bed asleep. Then and only then was I able to concentrate, was I awake enough to do so after being mentally fried from all of the days events.

Thankfully, the new job that I started a couple of weeks ago has afforded me nothing but the opportunity to code in Perl. Most of what I do is Perl coding and I am happier than a clam. I have learned so much in the last two weeks that I am beside myself at my adjustment to the learning curve. Before starting the job I had obtained Brian D Foy's "Student Workbook" that accompanies the Llama book (Learning Perl). The Llama book has questions and exercises at the end of every chapter, but the Student Workbook contains more exercises to assist with getting you in a better Perl state of mind.

Well, having completely turned my 'learning' attentiveness towards the realm of Perl, I have been thirsting for everything that I can get my hands on to satisfy my burgeoning hunger for Perl knowledge. One of the resources I turned to in hopes of "data....input" was podcasts. Upon searching though, I only found one true podcast related to Perl and that is "Perlcast". Don't get me wrong, Perlcast is absolutely AWESOME, even having a very prominent members of the Perl community, Randal Schwartz, as the roving reporter reporting all the Perl news that's fit to report. Other than reporting the latest Perl news, Perlcast also does interviews with members of the Perl community and segments from conferences.

For me though, this just isn't enough. I want MORE! I have searched a fair amount of the web and unfortunately cannot find any other Perl related podcasts. What was I hoping for? beleive it or not, I was hoping for a podcast on Perl that covered a different topic in each episode. Maybe go over topics such as:

• Scalars
• Arrays
• Hashes
• Modules (creating)
• many different episodes that each time choose a different module and go over it in detail
• Regular Expressions
• and so on, and so forth

There is such a multitude of possible topics, that the number of episodes could and would be immense.

Sure, some are out there probably saying, "So why aren't you producing these podcasts"? I have thought of that and believe it or not, I am still thinking about it. My biggest issue would be where to host it as I would need enough space to hold all of the podcast files. Yes, this area would be new to me and I don't have much expertise in creating these, so please bear with me. If anyone has any information, such as software recommendations for creating podcasts(OSS preferably) and also a good place to host it or share it as well would be great.

No, this is not an "I will do it", this is more of a "I will look into the possiblity". This would benefit me as well as I am still up and coming. I guess there is no better way to cure your hunger than to make your own meal(s).

What a difference a job makes

Well, after 6 1/2 years at the same job, I finally decided to get off my duff and see what was out there. Now, I am not one of those people who likes to hop from job to job because I like the stability and the stature you get from being at a job a long time. At my old job, I had quite the distinction of not only being the only true implementations Subject Matter Expert(SME), but I was also the only remaining 'original' member of the team. There was only two of us in the beginning and it was like that for quite a while, but as people left for other positions, I remained and held the product together. I wrote much of the original documentation for the project, maintaining it throughout my tenure.

Thing with my old job had steadily changed. The workload steadily increased (which isn't a bad thing), but with it came the stress of that increase. The clients would come into the queue and inevitably, the higher ups that were associated with the project would push my manager to assign the project immediately because they waited too long to get us involved to implement the project in a normal time line. The stress level really counts for a good percentage of my reason for leaving, not to mention that after our most recent merger, that ended up with the company we bought literally taking over the place, a whole slew of cutbacks were made in order to save everywhere they could. Now, cutting back is part of the large corporate environment and its expected. But, when they are cutting and cutting and then hiring a 'ba-zillion' middle managers because their organization believes in the "how many tiers can our organization have" approach with the "too many chiefs not enough indians" side effect that goes along with it, then you have raise a questionable eyebrow.

Sure, stopping the purchase of things like tissues and other goodies that made life a little easier is piddly because you can supply those yourself, but when things in the organization change so much that you are watching people walk out the door every week, even those with 20+ years, you get to thinking about #1.

So, I did and I took a look at what was out there. Believe it or not, it wasn't very long before I started going to some interviews, but the interview I was most anxious for was for the position I am in now.

I went from doing client implementations at a large bank, on an e-commerce platform with solutions such as AS2, HTTPS, SFTP, FTPS and a couple of other miscellaneous solutions, to working AT a client site for a company that does most of that companies tech work. What I am doing now it Secure FTP development using Perl.

The biggest pluses to my new job are that not only am I working what so far seems to be a great bunch of people, but I am doing coding with Perl almost every day. I have come to really like Perl in the last couple of years and to have a job that allows me to code in it is just INCREDIBLE!!!

Granted, there are those out there who do it every day and are either tired of it or are skewed by their development experiences who have told me not to be so excited about it, but I cannot help it. Try to remember back when you were excited about it, nobody could stop you from talking about it or keep you from your keyboard or the Perl forms. That is the state I am in now, just Loving what I am doing.

Of coding Perl in relation to Ksh scripting

I know that there are many differences between shell scripting and Perl, I am reminded of that daily while I constantly switch between the two. Although today, I ran across a difference that I wanted to share with everyone.

In scripting, you can do something like the following:

##### begin code #####

myVar = this

myNextVar = ${myVar}andthat

print $myNextVar

##### End Code #####

The output of this would be: thisandthat

The point above is that you can enclose the variable(except for the dollar sign) in curly braces and it is interpolated to its value and then whatever comes after the curly braces will be appended to the name.

Well, this becomes impossible to do as shown above as the curly braces are used for other things in Perl. So, I did some thinking and figured that I could use the concatenatin character to solve my problem. Here is the above code as how I would have written it in Perl:

##### Begin Perl Code #####

my $var = "this";
my $nextVar = "$var" . "andthat";

print $nextVar;

##### End Perl Code #####

The output of the above Perl code is exactly the same as the shell script, but as you can see, we have to put things together in a slightly different way.

Happy Perl coding!

What drives OSS?

It was recently posted to slashdot that boredom is what is driving open source developers. This may be true, but isn't it just one of the things that drives OSS development?

How many of us have researched for a production that does what we want, only to find that the software that does EXACTLY what we want is so far out of our price range that you would have to take out a loan to afford it? So, that said, isn't cost another thing that drives OSS? Most of the software out there is free, but if you want support, that is where you pay $$$, or find an online community. That is the advantage that OSS has over commercial software is even though you could pay for support if you wanted to, the community is large enough that you can normally find the support you need in a forum.

So it is boredom AND cost in my opinion, that drives OSS software developers. Granted, I have searched for a product and not found the features I would want. If I were a full fledged OSS developer, I would take to developing my own software in that case, leading to a third, albeit a little less immediate source of drive, but still another source that needs mentioning.

All in all, it is not just one thing, but many things that drives OSS developers, just like innovation.