Creating A Rubygems Mirror On Ubuntu 14.10

So I guess it is apparent that I am on a bit of a kick creating mirrors.  This is because they are creating mirrors internally at work and I want to have a better understanding of how things work (and what is needed to puppetize things). 

Before we can get down to the nitty-gritty of creating the mirror, we have to do a bit of prerequisite work first.   To start with, I will tell you that I am working on a fresh installation of Ubuntu 14.10.  That disclosed, ruby comes installed by default.  Please know that I will be only referencing the

The first thing that we need to get installed is the ruby-dev package.  You can install that with the following command:

$ sudo apt-get install ruby-dev

After you install that, make sure that your system is up to date:

$ sudo apt-get update
$ sudo apt-get upgrade

Now, we need to get a couple of gems installed:

$ sudo gem install net-http-persistent

Next we need to install a whole slew of things (inluding git):

sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev

Now it is time to install rbenv.  This tool will provide you the ability to install different versions of ruby (other than the default).  Its handy, so I am including it here for giggles.   Here are the steps to install rbenv (these should be done as you, not sudo/root):

$ git clone git://github.com/sstephenson/rbenv.git ~/.rbenv
- Add $HOME/.rbenv/bin to your PATH variable
- Add eval "$(rbenv init -)" to your .bashrc file

 Now you need to install the rubygems-mirror gem.  This is what is used to create the mirror:

$ sudo gem install rubygems-mirror

After that finishes, you will need to create the the '.mirrorrc' file:

$ vi ~/.gem/.mirrorrc

 In this file, you are going to specify where to grab the gems from and where to put them on your system:

---
- from: http://rubygems.org
  to: ~/path/to/put/the/files

You will need to make sure that you create the directory you specify in the "to:" portion.
After you save the .mirrorrc file and create the directory, you are ready to start mirroring:

$ gem mirror

This is going to take a while.  I ran it and it took several hours to download (and I have a fiber internet connection with 30Mb down).  So, once you start the download, you might as well go do something (watch a movie, read a book, hack on a Raspberry Pi, whatever you feel like.  Just monitor it once in a while to ensure it keeps going successfully.

Warning:  The download is many gigabytes and you are going to need some space for it.

That's it, you now have a mirror of Rubygems that you can reference in offline situations.  Stay tuned and I will put up a post on how to run a server that points to this mirror so that you can put it to good use.
 

Creating A Pypi Mirror

One of the things I have read a lot of people like to do, is to create a local pypi mirror.  That way, when the real pypi is on the fritz or you don't have internet access, you can still install modules and work on your pet projects.

I worked through a bunch of different modules for creating a pypi mirror, and most of them seemed to make a mirror of the most recent versions of modules.  Which would be fine, except I am anal and wanted as complete of a mirror as I could get. 

So, after playing around with a number of different modules I discovered Bandersnatch.  At first it looked promising (claiming that the mirror would be about 120 Gb).  Considering that the module documentation was probably written (and not necessarily updated) a couple of years ago, I could only imagine what that number is now. 

I followed the installation from the above link and installed the module (in a virtual environment) and got it running and let it run until completion (which was about 3-4 hours later).  I did a df of the directory and BLAM!, a little over 180Gb of moudules.  Just WOW!!  Now that is the mirror I was looking for. 

I still want to play with some of the other methods (as having a mirror of the most recent set of modules is also handy), but this is definitely what I was looking for.  Hopefully this information allows others to create their own Pypi mirror as well.  I would say quickly, but that will depend on the internet connection you are using.

I Have A New Domain!!!!

At long last, I have set up a new domain name.  The previous domain (www.parsed.co) ended up being too expensive to renew.  I now have a new domain name for the blog, something that goes better with the blogs name:   www.parsedcontent.com

I hope that all the readers that I had previously will return.  I will put this out there asap so that people are aware of the new domain name.  Thank you to all for your patience during this arduous transition.

raspi-config Missing On Kali Linux 2.0

A couple of months ago I purchased one of the shiny new RaspberryPi2's (you know, the one with the 900 Mhz quad core chip and the 1Gb of on-board ram) and have been playing with it.   Within the last couple of weeks, Kali Linux 2.0 was released.  Seeing as how I have this sweet RaspberryPi laying around, I figured, why not get Kali 2.0 running on it. 

So, for my birthday, my awesome wife got me a Class 10, 32Gb MicroSD card.  So I promptly loaded it with the freshly downloaded Kali Linux 2.0 image and booted it.  After logging in to the machine I did a 'df -h' and discovered that instead of the 32 Gb I expected to see (ok, so it would be 30Gb after reserve), I was only seeing around 7Gb of space.   I hadn't run into this quandry before as I had only had an 8gb card before. 

So, I did some googling and found that you can finagle the partition table with fdisk (which isn't installed by default, btw.  You will need to install the 'afflib-tools' package in order to get fdisk installed) and then reboot and resize the root partition.  Or, as a friend pointed out, you can simply run raspi-config and it will quickly (and quietly) do it for you. 

So I searched around and what do you know..... no raspi-config.  Apparently they don't see a need for this extremely useful utility on Kali, so you will need to install it yourself, which, after doing it, wasn't that awful.

First, download the latest version of raspi-config from  'http://archive.raspberrypi.org/debian/pool/main/r/raspi-config'.  Just search in there and either click to download it, or, if you are like me, copy the link and wget it. Note: Remember to download the .deb file as this is a debian distribution.

Next, you need to install the two prerequisites for raspi-config:

  # apt-get install triggerhappy lua5.1

After you have installed those two, you can simply change directories to wherever you downloaded raspi-config and issue the following command:

  # dpkg -i .deb

After that, you should be able to run raspi-config.  The first option is to resize the partition to reclaim space on the card.  That is what you want.  It will tell you to reboot.  Once you do, voila!, space reclaimed and your 'df -h' should show a ton more space (relatively that is).

parsed.co notice

It was nice to have the parsed.co domain name attached to my block, but unfortunately the domain is a bit pricier to renew than what I initially paid for it.  That said, please know that I have not renewed it and you will have to access my blog though blogger itself, until I can get a new domain attached.

My sincere apologies for this change.   Hopefully I will have a new URL on here in the near future.

Running Puppet For A Single Module

For the last 2 1/2 years I have been working with Puppet.  Sure, there is a lot of debate about puppet versus chef, but where I work we use puppet, so the debate is moot. 

I was recenlty adding a user account for someone that needed access to a couple of our legacy systems.  These systems are on a version of puppet code that we have not maintained for almost 2 years (ever since we re-worked our network and re-wrote our puppet code to be cleaner and more efficient).

Erring on the side of caution I checked and found that puppet hadn't been run on one of the servers for almost 2 years (which is scary).  But, I needed to get the person's account propogated out to the machines in question (as we only use puppet for account management.  We DO NOT manually create accounts).

So, I decided to run puppet JUST for the module in question.  This has the benefit of only applying the changes of the module you want and nothing else in the puppet ecosystem.

First thing to do was to run the command, but with the added '--noop'.  This will allow us to see what changes were going to be made, without actually applying them:

# puppet agent -vt --noop --tags=module

The '--tags=module' portion of that line allows you to specify which module you want to apply to the system during the puppet run.  The '--noop' tells puppet to run in what is essentially known as 'just kidding' mode.  This means it will run, but it won't apply anything.  Instead, it will tell you what it would have done.  This way, you can see what changes would be made, just in case there were some you didn't expect.  This is an excellent feature that can help you avoid any unexpected changes that can really cause things to go awry.

After that runs and you confirm that all is ok with what it would have done, simply run the command again, but without the --noop, and it will apply the changes.

sudo Vulnerability On OSx

I was perusing HackerNews and found an article mentioning that there is a vulnerability in sudo as Apple ships it.  The vulnerability is active because Apple ships its os with tty_tickets disabled.  What this means is that your logins to a machine (or terminal sessions as it were) are not treated as separate, but instead as linked. 

To demonstrate this, open two terminal windows and sudo to root.  You will be prompted for your password once, but not the second time.  To fix this, simply run 'visudo' (as root) and add the following line to the file.  ( I added a comment above it to say that it was a vulnerability fix:

# To turn off apple's giant privilege escalation vulnerability
Defaults tty_tickets

After you apply the fix, log out of your terminal sessions and open two new ones.  You should now have to provide your password on both of them.  

The Zen Of......

I was perusing around a couple of different new sites that I frequent and saw a post titled "The Zen Of DevOps".  Being in a DevOps position I decided to take a look.  What I read was pretty accurate with regards to the way one should be thinking with regards to DevOps.

Here is what the article said with regards to the Zen of DevOps:

Automated is better than manual.
 
Reusable is better than repeatable.
 
Multiple is better than single.
 
Single is better than unstable.
 
Continuous is better than planned.
 
Now is better than never.
 
Although never is often better than *right* now.
 
Special cases aren't special enough to be manual.
 
Collaboration rules over the rules.
 
In the face of ambiguity, refuse the temptation to guess.
 
There should be one-- and preferably only one --obvious way to automate.
 
If the automation is complex, it's a bad idea.
 
If the automation is simple, it may be a good idea.
 
Infrastructure as code is one honking great idea -- let's do more!

When planning out details during research for a project, these are certainly rules to design by. 
While we are on the topic of Zen, here is the popular "Zen of Python":

    Beautiful is better than ugly.
    Explicit is better than implicit.
    Simple is better than complex.
    Complex is better than complicated.
    Flat is better than nested.
    Sparse is better than dense.
    Readability counts.
    Special cases aren't special enough to break the rules.
    Although practicality beats purity.
    Errors should never pass silently.
    Unless explicitly silenced.
    In the face of ambiguity, refuse the temptation to guess.
    There should be one-- and preferably only one --obvious way to do it.
    Although that way may not be obvious at first unless you're Dutch.
    Now is better than never.
    Although never is often better than *right* now.
    If the implementation is hard to explain, it's a bad idea.
    If the implementation is easy to explain, it may be a good idea.
    Namespaces are one honking great idea -- let's do more of those!

I just thought I would share these.  Nothing technical, but certainly relevant to (at least) my every day.

Security should be concern #1 on the internet

I was recently reading hackernews when I came across a link to something called EveryoneAPI.  Supposedly a service that converts phone numbers into business intelligence.  So, out of curiosity I tried to visit the link, only to be presented with a blank page.  Odd.

So I went back to the hacker news page for it and visited the comments section.  On there were a bunch of people complaining about the same thing.  I almost laughed out loud when someone (apparently, supposedly someone that has something to do with EveryoneAPI) said that it had to be ad blocking software that people were using as an extension to their web browser and that they should "simply disable it" so that we could access their site.

One of the most important things to remember about online security is that if something was installed (by you or others) to keep you safe, the last thing you want to do is to disable it.  As we all know, there are some really nasty things out on the internet (virus', trojans, and even worse, ransomware).  The last thing you want is take advice such as this, which puts you in a position of being vulnerable to such things.  It is better to just take your mouse and close the window.

Your security on the internet is paramount.  This not only applies to you, but your computer as well.  Here are some guidelines that you really should follow:

1.  Make sure that you have a router between your computers and the internet.  The last thing you want to do is put your computer directly onto the internet for others to be able to get directly to.
2.  Make sure that you have the appropriate security setup on the router when installing it:

• Change the default administrator password
• Make sure that if is a wireless router, and you are using the wireless portion, that you turn on encryption at the strongest level (WPA2).
• When setting up the encryption, be sure to choose a very strong, unguessable password.  If you are like me and use products such as LastPas for credentials storage, the application has a built-in password generator.  Make sure to include all character sets, require every character type and set the password length to a high number (I try not to use anything less than 15 myself)

3.  Never, ever put any trust in a pop-up security alert.  Do not click them, just close the page and leave.
4.  Don't click on links in emails from people you do not know.  They are more than likely phishing.  Most financial institutions and credible businesses will not email you and ask you to provide them with your personal information.   Report any such emails as spam in your email system and move on.
5.  DO NOT open any email attachments sent to you from people you do not know.   This is also a very bad thing and could lead to your computer being infected or worse, taken over.
6.  Browsers such as Firefox and Chrome have extensions to prevent Phishing, called phishing blockers.  Pick one, install it and let it do its job.
7.   STOP or DO NOT use Internet Explorer (IE).  That browser is absolutely awful and you should stop using it immediately.  People who write virus' and other such malware love IE because of the ease with which it propagates their malicious code.  Firefox or Chrome are so much better and they are free.
8.  Use strong passwords anywhere you need to create an account (see note in #2 about creating strong passwords).
9.  Strong passwords don't make it easy to remember, so use something like LastPass (which is FREE) to remember the sites, logins and passowrds that you have.
10.  I know that some people will tell you to enable auto-updates on whatever type of system you are on, but I will just say to make sure you keep your software updated.  I do not use auto-updating myself as I am not always in a position to allow major updates to "just happen".
11.  If you are running on Windows, please be sure and use anti-virus software.  There are free options out there, just find one and use it.   As for Linux and OSx, I leave it up to you.

Ok, now that I have gone over some of the most prevalent security rules for you and your computer on the internet, go forth and surf safely and securely. 

Oh look.... my blog

Yes, its true.  I have completely neglected this blog over the past couple of months.  It is by no fault of my own, but instead I place the blame on the reason, "I was working".  Work has been quite busy, keeping me virtually chained, working on sprint stories. 

Don't get me wrong, I love my job, I am just giving a reason for the neglect that I have provided my readers.  The good news, as that all that hard work has provided me with some content which I plan on posting.  Hopefully I can get a post up here very soon so that I can quench your thirst for geek knowledge. 

Till then, keep coding!