Remembering Why

Years ago I was pretty heavily into Perl.  Such that every time I had to do something, or needed to automate something, my first thoughts were, "hmm, how would I do that in Perl?" or "Perl could do that!".  I would then fire up my editor, get something functionally working and be happy.   Don't get me wrong though, my love of Perl also came with its dislike of some things.  For instance, documentation always left questions in my mind as to "how did they get to that point", or "how do they assume that?".  And that led to me not being able to find the answers, and getting frustrated and moving to something else.  Even people I know knew perl wouldn't know the answers, which made things even more difficult and hard to swallow.   

I got to the point where I started to do coding in Python, as everyone around me at the time was using it, and well, peer pressure is a thing.  I ended up really liking it, and have done it since.  It was nice to have something so well documented, well supported and easy to use.  

That was a number of years ago (at least 10, to be honest).  Recently I got the bug to revisit perl, as I remembered the things I DID miss about it.  So, having recently started playing around with Jupyter notebook as part of my continued Python learning, I found Jupyter also supports other languages as part of kernel plugins.  I found the iperl plugin and decided to try to get it working.  Ugh.... Not only does the current iteration NOT function as others had noted (as they, I believe, were using earlier versions.  The posts were from several years ago), there are no current articles or posts about getting it working.  In fact, the documentation on the Devel::IPerl module is quite sparse.  The module also hasn't been touched in a long time.  

So, here I am again, realizing that not only do the same, similar issues plague me again, but nothing seems to have changed, at least in my 'fighting' to try and get this functioning.  Sure, I could just go and start coding in Perl, but a big part is asking, "Why?".  Why when there are other things that I already know, and are much more well supported and used. 

Perl still holds a placeholder due to things like its awesome regex's, and the things that it does do well.  But at this point, I just doing have the wanting to expend a ton of time on something that I probably won't use daily.  I have to plan my time better.  

Oh well, back to my regularly scheduled program.  

Happy Holidays, Everyone!

Pointing CPAN To Your Local Mirror

 I have been out of the Perl development arena for quite a while (about 10 years or so, to be brutally honest).  That's what happens when your career takes you in a different direction.  

Recently I re-found my wanting to code in Perl after reading some postings by a friend, which made me realize that I was kind of missing it.  

Anywho, that brings me to the reason for this article.  I am re-learning some things, and one of those things is the setting up of your own, local CPAN repository.   There are times when one may not have an internet connection, but wants to work, none-the-less.  This is where having a mirror of CPAN on your local machine is a good thing.  

I am not going to go into how to create the mirror, because there is quite the plethora of documentation and write-ups on that topic, but instead, I am going to write a quick bit on actually pointing to your local mirror. 

Now, it matters not where you have it, but for the sake of this article, let's say that you have the mirror located at /tmp/cpan_mirror  (yes, I am assuming that you are on Linux.  If you are on Windows, I hate to say it but you are on your own as I don't touch the thing).

The first thing you need to do is drop into the cpan cli.  You can do this simply by typing cpan at your command line and hitting enter.  You'll be dropped to the cpan prompt:

 cpan[?]>

Once there, you'll need to do the following things.  I will create a list of tasks first, and then put the commands after. 

• check the urllist and see what other mirrors are listed
• remove all other mirrors (saving the url's aside if you wish to reset them later)
• add your local repo path
• commit your changes
• reload cpan

Ok, so here are the commands:

$ cpan       (to drop into the cpan prompt)
cpan[?]> o conf urllist
cpan[?]> o conf urllist pop   <--- run this till any urls are out of the list
cpan[?]> o conf urllist "file:///tmp/cpan_mirror/"
cpan[?]> o conf commit
cpan[?]> reload cpan

After that, you should be able to, at the cpan prompt or using the cli, install modules.  You can test its pointing to your local repo by turning off your wifi and ensuring you cannot get to the internet.

To reset it back to internet based mirrors, just follow the same procedure, adding the mirror you like to use in place of your local path, and without the double quotes.

Enjoy!  And TMTOWTDI!!

 

 

Removing Unwanted Lines From A File

Intro

 This is a bit of a "back to basics" post.  I find it is good to revisit, now and then, as it keeps you sharp.  It also helps as a reminder when you find yourself needing to perform this task.  One never know when they need to remove lines from something like a log file. ( for say, hiding one's interactions on a server )

So, lets say you have a file ( we will call it logfile.txt) on a system, and you need to remove all the lines that contain the text foobar (I know, pretty standard, but you understand the usage).  I will preface this with the fact that this is being done on Linux.  And as with anything on Linux, there are a plethora of ways to do things.  So long as the end result is what you require, then the method was correct, even if there are quicker or more efficient ways.    These are just a couple of the quick ways that you could achieve this goal.  So without further ado, let's jump right in.

NOTE: Please keep in mind that all of these methods will achieve the same exact thing, just in different ways.

Method 1

So the first method is more my preferred method.  Why my preferred?  Well, because its a one-liner and it doesn't require me to do any moving of files to different names.  Its just quick and direct.  But also, you need to make sure you are absolutely positive that it is doing what you expect, as it is immediately effecting of your file and not reversible.

 

The Method:

$ sed -i '/sometext/d' logfile.txt

 In the above, the '-i' tells sed to edit the file in place (that's what makes this a more dangerous and immediately effecting version).  The /sometext/ is the text that you want to match on each line in the file.  (yes, the file will be read, line by line, matching against 'sometext' to check for a match).  The 'd' option says to delete that line if a match is found.  'logfile.txt' is the file to search in. 

This tends to work quick and is efficient.  It would help your situation to search the file, say with grep, first, and ensure of what you will be matching.  Caution is always a good thing to err on the side of, but that's just me. 

A Note Before Continuing:  The above method is the only immediate method I am presenting.  The other two methods I show, will involve the use of temporary files.  That said, they are the safer options, unless of course, you decide to script them, in which case, you make them more immediate.  Your choice.

Method 2

This method involves using grep and using the '-v' option, which will ignore the lines that match.  It will take the lines that do NOT match, and output them to the temporary file.  You will then take the temporary file, after grep runs over your file, and move it back to the original filename.  Or, you could take and name it something different, the option is yours.

 

The Method:

$ grep -v "sometext" logfile.txt > tempfile && mv tempfile logfile.txt

 

Again, you could move the tempfile to another filename.  Or, even better, after purging into the tempfile, rename the original file to a backup name, and then rename the tempfile to the original name.

 

 

Method 3

This is the last method I will cover here.  This involves using the awk utility.  Just as the others, it will do a match, but its method is to use a '!', which means DON'T match, which tells awk you want to match all lines that do not contain the matching text.  So any lines that do not have the 'sometext' matching text, will be output to the tempfile.  

 

The Method:

$ awk '!/sometext/' logfile.txt > tempfile && mv tempfile logfile.txt

 

And as noted previously, you can rename as you wish.  Either back to the original file name, or backing up the original name first, and then renaming.  Its totally up to you.  And also, totally scriptable.

 

Conclusion

I hope that this helps you manage this basic of tasks.  Enjoy!

 

 

Setting Up Your Own Remote Git Server

Introduction

 I don't know about anyone else, but I love having my GitHub / GitLab / BitBucket account, where I can store all of my code (whichever combination of those, or others you may use).  But, I find it handy to have my own local, self-hosted git server, where I can store my code, until I am at a point where I want it published up to one of those other sites.  Don't get me wrong, its not that I am not about sharing, but more about getting things to a usable point first.  Development takes time.   

Anywho, that is the 'gist' of this post..... what is involved in setting up your own, self-hosted git server, to store your code.  I could also note that this is super handy if you are also a bit paranoid.  Paranoia, to some degree, can be healthy.  Just don't let it get out of hand.  

My Setup

As I mentioned, I am running it on a Raspberry PI, but here are some further details on my setup:

• Raspberry Pi 3 B+
• A Raspberry Pi 3 mSata SSD Shield
• A 256 Gb mSata ssd stick
• A 16 Gb microSD card
• A 7" touch screen display (all hooked up with the pi so I don't need a separate monitor)

I am using a 256 Gb drive, as I want to have enough room for whatever projects I decide to work on.  Also, its expandable, so not limited to just that size.  Depending on how much you plan on working on, adjust your ssd stick size.  

Setting Up Your Git Server

Considerations

You will need to work out where you are going to run this system.  It could be a spare system (desktop or laptop).... it could be in the cloud somewhere....  or, like me, you could set it up on a Raspberry Pi on your local network.  I find this handy, as its portable, should I need to take it with me on a trip, and it also uses a minute amount of power to run.   No matter what, the first thing you'll need to do is work out where to run it, before you can go to the install and setup. 

Another consideration is, that if you are using an mSata ssd, you'll need to set it up, format it and get it setup so it mounts at boot.   

Install the necessary software

So the first thing you will need to do is install git on your system.  This varies from system to system, so I am not going to get into that, but just make sure that git is installed.

Setup The Git User

If there is not a git user setup after the software install (there wasn't on my system), then you will want to do the following:

sudo adduser git   (provide whatever password you want, but you have to set something)

sudo -u git -i

mkdir .ssh && chmod 700 .ssh

touch .ssh/authorized_keys && chmod 600 .ssh/authorized_keys

 

Add Developer SSH Key(s) To The Git User's authorized_keys File

For each developer that you have using your system, you are going to have to add their public SSH key to the git users's ~/.ssh/authorized_keys file.  If you copy their key up to the /tmp directory, then you can do something like this, as root:

cat /tmp/userkey.pub >> ~git/.ssh/authorized_keys

Setting Up A New Repository

For each new repository that you wish to host on the server, you'll  need to create an empty repository to host the code that developer(s) will push.  You'll need to have a base directory that will house all of your repositories.  For instance, I have my ssd drive mounted at /data on my raspberry pi, and have a directory in there called git.  So I host all my repositories at /data/git.

To setup and empty repository for your code, do the following (I will give examplease in /data/git, where I host mine.  Feel free to change to whatever you are using):

cd /data/git

mkdir projectName.git

cd projectName.git

git init --bare

You will see something similar to this output after the git command is run:

Initialized empty Git repository in /srv/git/project.git/

Developer Setup To Push To Your New Repository

Now that you have your repository created and ready to receive code, all you need to do is have your developer point at it and push their code.  Here is how to point your repository at your new repo:

git remote add origin git@gitserver:/data/git/projectName.git

After they set that up, they just need to run:

git push origin master

 

Securing Your Setup

After you have gone through the trouble of setting up all of this, you are probably going to want to secure it, especially if you are sharing it with others.

Change The Shell For git User

NOTE:  Please know that once you do this, you can no longer just switch to the git user.  Anything you do withing the git user home directory will need to be as the root user.  So ensure you are ready to commit to this

The first thing you need to do is change the shell that is used by the git user to be git-shell.  git-shell is a special shell that comes with git, that only allows the git user to act for git related activities, and does not allow normal account shell access.  This is needed, as developer's ssh keys are in the git users' authorized_keys file, and they would otherwise be able to connect as git user, to your system.  This will prevent that. 

To change the shell for git user, run this:

sudo chsh git -s $(which git-shell)

Remove Port-Forwarding Ability For git User

The second thing you need to do to secure the server, is to remove the ability of users to get port-forwarded access, which would give them access to any host that can connect to the git server.  To remove this ability, you need to prepend the following text before the ssh-rsa portion of each and every developer key in the authorized_keys file:

no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty

 

After performing both of the above steps for security purposes, users will still be able to use git commands, but will be unable to get any shell on the system (or other systems).  

Some Notes

After you put the security precautions above into place, you will not be able to become the git user in order to create the new repositories and such.  If this is YOUR server, and nobody else will be on it, then you don't need to use the security precautions above, as you most likely aren't going to pwn your own data.   To be honest, if you're going to share with others, it would most likely be via one of the aforementioned services and not your own, self-manage git servers.  But if you do set something like this up and share with others at some point, make sure you trust them, or put the measures into place.  

It also worthwhile to note that scripting the repo creation and its modifications is probably a good idea.  Once I get that created, I will share them in another post.  

You'll notice that I did not get into any GUI interfaces, like Gitea, Gitlab, etc.  The reason is, I just wanted to provide guidance on setting up a plain git server for your code.  As such, no gui is needed for that.  If you want one of those, and decide to use one, there are a plethora of tutorials out there for installing and setting them up.   

Thoughts On Ubuntu 22.04 (UPDATED)

 So here I am a few days ago, sitting at my computer, working on a project, and a pop-up appears asking me that Ubuntu 22.04 had been release, asking if I wanted to upgrade to it.  (Yes, I know that I can limit what I am notified of, I actually choose to know when the latest is available).  

I think, "Sure, why not." and clicked through the plethora of buttons to get it up upgrade (agreeing to the upgrade no less than 3 or 4 times). So my machine goes into upgrading and I decide to let it do its thing.  An hour or so later, I return and its at a login prompt, as it had rebooted.  Only, the login page is now dark themed.  Very nice!!!  The upgrade results were starting off on a good note.  

While its just a lot of packages that were upgraded, as well as the underlying software, I will say that there was an extremely annoying side-effect of this upgrade.  I have some projects that I have setup, including having done pip installs of packages and such for the software.  It seems that during the upgrade, all of those packages and settings went away.  Seriously?  Just *POOF*?  What the heck, Ubuntu?  Way to blow away my installs and configs.  

So, I had to spend some time re-setting up the stuff I regularly used (and some I was working on).  All of the items, by the way, were already using Python 3, so nothing except a newer version being installed.  

I just wanted to share this annoyance, in case you are planning on upgrading.  You will need to plan on some extra time, re-setting up some things once the upgrade completes.

UPDATE:  After some research, and noting issues seen on other machines (non-ubuntu) and work systems, this seems to be something related to a python3 upgrade.  On my work Mac, it seems that Python3 upgraded, and put itself in a different location, but did not take on what was previously installed.  It caused a ton of confusion in my system.  There were two different versions of pip3 and two different versions of python3.  It took using 'which -a' to figure this out and zero in on the issue.  In the end, I had to use an alias for each on my Mac to set things right.  

Here on my Ubuntu system though, the python3 upgrade simply wiped out everything that was installed with the previous python3 installation, that wasn't part of the standard library.  Thanks for that.  Way to seriously cause broken software that is used constantly.

So Long.... Youtube-DL

 A couple of years ago, I decided to write a quick script called ytdl, short for 'youtube downloader'.  I wrote it as a wrapper for the youtube-dl application, as I was tired of always trying to remember the options that I used when downloading either a video or a song (in audio format only) from youtube. 

Recently, I revisited the application as I needed to grab a couple of things off of youtube, and found that when running it, the downloads were CRAZY slow!  By CRAZY slow, I mean in the area of only getting between 25k and 50k max as a download speed.  I was looking at 1-2 hours to download each of the videos I was wanting to grab.  Considering in the past it would only have taken me a few minutes to download the videos I wanted (in total), I decided to use my Google Foo and see what was up. 

Sure enough, it appears that:

1. youtube.com was throttling the youtube-dl application.  This was confirmed by the plethora of complaints from the community that uses(used) it.  

2. I discovered that youtube-dl was no longer being maintained, and that the last version (from december of last year), was the final version. 

Well, that's not good.  But, what is good, is that there was a replacement...... yt-dlp!!!  Excellent, a successor.  So, I proceeded to download yt-dlp and incorporate it into my script, utterly replacing the old, useless youtube-dl.  

After some tweaking to get the downloaded files the way I like them, I can happily say that I am now seeing download speeds in the megabytes!!!  (7-13Mb, just on the dl's I did today).  

So, I have updated all the download types in the script and pushed up a new version.  If you haven't played with it, please feel free to give it a go.  And, if you have any download sites that you'd like added to the script (that are supported by it), please feel free to open an issue for an improvement.

Here is a link to my ytdl project.  Enjoy!

Removing Tag(s) From A Git Repository

 If you work on a team that is constantly developing their code, then you are probably dealing with at least one repository that gets periodically tagged (such as when a code release is performed).  If that is the case, then you have probably (or not) had to deal with setting the tags that get applied to the repository.  

If so, then you would be able to see the tags on a repository with the following command:

   $ git tag

If you are looking for a tag, but you only know part of it, you can certainly use grep to weed through the potentially large list of tags that will be presented to you.  Now, if you happen to set a tag incorrectly, or apply it to the wrong repository (or branch of that repository, for instance), you are not stuck.  You can remove it.  Here are the steps:

1. First, ensure you are in the repository in question and in the correct branch, then do a 'git pull' to ensure you are at the latest change
2. Run 'git tag | grep <tag>' to get the tag name you need to remove
3. Run 'git tag -d <tag_name>'        <--- This will remove the tag in your local repository
4. Run 'git push --delete origin <tag_name>'    <--- This will remove the tag from the remote repo
5. Run 'git pull'

From here, you will need to run the 'git tag' command in step 2 and verify that the tag has been removed locally.  You can also then go up to GitHub, Gitlab, etc, and check the repository and ensure that the tag has been removed there as well.   

Read Medium Articles For Free

 There is nothing more annoying to someone who is just trying to read about something they are trying to learn about, than to have a site say, "Hey, you've reached your 'free' limit!".  That is exactly what medium.com does with its articles.  You need to log in to read further (so it says), and then, when you do, they STILL won't let you go any further until you pay for the right to read their articles.  Call me cheap, but I believe in the freedom of information.  

So, a little Google-foo and I discovered that there is a workaround for this that works in Chrome/Brave:

- Load up the article you want to read

- In the URL bar at the top, click the little padlock (the one that indicates that the site is secured)

- In there, it will tell you how many cookies there are.  Click on "Cookies".  It will bring up a window similar to this:

- Click on 'medium.com' and click 'Block' down below, and then Done.

Chrome/Brave will then ask you to reload the page.  Once you do, you should be able to read the article without the annoying message telling you that you haven't paid them for the privilege yet.  

Enjoy! 

**UPDATE:  I know its been quite a while since I posted this, but it is worth noting that this fix is truly only a temporary fix.  I tend to browse using a VPN, and find that the fix only works for a little while.   Once it runs out, I just switch to a different vpn location and voila, I can browse a little while longer.  Its not perfect, but the Scottish in me doesn't want to pay for Medium at the moment.  

Also, if you have this cookie block in place and you try to sign up for membership on Medium, the link they send you will hang indefinitely.  You'll have to remove the block, retry the link, and things should work.  Just noting.  

Installing Dependencies for dpkg Software Installation

Let's say you download any of the plethora of '.deb' packages that exist out there, and want to install it on your Ubuntu system (or the like).  You would use something similar to the following to do that installation:

    sudo dpkg -i imager_1.7.2_amd64.deb

Now, let's say you run that, but are then presented with output stating that there are a bunch of unmet dependencies.  Disconcerting, sure, but its not the end of the world, for sure.  For example:

$ sudo dpkg -i imager_1.7.2_amd64.deb 

Selecting previously unselected package rpi-imager.

(Reading database ... 195449 files and directories currently installed.)

Preparing to unpack imager_1.7.2_amd64.deb ...

Unpacking rpi-imager (1.7.2) ...

dpkg: dependency problems prevent configuration of rpi-imager:

 rpi-imager depends on libqt5qml5 (>= 5.10.0); however:

  Package libqt5qml5 is not installed.

 rpi-imager depends on qml-module-qtquick2; however:

  Package qml-module-qtquick2 is not installed.

 rpi-imager depends on qml-module-qtquick-controls2; however:

  Package qml-module-qtquick-controls2 is not installed.

 rpi-imager depends on qml-module-qtquick-layouts; however:

  Package qml-module-qtquick-layouts is not installed.

 rpi-imager depends on qml-module-qtquick-templates2; however:

  Package qml-module-qtquick-templates2 is not installed.

 rpi-imager depends on qml-module-qtquick-window2; however:

  Package qml-module-qtquick-window2 is not installed.

 rpi-imager depends on qml-module-qtgraphicaleffects; however:

  Package qml-module-qtgraphicaleffects is not installed.

dpkg: error processing package rpi-imager (--install):

 dependency problems - leaving unconfigured

Processing triggers for mailcap (3.69ubuntu1) ...

Processing triggers for gnome-menus (3.36.0-1ubuntu1) ...

Processing triggers for desktop-file-utils (0.26-1ubuntu2) ...

Processing triggers for hicolor-icon-theme (0.17-2) ...

Errors were encountered while processing:

 rpi-imager

I know that is a lot of output, sorry about that.  But I left it that way for effect and to give a full example.  Now, what you will need to do in this case, is run the following:

        sudo apt -f install -y

That will take and install any dependencies that were found during the previous software installation attempt, and install them.  After that is done, you can re-run the dpkg command to install your software, and it "should" just work and install your software.  ( I say should in quotes, as nothing is guaranteed. )

PyBlueprint update

 Since I have started this project, I have had an issue where the "checks" option, to verify that everything was correctly installed, has been failing to detect virtualenvwrapper as installed, even though it is in the 'pip list' output.  

After a bunch of digging and research, I have discovered a way to do the checks that seems to be working without issue.  I have installed it on a couple of different systems, both Linux and Mac, and things seem to be functional.  That is a breath of fresh air from me, as I have hated the error happening (and subsequently being ignored by me as I knew it was installed). 

So what is next?  I plan on adding support for more languages (shell, perl, and fix the ruby implementation to be more automated).  I also want to give an option for setting up directories for code that has been checked out from a repo, so that it can be added and setup more easily than trying to remember the commands to add it.  

We shall see how those options come along and hopefully I can advance this software's language support.  If anyone has any requests, by all means, please leave a comment and I will see about adding to my list of items.  Heck, the best way would be to open an enhancement in the repo.  You can do this by creating an issue and title it as an enhancement.  

Happy Geeking!

Firefox and Your Bookmarks

 Recovering your bookmarks from firefox

I am writing this as an alternate method of retrieving your bookmarks, other than using Firefox itself.  I had an issue where the main drive in one of my servers decided it didn't want to boot anymore.  I could still access the data on it via an external cable, but the system became unbootable.  

In searching, I found that Firefox stores its book marks in a file called 'places.sqlite'.  That's correct, it is an sqlite database, which means bookmarks are not the only things in this file.  The file is stored in ~/.mozilla/firefox/<blah>.default-release.  Now, i am on Firefox 82 and my full path is:  ~/.mozilla/firefox/6pcydzru.default-release.  

Now, the first thing to do is to go to that directory and copy that file to some place else (don't move it, copy it).

    cp places.sqlite ~   <- Yes, I simply copied it to my home directory

The reason for this is that Firefox locks this file and working around that lock is as simple as copying it to another location.  And you can keep the same name, if you wish, its not an issue.

Now, if you were to load up that file into sqlite3 and take a look, you would see that the bookmarks are certainly there, in a table called moz_bookmarks, but the urls are hashed and not readable.  Here is what it looks like when I loaded it up and took a look:

$ sqlite3 places.sqlite
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> .tables
moz_anno_attributes    moz_historyvisits      moz_meta             
moz_annos              moz_inputhistory       moz_origins          
moz_bookmarks          moz_items_annos        moz_places           
moz_bookmarks_deleted  moz_keywords         
sqlite> select * from moz_bookmarks;
1|2||0|0||||1602647046341000|1604087203227000|root________|1|1
2|2||1|0|menu|||1602647046341000|1602647046960000|menu________|1|3
3|2||1|1|toolbar|||1602647046341000|1602647046988000|toolbar_____|1|3
4|2||1|2|tags|||1602647046341000|1602647046341000|tags________|1|1
5|2||1|3|unfiled|||1602647046341000|1604087203227000|unfiled_____|1|8
6|2||1|4|mobile|||1602647046545000|1602647046943000|mobile______|1|2
7|2||2|0|Mozilla Firefox|||1602647046960000|1602647046960000|hk7fo0BVoSNm|0|1
8|1|3|7|0|Help and Tutorials|||1602647046960000|1602647046960000|mtMmQ-E9oI58|0|1
9|1|4|7|1|Customize Firefox|||1602647046960000|1602647046960000|DrTu2wYZmIj-|0|1
10|1|5|7|2|Get Involved|||1602647046960000|1602647046960000|-sWrRJ5NeO8R|0|1
11|1|6|7|3|About Us|||1602647046960000|1602647046960000|LCI5sKvqimve|0|1
12|2||2|1|Ubuntu and Free Software links|||1602647046960000|1602647046960000|wvi_2QDOZLm9|0|1
13|1|7|12|0|Ubuntu|||1602647046960000|1602647046960000|Zfpq8U7qHTzz|0|1
14|1|8|12|1|Ubuntu Wiki (community-edited website)|||1602647046960000|1602647046960000|6A8e4nK1328m|0|1

This isn't really usable, by any stretch, at the moment.  But a little googling, and I found a site called purposeful.co.uk, whom posted a nice shell script which would do the job for you.  
The link to this is: http://www.purposeful.co.uk/software/places2bookmarks/

The script is on there for you to copy to your system, I am not going to repost it here.  But, for readability, I will say that I modified a single line in the script.  Here is the modified line:

echo "$3  <A HREF=\"$url\">$title</A><br />"

As you can see, I added a '<br />' so each link would be on its own line, making things much nicer, and one entry per line.

Once you have the script in place (and the modification, if you so choose), you can run it like this: 

 ./places2bookmarks.sh ./places.sqlite > bookmarks.html

You will need to ensure that you replace the './' with the correct path to the files, if that is incorrect.  For me, I had them in the same directory

This doesn't seem huge thing, and sure, you could use the gui, but that is not me.  I prefer to do things on the command line and figure them out. Besides, I had fun doing it and learning about it, and thought I would share with everyone in case anyone else found it fun and fascinating.

Enjoy!

Untracking files in git

In one of my current projects, pyblueprint-py3, I have a config file.  This file gets edited locally after cloning the repository, and those changes get seen by git.  That is a problem.  This is a file that is distributed with default values that the user edits, and git does not need to know about these changes.  But, how do we tell git not to care about changes to a file?  Good question, and here is the answer.

For your config file (or whatever file it is that you want to prevent git from tracking), create it in your repository, and then issue the following command:

    git update-index --skip-worktree file_name

The file_name should include the path if it isn't in the current directory (ie: its in a subdirectory).

If the file that you don't want to track is already on git's radar, then you can do the following:

    git rm 
    git rm -r --cached 
    git commit -m'- Cleaning up and removing file to be ignored'
    git push

Then, add the file and edit the conf file with the generic placeholder and do this:

    git add 

    git update-index --skip-worktree 
    git commit -m'- Committing new, untracked file'
    git push

One thing to remember, is that when someone clones your repository, before they do anything, they will need to reissue the 'git update-index' command above for the specific file you untracked.  That should be noted in your documentation.

pyblueprint-py3 Now Exists

What a crazy 3/4 of a year its been.  The last company I worked for kept going down hill for the employees with a ton of management change-over, which was in no way good.  Eventually, things had to change and I needed to get back to a happy place.  At the turn of the year I started a new job, and BOY, am I happier.  While I am still ramping up at my new place of employment, I am certainly having a lot more fun here than I did the last couple years at my old job.  Considering what is going on over there, I do not envy my old colleagues.

Enough about the craziness.  I recently sat down and took the time to convert the pyblueprint project over to using Python3.  Not only that, there is a new option for specifying a language.  It defaults to python, but it will create a directory structure for whatever language you choose.  But, python and Ruby have support for the virtualenviroment that this project sets up (as does Go and NodeJS, but I haven't yet put together docs and such for that.  That is coming though.  I migrated the project to a new name, call pyblueprint-py3.  Feel free to download it and play with it.  Hopefully it works for you.  If there are issues, please open a ticket in gitlab under the project.  Also, I am open to feature requests, should anyone have any. 

Hopefully I can get more additions on to this project in the near future. 

A new home for my projects and a project update

New Home

I know that there is always an expectation that, if you are a developer of some sort, you store your code into Github (at least that is the first one that comes to people's mind).  I have a Github acct, but, because their private repos are not free, I haven't truly used them for storing my repos.  And considering that Microsoft now owns them, I really won't be using them for my code.

Up until now, I have relied on Bitbucket (currently owned by Atlassian), as it offers unlimited free private and public repos.  We have both Bitbucket and Gitlab at my work (local installs for both).  After all of the increased fanfare for Gitlab (after the Microsoft purchase of Github) and my exposuer to it at work, I have made the switch to store my repos there as well.  Unless they change their policies, I will probably be with them quite a while.  

Project Update:

I did want to say that I have migrated my 'pyblueprint' project over to Gitlab.  Thankfully Gitlab makes the migration to them painless, and it was just a click of a couple of buttons to get the repo moved over.  The 'hard part', if you will (although not really), was fixing my git remote to point to the new repository location.  

I do want to say that I have added support to the project for creating a repository up in Gitlab.  Seeing as how I am now using it, why not.  :)  While testing out the application to ensure everything was still working afterwards, I discovered that my checks were not working.  Why?  Because the maintainers of 'pip' decided that, as of pip 10.0.0, they no longer support pip bing used programatically.  If you drop into IDLE, do an 'import pip' and then do a 'dir(pip)', you will now notice that there are no methods associated (aside from the underscore methods).  This completely broke my checks as I had relied on that useful bit.  No worries though, the checks are fixed and functioning as expected.  

Also, if anyone is using this application (don't know if anyone is) and they would like support for an alternative code repository, please feel free to open an enhancement ticket and I will see about adding support for it.  

That's it for now, happy coding!

Setting up MELPA for Emacs packages

In my journey to learn emacs, one of the first things that I wanted to do was change the theme so that it was more pleasing to the eye.  Anyone who uses a tool enough knows that if you can personalize it for you, you personalize it. 

So in researching themes ( see the emacs themes site ), I kept seeing references to MELPA, which is an emacs package manager.  Sweet!  Unfortunately, my quest to install the theme I wanted was met with the fact that it is NOT configured to use MELPA by default.  Thus, the reason for this write-up. 

In order to configure emacs to see the MELPA packages, you need to edit your .emacs file in your home directory.  If you have at least opened emacs, this file should exist.  If not, just create it and open it for editing.

To correctly enable MELPA, you will need the following as the first section of the file:

;; load emacs 24's package system. Add MELPA repository.
(when (>= emacs-major-version 24)
  (require 'package)
  (add-to-list
   'package-archives
   ;; '("melpa" . "http://stable.melpa.org/packages/") ; many packages won't show if using stable
   '("melpa" . "http://melpa.milkbox.net/packages/")
   t))

 After you save the .emacs file, you will need to restart emacs if it is already running.  If not, start it up.  Once running, you will need to his what in the emacs world is referred to as M-x.  On a mac, that is Option + the 'x' key.  I don't run Windows, so I really couldn't begin to tell you what it is on there. 

That will drop you into the M-x mini buffer.  There, type 'list-packages'.  In the main window, a columned list should appear.  The 4th column is 'archive' and should have packages that say 'melpa' (as well as 'gnu').  If so, you are good to install the theme you found. 

To install a melpa package, drop back into the M-x mini buffer and type 'package-install'.  This will change the prompt to "Install package: " which will wait for a package name.  The nice thing is that tab-complete works in emacs.  So start typing a package name and hit tab, and a list will appear in the are right above the mini buffer. Once you complete installing the packages you need, go ahead and restart emacs. 

I certainly hope this makes your emacs package installation easier.  :)

Straddling the line between editors

Since I started in Unix/Linux about 20 years ago, I have been a staunch user and supporter of vi.  I have known plenty of people that were emacs aficionados, but haven't really taken the time to learn it to the point of being proficient.  I did look at it a few years ago after starting my current job, but with all of the new things I was learning, I forwent it in favor of the things I actually needed to learn at my job.

Now, 5 1/2 year later, I have decided that I am at a point where I finally have time to learn it.  I know what vi lovers are thinking, "hell no!", and I get that.  I was like that for years.  But, after getting to work for the last 5 1/2 years with a friend that I have now known for 16 years, I have seen him operate pretty seemlessly in emacs, without needing to go to the shell that often.  I have seen some pretty cool features that nicely streamlined his workflow.

Its never easy picking up a new tool, that's for sure, especially when you like the features of your current tool, but sometimes you have to make concessions to go farther.  All that said, don't be surprised if you see emacs related posts here, as I am wanting to share what I am learning.

In closing I will provide the two initial links that I am using to start my journey down the emacs path:

Absolute Beginner's Guide To Emacs
Practical Emacs Tutorial


Hopefully, if you decide to also venture down this road, those will help you as well.  I will say that so far, the Absolute Beginner's Guide has been awesome!!

Setup WiFi to auto-connect regardless of logged in state (UPDATED)

There are plenty of times where I would like to start up my desktop machine (running Ubuntu 16.04 LTS) and just have it boot and me log into it via SSH so that I can get some work done.  Well, in order to achieve that wish, you need to have your wifi connection auto-connnect at boot time.

To do this, there are some things you need to do:

1.  Go to your wifi connection in the upper right of your screen and click on "Edit Connections".
2.  Once in there, you need to click on your connected wifi ssid and select "Edit".
3.  Go to the "General" tab and select "All users may connect to this network".  (without this, the rest of this is moot and it will not work)

Ok, that is the only gui portion of this setup.  Next:

4.  Create the file /etc/wpa_supplicant.conf and add the content that follows:

ctrl_interface=DIR=/var/run/wpa_supplicant group=wheel
network={
  ssid="your_connection_ssid"
  scan_ssid=1
  key_mgmt=WPA-PSK
  psk="password"
}

In the above code, you will input your ssid that you would like to connect to and also enter your password for the connection.  If you are using an encryption method other than WPA/WPA2, then you will need to reference the man page for wpa_supplicant.conf.  There are plenty of examples to help you out. 

After you have that file all setup, save it.

5.  Issue the following command on the command line, as root or with sudo:

wpa_supplicant -B -i wlp4s0 -c /etc/wpa_supplicant.conf

You should see a message that says something to the effect that wpa_supplicant initialization was successful.

After you have completed those steps, go ahead and reboot the machine and watch the login screen.  If all is correct and right with the configuration, you should see the wifi connection show as connected.  You can test this by ssh'ing to the machine.  Enjoy!


UPDATE:  I took the opportunity to reinstall my Linux system to go from Ubuntu 16.04 LTS to Ubuntu 18.04.  I know its a bit extreme to do a full re-install, but:

1.  An distribution upgrade wasn't yet available
2.  There wasn't anything I couldn't re-setup on the machine
3.  Ubuntu moved away from Unity to the Gnome desktop, so I wanted everything clean and fresh.

I will say this though, auto-connection of the wifi (seeing as how you tell it how to connect during installation) just WORKS with Ubuntu 18.04.  You just have to make sure to install the openssh-server package.

How to list available versions of a Pypi package

While working on requirements that someone had for a pypi package, I found myself needing to list out the versions of the package that were available.  I didn't want to have to go to the pypi website, navigate to the package and then figure out how to see the available versions.  So, to the Googles I went.

In order to install a specific version of a package, you would run:

pip install =1.0.0

in order to install version 1.0.0 of said package.  But, if you provide nonsense as the option, like so:

pip install =blah

then what you get back is an error listing all of the available versions for that package:

> pip install mod_wsgi==blah
Collecting mod_wsgi==blah
Could not find a version that satisfies the requirement mod_wsgi==blah (from versions: 4.1.0, 4.1.1, 4.1.2, 4.1.3, 4.2.0, 4.2.1, 4.2.2, 4.2.3, 4.2.4, 4.2.5, 4.2.6, 4.2.7, 4.2.8, 4.3.0, 4.3.1, 4.3.2, 4.4.0, 4.4.1, 4.4.2, 4.4.3, 4.4.4, 4.4.5, 4.4.6, 4.4.7, 4.4.8, 4.4.9, 4.4.10, 4.4.11, 4.4.12, 4.4.13, 4.4.14, 4.4.15, 4.4.16, 4.4.17, 4.4.18, 4.4.19, 4.4.20, 4.4.21, 4.4.22, 4.4.23, 4.5.0, 4.5.1, 4.5.2, 4.5.3, 4.5.4, 4.5.5, 4.5.6, 4.5.7, 4.5.8, 4.5.9, 4.5.10, 4.5.11, 4.5.12, 4.5.13, 4.5.14, 4.5.15, 4.5.16, 4.5.17, 4.5.18, 4.5.19, 4.5.20, 4.5.21, 4.5.22, 4.5.23, 4.5.24, 4.6.0, 4.6.1, 4.6.2, 4.6.3, 4.6.4)
No matching distribution found for mod_wsgi==blah

Good to know its that easy.

python pip throwing SSL: TLSV1 alert (SOLUTION)

I recently ran into an issue with Python pip, where I attempted to run a 'pip install' of a module and was presented with the following error:

> pip install pymongo
Collecting pymongo
Could not fetch URL https://pypi.python.org/simple/pymongo/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) - skipping
Could not find a version that satisfies the requirement pymongo (from versions: )
No matching distribution found for pymongo

No matter what I did (install, update and even just search the pip repository) I was presented with this same error.  It was certainly a hard, fast stop to whatever I was attempting to do.

But, after some googling and reading of many people's accounts of their trials and tribulations attempting to solve it, I finally found a solution that works.  I happened upon the this posting on StackOverflow, which leads you to the fact that you have to upgrade pip.  Unfortunately, attempting to upgrade it with itself will fail as above.  But, you should instead run the following:

curl https://bootstrap.pypa.io/get-pip.py | python

That upgraded pip directly from the website using its install script.  It is a work around, yes, but one that seems to have worked for me (and apparently others).  Hopefully this solves your issue(s) as well.

**UPDATE: You will need to run the curl command in each virtualenv environment that you have, as each one will have its own installation of python.

'_remove_dead_weakref' error after upgrading to Ubuntu 17.10

Well, after finally getting my Flask app running and returning data, I upgraded my system to Ubuntu 17.10 (from 17.04).  Today, I tried to start my flask app and was greeted with a bunch of Python error output and the following error:

ImportError: cannot import name '_remove_dead_weakref'

I was baffled as the app worked the day before.  So after some digging around the Googles, I found that the issue was caused by my upgrade to 17.10.

The fix....... delete your virtual environment and rebuild it.  I am not saying delete your project directory, I am saying delete your envs directory (which is where the executables for the virtual environment are.  Some people put it in the same directory.  If you are like me, you have a central location where you keep them. 

You will need to remove that entire directory, rebuild it and then re-run the virtualenv command to create the virtualized environment and then install any modules that you had previously installed (as they get stored in the directory structure you just deleted.  After that, your app and its parts should work fine.