Thursday, August 26, 2010

Customizing Unix Shell

A few tools that really make working in a unix shell more enjoyable.

Byobu - Replaces "screen" for a screen with a more useful display

Change .bashrc file to include the following

# Shell colors
export LS_OPTIONS='--color=auto'
eval "`dircolors`"
alias ls='ls $LS_OPTIONS'

alias screen='byobu'


Create .vimrc file to make working in VIM easier

" URL: http://vim.wikia.com/wiki/Example_vimrc
" Authors: http://vim.wikia.com/wiki/Vim_on_Freenode
" Description: A minimal, but feature rich, example .vimrc. If you are a
" newbie, basing your first .vimrc on this file is a good choice.
" If you're a more advanced user, building your own .vimrc based
" on this file is still a good idea.

"------------------------------------------------------------
" Features {{{1
"
" These options and commands enable some very useful features in Vim, that
" no user should have to live without.

" Set 'nocompatible' to ward off unexpected things that your distro might
" have made, as well as sanely reset options when re-sourcing .vimrc
set nocompatible

" Attempt to determine the type of a file based on its name and possibly its
" contents. Use this to allow intelligent auto-indenting for each filetype,
" and for plugins that are filetype specific.
filetype indent plugin on

" Enable syntax highlighting
syntax on

" Set font according to system
set gfn=Monospace\ 10
set shell=/bin/bash

if has("gui_running")
set guioptions-=T
set t_Co=256
set background=dark
colorscheme peaksea
set nonu
else
colorscheme zellner
set background=dark

set nonu
endif

set encoding=utf8
try
lang en_US
catch
endtry

set ffs=unix,dos,mac "Default file types

"------------------------------------------------------------
" Must have options {{{1
"
" These are highly recommended options.

" One of the most important options to activate. Allows you to switch from an
" unsaved buffer without saving it first. Also allows you to keep an undo
" history for multiple files. Vim will complain if you try to quit without
" saving, and swap files will keep you safe if your computer crashes.
set hidden

" Better command-line completion
set wildmenu

" Show partial commands in the last line of the screen
set showcmd

" Highlight searches (use <C-L> to temporarily turn off highlighting; see the
" mapping of <C-L> below)
set hlsearch

" Modelines have historically been a source of security vulnerabilities. As
" such, it may be a good idea to disable them and use the securemodelines
" script, <http://www.vim.org/scripts/script.php?script_id=1876>.
" set nomodeline


"------------------------------------------------------------
" Usability options {{{1
"
" These are options that users frequently set in their .vimrc. Some of them
" change Vim's behaviour in ways which deviate from the true Vi way, but
" which are considered to add usability. Which, if any, of these options to
" use is very much a personal preference, but they are harmless.

" Use case insensitive search, except when using capital letters
set ignorecase
set smartcase

" Allow backspacing over autoindent, line breaks and start of insert action
set backspace=indent,eol,start

" When opening a new line and no filetype-specific indenting is enabled, keep
" the same indent as the line you're currently on. Useful for READMEs, etc.
set autoindent

" Stop certain movements from always going to the first character of a line.
" While this behaviour deviates from that of Vi, it does what most users
" coming from other editors would expect.
set nostartofline

" Display the cursor position on the last line of the screen or in the status
" line of a window
set ruler

" Always display the status line, even if only one window is displayed
set laststatus=2

" Instead of failing a command because of unsaved changes, instead raise a
" dialogue asking if you wish to save changed files.
set confirm

" No sound on errors
set novisualbell
set noerrorbells
set t_vb=
set tm=500

" Enable use of the mouse for all modes
set mouse=v

" Set the command window height to 2 lines, to avoid many cases of having to
" "press <Enter> to continue"
set cmdheight=2

" Display line numbers on the left
" set number

" Quickly time out on keycodes, but never time out on mappings
set notimeout ttimeout ttimeoutlen=200

" Use <F11> to toggle between 'paste' and 'nopaste'
set pastetoggle=<F11>


"------------------------------------------------------------
" Indentation options {{{1
"
" Indentation settings according to personal preference.

" Indentation settings for using 2 spaces instead of tabs.
" Do not change 'tabstop' from its default value of 8 with this setup.
set shiftwidth=2
set softtabstop=2
set expandtab

" Indentation settings for using hard tabs for indent. Display tabs as
" two characters wide.
"set shiftwidth=2
"set tabstop=2


"------------------------------------------------------------
" Mappings {{{1
"
" Useful mappings

" Map Y to act like D and C, i.e. to yank until EOL, rather than act as yy,
" which is the default
map Y y$

" Map <C-L> (redraw screen) to also turn off search highlighting until the
" next search
nnoremap <C-L> :nohl<CR><C-L>


"------------------------------------------------------------

Monday, April 26, 2010

SQL to Search All Tables for Column Name

declare @search_string  varchar(50)

set @search_string = ''

SELECT table_name=sysobjects.name,
column_name=syscolumns.name,
datatype=systypes.name,
length=syscolumns.length
FROM sysobjects
JOIN syscolumns ON sysobjects.id = syscolumns.id
JOIN systypes ON syscolumns.xtype=systypes.xtype
WHERE sysobjects.xtype='U'
and syscolumns.name LIKE '%' + @search_string + '%'
ORDER BY sysobjects.name,syscolumns.colid


Credit to: http://codesnippets.joyent.com/posts/show/337

Thursday, April 22, 2010

SQL Query to Search All Tables for an INT

Here is a really good query I want to remember for searching all tables in a database for a specific integer.
declare  @sql varchar(8000), @tbl varchar(255), @col varchar(255), @data  varchar(50)

set @data = '249753'

declare cur_tbl cursor for
select a.name, b.name from sysobjects a, syscolumns b, systypes c where a.id = b.id and a.type = 'U' and c.xtype = b.xtype and c.name in ( 'int' )
open cur_tbl
fetch next from cur_tbl into @tbl, @col
while @@fetch_status = 0
begin
set @sql = '
if exists (select * from [' + @tbl + '] where convert( varchar(255), [' + @col + '] ) = ''' + @data + ''')
select tbl=''' + @tbl + ''', col=''' + @col + ''', [' + @col + '], * from [' + @tbl + '] where convert( varchar(255), [' + @col + '] ) = ''' + @data + '''
'
exec(@sql)

fetch next from cur_tbl into @tbl, @col
end
close cur_tbl
deallocate cur_tbl


Or to search for a string
declare  @sql varchar(8000), @tbl varchar(255), @col varchar(255), @data  varchar(50)

set @data = 'string you are looking for'

declare cur_tbl cursor for
select a.name, b.name from sysobjects a, syscolumns b, systypes c where a.id = b.id and a.type = 'U' and c.xtype = b.xtype and c.name in ( 'varchar', 'nvarchar', 'text', 'ntext' )
open cur_tbl
fetch next from cur_tbl into @tbl, @col
while @@fetch_status = 0
begin
set @sql = '
if exists (select * from [' + @tbl + '] where convert( varchar(255), [' + @col + '] ) like ''%' + @data + '%'')
select tbl=''' + @tbl + ''', col=''' + @col + ''', [' + @col + '], * from [' + @tbl + '] where convert( varchar(255), [' + @col + '] ) like ''%' + @data + '%''
'
exec(@sql)

fetch next from cur_tbl into @tbl, @col
end
close cur_tbl
deallocate cur_tbl


Credit to: http://www.experts-exchange.com/Microsoft/Development/MS-SQL-Server/Q_24142162.html

Monday, March 8, 2010

AV2010 has gotten worse!

It was preventing me from installing Malwarebytes, and everytime I would run any program the virus would be started. I had to use the following registry fix:

Windows Registry Editor Version 5.00

[-HKEY_CURRENT_USER\Software\Classes\.exe]
[-HKEY_CURRENT_USER\Software\Classes\secfile]
[-HKEY_CLASSES_ROOT\secfile]
[-HKEY_CLASSES_ROOT\.exe\shell\open\command]

[HKEY_CLASSES_ROOT\.exe]
@=”exefile”
“Content Type”=”application/x-msdownload”

Then save as fix.reg and run it. After restarting the PC I was able to install Malwarebytes Anti-malware and remove the viruses.

Thanks to Kwisatz - http://www.virusremovalguru.com/?p=5528#comment-34676

Monday, January 25, 2010

Steps to creating a EBS image from an AMI (Windows)

Download Amazon's EC2 API Tools to C:\ec2-api-tools

# Run the following on a remote machine with Amazon's EC2 API Tools installed
# Adjust these variables to match your own system
set EC2_HOME=c:\ec2-api-tools
set JAVA_HOME=c:\Program Files\Java\jre6

# Update PATH
set PATH=%PATH%:c:\ec2-api-tools\bin

#Test by running
ec2-version
# Output: 1.3-46266 2009-11-30

# Add keys
set EC2_PRIVATE_KEY=c:\aws-keys\pk-XXXXXXX.pem
set EC2_CERT=c:\aws-keys\cert-XXXXXXX.pem

# Use AWS Management Console
# Create new instance based on existing AMI
# Once instance is launch and running create a volume, make sure that the availability zone is the same as the instance, set device to /dev/sdh
# Attach the volume to the instance
# Connect to the instance using Putty
# Create the following script
vi instance-to-ebs-ami.sh

#!/bin/bash
# Run this script on the instance to be bundled
# tested with Canonical Ubuntu 9.10 base ami

EBS_DEVICE=${1:-'/dev/sdh'}
IMAGE_DIR=${2:-'/mnt/tmp'}
EBS_MOUNT_POINT=${3:-'/mnt/ebs'}

mkdir -p $EBS_MOUNT_POINT
mkfs.ext3 ${EBS_DEVICE}
mount ${EBS_DEVICE} $EBS_MOUNT_POINT

#make a local working copy
mkdir /mnt/tmp
rsync --stats -av --exclude /root/.bash_history --exclude /home/*/.bash_history --exclude /etc/ssh/ssh_host_* --exclude /etc/ssh/moduli --exclude /etc/udev/rules.d/*persistent-net.rules --exclude /var/lib/ec2/* --exclude=/mnt/* --exclude=/proc/* --exclude=/tmp/* / $IMAGE_DIR

#ensure that ami init scripts will be run
chmod u+x $IMAGE_DIR/etc/init.d/ec2-init-user-data

#clear out log files
cd $IMAGE_DIR/var/log
for i in `ls ./**/*`; do
echo $i && echo -n> $i
done

cd $IMAGE_DIR
tar -cSf - -C ./ . | tar xvf - -C $EBS_MOUNT_POINT
#NOTE, You could rsync / directly to EBS_MOUNT_POINT, but this tar trickery saves some space in the snapshot

umount $EBS_MOUNT_POINT

chmod +x instance-to-ebs-ami.sh
./instance-to-ebs-ami.sh
exit

# Go back to AWS Management Console
# Detach the volume
# Create a snapshot of the volume

# Now, we register the snapshot to be able to create an EBS image
# Run this command from your remote machine with Amazon's EC2 API Tools installed
# Fill in the required variables

ec2-register --snapshot snap-xxxxxx --kernel aki-xxxxxxx --ramdisk ari-xxxxxxxx --description="" --name="debian-5.0-lenny-base-20091011" --architecture i386 --root-device-name /dev/sda1
#sample output => IMAGE ami-xxxxxxx


References/Credits:
http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html
http://gist.github.com/249915

Try this script for creating automated snapshots:
http://developer.amazonwebservices.com/connect/entry.jspa?categoryID=100&externalID=1663
or
http://johnwoconnor.blogspot.com/2009/06/automatic-volume-snapshots-using-amazon.html

Friday, July 17, 2009

Creating a Remote Git Repo

Here are some steps I used to create a remote Git repo based on my local files.

Create the folder for the new application and change to that directory.
> mkdir /var/www/html/newapp
> cd /var/www/html/new
app

Make sure git is installed, if not install it
> sudo apt-get install git-core git-doc

Initialize the folder to become a git directory
> git init

********* Now switch to your local files ***********
> cd ~/myapp

Initialize git with the local folder as well
>git init

Tell git about yourself
>git config user.name "Your Name"
>git config user.email "Your Email"


Now, tell git which server to talk too
>git remote add origin username@yourdomain.com:/var/www/html/newapp/.git
Make sure that this user has SSH access to the box.


>git config branch.master.remote origin
>git config branch.master.merge refs/heads/master


Make your initial commit and push it to the remote server.
>git commit -a -m "Initial commit"
>git push origin master


And viola! Your files will be pushed to the remote server. Now, I was wondering why when I went to access the server, the files didn't seem to be in the working directory, even though I could see the log showed my Initial commit. It turns out that when things are pushed to master, they enter a limbo state. You have to do a git checkout -f, to actual have the commit become live. Note that doing a git checkout -f, will throw away any changes that have been made on the remote server.

To make the updating of the server easier, you can do the following:
> vi /var/www/html/newapp/.git/hooks/post-receive
Add - git checkout -f
Save the file
>chmod +x /var/www/html/newapp/.git/hooks/post-receive

Friday, June 26, 2009

Read-optimize your source code

A good article about writing well-written code that will make it easy for other developers to quickly understand your code.

http://www.brendel.com/consulting/blog/2009/06/read-optimized-source-code.html

Another useful unix command to find text in many files

find . -type f -exec grep -il "text to find" {} \;

Will give you a list of all the files that contain "text to find", starting with your current directory and searching all sub-directories.

Tuesday, June 9, 2009

Useful unix command to find and replace text in many files

Here is a useful unix command I used today to find and replace text across multiple files and multiple directories.

find . -type f -exec sed -i "s/string1/string2/g" {} \;

This will replace all instances of string1 with string2.

Credit to Google and http://www.sb.fsu.edu/~soma/Proj/How-to/Search+Replace.pdf for helping me figure this out.

Wednesday, May 6, 2009

Koha April, 7th meeting

Galen plans to do a feature freeze at the end of August and release 3.2 at the end of October.

Before June we should be able to test BibLibre's acq work and Liblime will be reporting on there development work.

Galen will be doing a Release Manager blog for Koha 3.2. If Nicole has time she will take a first stab at improving the Koha User list welcome message to include a good Koha FAQ and links to helpful tips and guides.

hdl mentioned that Bugzilla can accept emails...interesting, I will have to look into this and see if it can improve my workflow in tracking and dealing with bugs.

All new enhancements will be tracked using Bugzilla in order to keep everything in one place. This will help the community be better informed about what other Koha libraries are sponsoring and what developers are working on.

Tuesday, April 28, 2009

Random Idea: Have PACMon manage computer updates

Writing this down so I don't forget.

Have PACMon display a list of each upgrade that a PAC needs, and include a way of ignoring certain updates, with option of viewing the ignore list. Then click upgrade and it will automatically send message to the PACs to get updates. Also include display of seeing which PACs need which updates. These tasks would be perfomed using apt and could include custom repositories if special versions of software are required.

Wednesday, April 22, 2009

Post-KohaCon news

From Galen Charlton,
  • (Re)starting a weekly update newsletter for changes to Koha 3.2
  • [Galen] will be calling a monthly IRC meeting to discuss the status of Koha development. The next meeting will be Wednesday, 6 May 2009 at 19:00 UTC.
  • Making some improvements to using the bug database to record who is working on various bugs and enhancements.
  • Setting a convention to tag regressions so that they get a higher priority for bugfixes.
  • Making use of the bug voting feature in Bugzilla to help Koha users and contributors prioritize bugs.

Can you help Koha?

The Koha project is seeking:
  • Quality Assurance manager(s), to review all patches that are sent in to uphold a particular level of quality among the Koha code base. It was suggested that perhaps this be a funded position through an organization or the KUDOS user group since it would take a significant amount of time. It was also suggested that if a QA manager can not be found that developers do more peer review and sign-off on each others patches.
  • Bug wrangler(s), to review the bugs at http://bugs.koha.org and make sure that old bugs are closed and all bugs are valid. The number of open bugs listed at the Koha bugzilla site is currently 827.
  • Funding or development of core code re-factoring, to clean existing Koha structure, make the entire development process more efficient, and increase code speed and readability. It is hard to find a specific library to want to sponsor an enhancement that would be something that only takes place behind the scenes, so it was suggested that this may fall to a group like the KUDOS user group.
These are just a few of the major roles that the Koha project is looking to fill. The project can always use more developers, documentation writers, translators, bug testers, and community adovocates. Anyone looking for more information on getting involved in the Koha project can visit http://wiki.koha.org/doku.php

Sunday, April 19, 2009

FRBR Diagram

FRBR diagram from code4lib - http://www.frbr.org/files/frbr-er-diagram-5.png

Koha Thinking Outside the Box

Do we need MARC records? We do need the ability to import and edit in MARC record format, by why do we store it in the database in that format? In this day and age where searching technology has advanced considerably, why are we still cataloging in a format that hinders search results and is much more time consuming then taking a Google approach and doing a full-text type search.

Caching would greatly improve performance. Using memcached would greatly increase this because it would store more information in memory. Nginx is a substitute to apache and apparently works very well with memcached and also would increase performance. Nginx talks directly to kernel and we be incredibly fast at serving static content. The NY Times profiler is a great tool to take a look where are the slow areas of a project. It would be great if someone could intstrument a way to make it easy to run the profiler against Koha. Idea for re-writing major dependcies of Koha like MARC and re-submitting it to CPAN. Using Yslow with Firebug extension is a good one to help determine what is taking a long time to load for front-end web sites.

Switch the sessions and zebraqueue table to MyISAM because innodb never shrinks an only grows and grows.

Koha Sponsoring Discussion

This is a late post, but I wanted to write down some information from the sponsorship talk that occurred on the last session of the Koha Con users conference(April 17, 2009).

There was a heated discussion in regards to the best way for the community to get better about informing each other what projects are being sponsored and how best to do resource sharing. It was agreed upon that the community would use the existing to Bugzilla (http://bugs.koha.org) to input enhancement work and sponsored development. Since Bugzilla doesn't include fields for all the information we'd like to see about sponsored development features it was suggested that there be a sort of template/example bug placed so that others could see all the information that needed to be included into the comments field.

Many people are hoping that with this method if someone doesn't have money to sponsor development or would like to help partially fund a development it will make it possible for others who are also interested in the idea the add feedback or even contribute.

I like this idea, however, ultimately I would like to see a better tool developed as it could have a very positive impact on the Koha community. If there were a way for vendors to bid on features directly from this tool and a way for libraries or individuals to contribute to the feature right from the browser, it would just make the entire process much easier and efficient. You would be able to have a meter or similar feature to say something like, "Only $200 left needed to sponsors this project". It will also make it easy for people to vote and make certain feature requests more of a higher priority without having to search through the thousands of bugs in bugzilla. This type of system could also have a method for people to say I have $1000 to offer for this project, who would be interested in developing it.

Does this type of service exist anywhere is there any type of open source tool similar? Could this be a website with support for many different projects to organize their development efforts?

Koha Dev Workflow Tips and Tricks

pieces "borrowered" from Nicole Engard blog post (thank you!)

Git
Chris - Git has a built in garbage collector $ git gc - if you run it after creating a branch and before checking it out, makes switch branches much faster (make a branch, run this and then checkout the branch).

Joe - has a script he makes on different servers that he uses to get his shell to where he wants it to be for testing, it sets his self created variables and standard values (http://blogs.liblime.com/developers/2009/04/19/simple-shell-trick/). Also he posted a good tip on the LibLime blog: http://blogs.liblime.com/developers/2009/04/10/simple-git-trick-for-bash/

Galen - One of the things that he does since he can’t claim to have the visual design skills that Owen does - he has become a real stickler about the HTML that is on the OPAC & Staff client - making it appear as XHTML. There are tools in Firefox that can make for a good development environment: Fire Bug, for validation the HTML validator plugin in excellent at doing it quickly without submitting your site to an online validator, Firefox accessibility plugin lets you run automated tests against your site to meet requirements for ADA, Also the web developer plugin, Yahoo! Dev tools are slow, but they provide valuable info. Something that is useful, but not a dev tool, is the Zotero plugin (citation manager) www.zotero.org — geared toward people to do lots of research online.

Unix
-screen command very useful! Look up example .screenrc files
-cluster ssh, ability to have multiple ssh sessions open and run all commands at same time http://sourceforge.net/projects/clusterssh/
-sshfs mount a remote file system
-nohup will run script in the background which is good for like doing a zebra rebuild in case your ssh session gets disconnected

grep for file and | to xargs

Saturday, April 18, 2009

Git tips

tig - cool way to look at commits

git send-email 0001-patch1 0002-patch2 etc etc

To get patches from email you can use mutt, fetchmail, etc. You can even use Thunderbird to get patches, by saving it as a .eml file.

git am -u -s -3 -i /path/to/patch.eml

-u = use utf-8
-s = sign the commit
-3 = use three-way merge, Galen says that this just makes it work
-i = use interactive method

Notes on Koha structure

Make sure that get_user_and_template is the first call and no actions are required before it. It is a security risk if any calls are made before that line because the user could run it directly if they know the url without authentication.

#!/usr/bin/perl
use C4::something;
use C4::something else;
user C4::Output;

(_) = get_template_and_user (..)

my $query (or my $input) = new CGI; //allows you to grab params

my $biblionumber = $query->param('biblionumber');

sub functions should follow the naming convention, GetSomethingAndSomething

Get
New
Mod
Del

$op = 'mod(get/new/del)' // often used means operation
output_html_with_http_headers; (nothing executed after)