Danny's Blog

Customizing Unix Shell

2010-08-26T08:32:00.001-07:00

A few tools that really make working in a unix shell more enjoyable.

Byobu - Replaces "screen" for a screen with a more useful display

Change .bashrc file to include the following


# Shell colors
export LS_OPTIONS='--color=auto'
eval "`dircolors`"
alias ls='ls $LS_OPTIONS'

alias screen='byobu'

Create .vimrc file to make working in VIM easier


" URL: http://vim.wikia.com/wiki/Example_vimrc
" Authors: http://vim.wikia.com/wiki/Vim_on_Freenode
" Description: A minimal, but feature rich, example .vimrc. If you are a
"              newbie, basing your first .vimrc on this file is a good choice.
"              If you're a more advanced user, building your own .vimrc based
"              on this file is still a good idea.

"------------------------------------------------------------
" Features {{{1
"
" These options and commands enable some very useful features in Vim, that
" no user should have to live without.

" Set 'nocompatible' to ward off unexpected things that your distro might
" have made, as well as sanely reset options when re-sourcing .vimrc
set nocompatible

" Attempt to determine the type of a file based on its name and possibly its
" contents.  Use this to allow intelligent auto-indenting for each filetype,
" and for plugins that are filetype specific.
filetype indent plugin on

" Enable syntax highlighting
syntax on

" Set font according to system
set gfn=Monospace\ 10
set shell=/bin/bash

if has("gui_running")
  set guioptions-=T
  set t_Co=256
  set background=dark
  colorscheme peaksea
  set nonu
else
  colorscheme zellner
  set background=dark

  set nonu
endif

set encoding=utf8
try
    lang en_US
catch
endtry

set ffs=unix,dos,mac "Default file types

"------------------------------------------------------------
" Must have options {{{1
"
" These are highly recommended options.

" One of the most important options to activate. Allows you to switch from an
" unsaved buffer without saving it first. Also allows you to keep an undo
" history for multiple files. Vim will complain if you try to quit without
" saving, and swap files will keep you safe if your computer crashes.
set hidden

" Better command-line completion
set wildmenu

" Show partial commands in the last line of the screen
set showcmd

" Highlight searches (use <C-L> to temporarily turn off highlighting; see the
" mapping of <C-L> below)
set hlsearch

" Modelines have historically been a source of security vulnerabilities.  As
" such, it may be a good idea to disable them and use the securemodelines
" script, <http://www.vim.org/scripts/script.php?script_id=1876>.
" set nomodeline


"------------------------------------------------------------
" Usability options {{{1
"
" These are options that users frequently set in their .vimrc. Some of them
" change Vim's behaviour in ways which deviate from the true Vi way, but
" which are considered to add usability. Which, if any, of these options to
" use is very much a personal preference, but they are harmless.

" Use case insensitive search, except when using capital letters
set ignorecase
set smartcase

" Allow backspacing over autoindent, line breaks and start of insert action
set backspace=indent,eol,start

" When opening a new line and no filetype-specific indenting is enabled, keep
" the same indent as the line you're currently on. Useful for READMEs, etc.
set autoindent

" Stop certain movements from always going to the first character of a line.
" While this behaviour deviates from that of Vi, it does what most users
" coming from other editors would expect.
set nostartofline

" Display the cursor position on the last line of the screen or in the status
" line of a window
set ruler

" Always display the status line, even if only one window is displayed
set laststatus=2

" Instead of failing a command because of unsaved changes, instead raise a
" dialogue asking if you wish to save changed files.
set confirm

" No sound on errors
set novisualbell
set noerrorbells
set t_vb=
set tm=500

" Enable use of the mouse for all modes
set mouse=v

" Set the command window height to 2 lines, to avoid many cases of having to
" "press <Enter> to continue"
set cmdheight=2

" Display line numbers on the left
" set number

" Quickly time out on keycodes, but never time out on mappings
set notimeout ttimeout ttimeoutlen=200

" Use <F11> to toggle between 'paste' and 'nopaste'
set pastetoggle=<F11>


"------------------------------------------------------------
" Indentation options {{{1
"
" Indentation settings according to personal preference.

" Indentation settings for using 2 spaces instead of tabs.
" Do not change 'tabstop' from its default value of 8 with this setup.
set shiftwidth=2
set softtabstop=2
set expandtab

" Indentation settings for using hard tabs for indent. Display tabs as
" two characters wide.
"set shiftwidth=2
"set tabstop=2


"------------------------------------------------------------
" Mappings {{{1
"
" Useful mappings

" Map Y to act like D and C, i.e. to yank until EOL, rather than act as yy,
" which is the default
map Y y$

" Map <C-L> (redraw screen) to also turn off search highlighting until the
" next search
nnoremap <C-L> :nohl<CR><C-L>


"------------------------------------------------------------

SQL to Search All Tables for Column Name

2010-04-26T08:44:00.000-07:00

declare @search_string  varchar(50)

set @search_string = ''

SELECT table_name=sysobjects.name,
column_name=syscolumns.name,
datatype=systypes.name,
length=syscolumns.length
FROM sysobjects
JOIN syscolumns ON sysobjects.id = syscolumns.id
JOIN systypes ON syscolumns.xtype=systypes.xtype
WHERE sysobjects.xtype='U'
and syscolumns.name LIKE '%' + @search_string + '%'
ORDER BY sysobjects.name,syscolumns.colid

Credit to: http://codesnippets.joyent.com/posts/show/337

SQL Query to Search All Tables for an INT

2010-04-22T14:58:00.000-07:00

Here is a really good query I want to remember for searching all tables in a database for a specific integer.

declare  @sql varchar(8000), @tbl varchar(255), @col varchar(255), @data  varchar(50)

set @data = '249753'

declare cur_tbl cursor for
     select a.name, b.name from sysobjects a, syscolumns b, systypes c where a.id = b.id and a.type = 'U' and c.xtype = b.xtype and c.name in ( 'int' )
open cur_tbl
fetch next from cur_tbl into @tbl, @col
while @@fetch_status = 0
begin
     set @sql = '
                       if exists (select * from [' + @tbl + '] where convert( varchar(255), [' + @col + '] ) = ''' + @data + ''')
                             select tbl=''' + @tbl + ''', col=''' + @col + ''', [' + @col + '], * from [' + @tbl + '] where convert( varchar(255), [' + @col + '] ) = ''' + @data + '''
                       '
     exec(@sql)

     fetch next from cur_tbl into @tbl, @col
end
close cur_tbl
deallocate cur_tbl

Or to search for a string

declare  @sql varchar(8000), @tbl varchar(255), @col varchar(255), @data  varchar(50)

set @data = 'string you are looking for'

declare cur_tbl cursor for
      select a.name, b.name from sysobjects a, syscolumns b, systypes c where a.id = b.id and a.type = 'U' and c.xtype = b.xtype and c.name in ( 'varchar', 'nvarchar', 'text', 'ntext' )
open cur_tbl
fetch next from cur_tbl into @tbl, @col
while @@fetch_status = 0
begin
      set @sql = '
                        if exists (select * from [' + @tbl + '] where convert( varchar(255), [' + @col + '] ) like ''%' + @data + '%'')
                              select tbl=''' + @tbl + ''', col=''' + @col + ''', [' + @col + '], * from [' + @tbl + '] where convert( varchar(255), [' + @col + '] ) like ''%' + @data + '%''
                        '
      exec(@sql)

      fetch next from cur_tbl into @tbl, @col
end
close cur_tbl
deallocate cur_tbl

Credit to: http://www.experts-exchange.com/Microsoft/Development/MS-SQL-Server/Q_24142162.html

AV2010 has gotten worse!

2010-03-08T14:26:00.000-08:00

It was preventing me from installing Malwarebytes, and everytime I would run any program the virus would be started. I had to use the following registry fix:

Windows Registry Editor Version 5.00

[-HKEY_CURRENT_USER\Software\Classes\.exe]
[-HKEY_CURRENT_USER\Software\Classes\secfile]
[-HKEY_CLASSES_ROOT\secfile]
[-HKEY_CLASSES_ROOT\.exe\shell\open\command]

[HKEY_CLASSES_ROOT\.exe]
@=”exefile”
“Content Type”=”application/x-msdownload”

Then save as fix.reg and run it. After restarting the PC I was able to install Malwarebytes Anti-malware and remove the viruses.

Thanks to Kwisatz - http://www.virusremovalguru.com/?p=5528#comment-34676

Steps to creating a EBS image from an AMI (Windows)

2010-01-25T14:47:00.000-08:00

Download Amazon's EC2 API Tools to C:\ec2-api-tools

# Run the following on a remote machine with Amazon's EC2 API Tools installed
# Adjust these variables to match your own system
set EC2_HOME=c:\ec2-api-tools
set JAVA_HOME=c:\Program Files\Java\jre6

# Update PATH
set PATH=%PATH%:c:\ec2-api-tools\bin

#Test by running
ec2-version
# Output: 1.3-46266 2009-11-30

# Add keys
set EC2_PRIVATE_KEY=c:\aws-keys\pk-XXXXXXX.pem
set EC2_CERT=c:\aws-keys\cert-XXXXXXX.pem

# Use AWS Management Console
# Create new instance based on existing AMI
# Once instance is launch and running create a volume, make sure that the availability zone is the same as the instance, set device to /dev/sdh
# Attach the volume to the instance
# Connect to the instance using Putty
# Create the following script
vi instance-to-ebs-ami.sh


#!/bin/bash
# Run this script on the instance to be bundled
# tested with Canonical Ubuntu 9.10 base ami

EBS_DEVICE=${1:-'/dev/sdh'}
IMAGE_DIR=${2:-'/mnt/tmp'}
EBS_MOUNT_POINT=${3:-'/mnt/ebs'}

mkdir -p $EBS_MOUNT_POINT
mkfs.ext3 ${EBS_DEVICE}
mount ${EBS_DEVICE} $EBS_MOUNT_POINT

#make a local working copy
mkdir /mnt/tmp
rsync --stats -av --exclude /root/.bash_history --exclude /home/*/.bash_history --exclude /etc/ssh/ssh_host_* --exclude /etc/ssh/moduli --exclude /etc/udev/rules.d/*persistent-net.rules --exclude /var/lib/ec2/* --exclude=/mnt/* --exclude=/proc/* --exclude=/tmp/* / $IMAGE_DIR

#ensure that ami init scripts will be run
chmod u+x $IMAGE_DIR/etc/init.d/ec2-init-user-data

#clear out log files
cd $IMAGE_DIR/var/log
for i in `ls ./**/*`; do
echo $i && echo -n> $i
done

cd $IMAGE_DIR
tar -cSf - -C ./ . | tar xvf - -C $EBS_MOUNT_POINT
#NOTE, You could rsync / directly to EBS_MOUNT_POINT, but this tar trickery saves some space in the snapshot

umount $EBS_MOUNT_POINT

chmod +x instance-to-ebs-ami.sh
./instance-to-ebs-ami.sh
exit

# Go back to AWS Management Console
# Detach the volume
# Create a snapshot of the volume

# Now, we register the snapshot to be able to create an EBS image
# Run this command from your remote machine with Amazon's EC2 API Tools installed
# Fill in the required variables


ec2-register --snapshot snap-xxxxxx --kernel aki-xxxxxxx --ramdisk ari-xxxxxxxx --description="" --name="debian-5.0-lenny-base-20091011" --architecture i386 --root-device-name /dev/sda1
#sample output => IMAGE ami-xxxxxxx

References/Credits:
http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html
http://gist.github.com/249915

Try this script for creating automated snapshots:
http://developer.amazonwebservices.com/connect/entry.jspa?categoryID=100&externalID=1663
or
http://johnwoconnor.blogspot.com/2009/06/automatic-volume-snapshots-using-amazon.html

Creating a Remote Git Repo

2009-07-17T13:05:00.001-07:00

Here are some steps I used to create a remote Git repo based on my local files.

Create the folder for the new application and change to that directory.
> mkdir /var/www/html/newapp
> cd /var/www/html/newapp

Make sure git is installed, if not install it
> sudo apt-get install git-core git-doc

Initialize the folder to become a git directory
> git init

********* Now switch to your local files ***********
> cd ~/myapp

Initialize git with the local folder as well
>git init

Tell git about yourself
>git config user.name "Your Name"
>git config user.email "Your Email"

Now, tell git which server to talk too
>git remote add origin username@yourdomain.com:/var/www/html/newapp/.git
Make sure that this user has SSH access to the box.

>git config branch.master.remote origin
>git config branch.master.merge refs/heads/master

Make your initial commit and push it to the remote server.
>git commit -a -m "Initial commit"
>git push origin master

And viola! Your files will be pushed to the remote server. Now, I was wondering why when I went to access the server, the files didn't seem to be in the working directory, even though I could see the log showed my Initial commit. It turns out that when things are pushed to master, they enter a limbo state. You have to do a git checkout -f, to actual have the commit become live. Note that doing a git checkout -f, will throw away any changes that have been made on the remote server.

To make the updating of the server easier, you can do the following:
> vi /var/www/html/newapp/.git/hooks/post-receive
Add - git checkout -f
Save the file
>chmod +x /var/www/html/newapp/.git/hooks/post-receive

2 Good articles for programmers

2009-07-13T08:32:00.000-07:00

5 Tips for creating good code every day; or how to become a good software developer
http://www.makinggoodsoftware.com/2009/05/15/5-tips-for-creating-good-code-every-day-how-to-become-a-good-software-developer/

5 Top non-technical mistakes made by programmers
http://www.makinggoodsoftware.com/2009/07/07/5-top-non-technical-mistakes-made-by-programmers/

Read-optimize your source code

2009-06-26T13:53:00.000-07:00

A good article about writing well-written code that will make it easy for other developers to quickly understand your code.

http://www.brendel.com/consulting/blog/2009/06/read-optimized-source-code.html

Another useful unix command to find text in many files

2009-06-26T13:49:00.000-07:00

find . -type f -exec grep -il "text to find" {} \;

Will give you a list of all the files that contain "text to find", starting with your current directory and searching all sub-directories.

Useful unix command to find and replace text in many files

2009-06-09T13:32:00.001-07:00

Here is a useful unix command I used today to find and replace text across multiple files and multiple directories.

find . -type f -exec sed -i "s/string1/string2/g" {} \;

This will replace all instances of string1 with string2.

Credit to Google and http://www.sb.fsu.edu/~soma/Proj/How-to/Search+Replace.pdf for helping me figure this out.

Koha April, 7th meeting

2009-05-06T12:16:00.000-07:00

Galen plans to do a feature freeze at the end of August and release 3.2 at the end of October.

Before June we should be able to test BibLibre's acq work and Liblime will be reporting on there development work.

Galen will be doing a Release Manager blog for Koha 3.2. If Nicole has time she will take a first stab at improving the Koha User list welcome message to include a good Koha FAQ and links to helpful tips and guides.

hdl mentioned that Bugzilla can accept emails...interesting, I will have to look into this and see if it can improve my workflow in tracking and dealing with bugs.

All new enhancements will be tracked using Bugzilla in order to keep everything in one place. This will help the community be better informed about what other Koha libraries are sponsoring and what developers are working on.

Random Idea: Have PACMon manage computer updates

2009-04-28T14:54:00.000-07:00

Writing this down so I don't forget.

Have PACMon display a list of each upgrade that a PAC needs, and include a way of ignoring certain updates, with option of viewing the ignore list. Then click upgrade and it will automatically send message to the PACs to get updates. Also include display of seeing which PACs need which updates. These tasks would be perfomed using apt and could include custom repositories if special versions of software are required.

Post-KohaCon news

2009-04-22T12:31:00.000-07:00

From Galen Charlton,

(Re)starting a weekly update newsletter for changes to Koha 3.2
[Galen] will be calling a monthly IRC meeting to discuss the status of Koha development. The next meeting will be Wednesday, 6 May 2009 at 19:00 UTC.
Making some improvements to using the bug database to record who is working on various bugs and enhancements.
Setting a convention to tag regressions so that they get a higher priority for bugfixes.
Making use of the bug voting feature in Bugzilla to help Koha users and contributors prioritize bugs.

Can you help Koha?

2009-04-22T12:06:00.000-07:00

The Koha project is seeking:

Quality Assurance manager(s), to review all patches that are sent in to uphold a particular level of quality among the Koha code base. It was suggested that perhaps this be a funded position through an organization or the KUDOS user group since it would take a significant amount of time. It was also suggested that if a QA manager can not be found that developers do more peer review and sign-off on each others patches.
Bug wrangler(s), to review the bugs at http://bugs.koha.org and make sure that old bugs are closed and all bugs are valid. The number of open bugs listed at the Koha bugzilla site is currently 827.
Funding or development of core code re-factoring, to clean existing Koha structure, make the entire development process more efficient, and increase code speed and readability. It is hard to find a specific library to want to sponsor an enhancement that would be something that only takes place behind the scenes, so it was suggested that this may fall to a group like the KUDOS user group.

These are just a few of the major roles that the Koha project is looking to fill. The project can always use more developers, documentation writers, translators, bug testers, and community adovocates. Anyone looking for more information on getting involved in the Koha project can visit http://wiki.koha.org/doku.php

FRBR Diagram

2009-04-19T12:55:00.000-07:00

FRBR diagram from code4lib - http://www.frbr.org/files/frbr-er-diagram-5.png

Koha Thinking Outside the Box

2009-04-19T11:38:00.000-07:00

Do we need MARC records? We do need the ability to import and edit in MARC record format, by why do we store it in the database in that format? In this day and age where searching technology has advanced considerably, why are we still cataloging in a format that hinders search results and is much more time consuming then taking a Google approach and doing a full-text type search.

Caching would greatly improve performance. Using memcached would greatly increase this because it would store more information in memory. Nginx is a substitute to apache and apparently works very well with memcached and also would increase performance. Nginx talks directly to kernel and we be incredibly fast at serving static content. The NY Times profiler is a great tool to take a look where are the slow areas of a project. It would be great if someone could intstrument a way to make it easy to run the profiler against Koha. Idea for re-writing major dependcies of Koha like MARC and re-submitting it to CPAN. Using Yslow with Firebug extension is a good one to help determine what is taking a long time to load for front-end web sites.

Switch the sessions and zebraqueue table to MyISAM because innodb never shrinks an only grows and grows.

Koha Sponsoring Discussion

2009-04-19T11:01:00.000-07:00

This is a late post, but I wanted to write down some information from the sponsorship talk that occurred on the last session of the Koha Con users conference(April 17, 2009).

There was a heated discussion in regards to the best way for the community to get better about informing each other what projects are being sponsored and how best to do resource sharing. It was agreed upon that the community would use the existing to Bugzilla (http://bugs.koha.org) to input enhancement work and sponsored development. Since Bugzilla doesn't include fields for all the information we'd like to see about sponsored development features it was suggested that there be a sort of template/example bug placed so that others could see all the information that needed to be included into the comments field.

Many people are hoping that with this method if someone doesn't have money to sponsor development or would like to help partially fund a development it will make it possible for others who are also interested in the idea the add feedback or even contribute.

I like this idea, however, ultimately I would like to see a better tool developed as it could have a very positive impact on the Koha community. If there were a way for vendors to bid on features directly from this tool and a way for libraries or individuals to contribute to the feature right from the browser, it would just make the entire process much easier and efficient. You would be able to have a meter or similar feature to say something like, "Only $200 left needed to sponsors this project". It will also make it easy for people to vote and make certain feature requests more of a higher priority without having to search through the thousands of bugs in bugzilla. This type of system could also have a method for people to say I have $1000 to offer for this project, who would be interested in developing it.

Does this type of service exist anywhere is there any type of open source tool similar? Could this be a website with support for many different projects to organize their development efforts?

Koha Dev Workflow Tips and Tricks

2009-04-19T07:09:00.000-07:00

pieces "borrowered" from Nicole Engard blog post (thank you!)

Git
Chris - Git has a built in garbage collector $ git gc - if you run it after creating a branch and before checking it out, makes switch branches much faster (make a branch, run this and then checkout the branch).

Joe - has a script he makes on different servers that he uses to get his shell to where he wants it to be for testing, it sets his self created variables and standard values (http://blogs.liblime.com/developers/2009/04/19/simple-shell-trick/). Also he posted a good tip on the LibLime blog: http://blogs.liblime.com/developers/2009/04/10/simple-git-trick-for-bash/

Galen - One of the things that he does since he can’t claim to have the visual design skills that Owen does - he has become a real stickler about the HTML that is on the OPAC & Staff client - making it appear as XHTML. There are tools in Firefox that can make for a good development environment: Fire Bug, for validation the HTML validator plugin in excellent at doing it quickly without submitting your site to an online validator, Firefox accessibility plugin lets you run automated tests against your site to meet requirements for ADA, Also the web developer plugin, Yahoo! Dev tools are slow, but they provide valuable info. Something that is useful, but not a dev tool, is the Zotero plugin (citation manager) www.zotero.org — geared toward people to do lots of research online.

Unix
-screen command very useful! Look up example .screenrc files
-cluster ssh, ability to have multiple ssh sessions open and run all commands at same time http://sourceforge.net/projects/clusterssh/
-sshfs mount a remote file system
-nohup will run script in the background which is good for like doing a zebra rebuild in case your ssh session gets disconnected

grep for file and | to xargs

Git tips

2009-04-18T12:19:00.000-07:00

tig - cool way to look at commits

git send-email 0001-patch1 0002-patch2 etc etc

To get patches from email you can use mutt, fetchmail, etc. You can even use Thunderbird to get patches, by saving it as a .eml file.

git am -u -s -3 -i /path/to/patch.eml

-u = use utf-8
-s = sign the commit
-3 = use three-way merge, Galen says that this just makes it work
-i = use interactive method

Notes on Koha structure

2009-04-18T07:29:00.000-07:00

Make sure that get_user_and_template is the first call and no actions are required before it. It is a security risk if any calls are made before that line because the user could run it directly if they know the url without authentication.

#!/usr/bin/perl
use C4::something;
use C4::something else;
user C4::Output;

(_) = get_template_and_user (..)

my $query (or my $input) = new CGI; //allows you to grab params

my $biblionumber = $query->param('biblionumber');

sub functions should follow the naming convention, GetSomethingAndSomething

Get
New
Mod
Del

$op = 'mod(get/new/del)' // often used means operation
output_html_with_http_headers; (nothing executed after)

Status of Development for Koha 3.2

2009-04-17T08:46:00.000-07:00

presented by Galen Charlton

Goals for Koha 3.2 (an ambitious release)

New acquisitions module
Holdings support
Many circulation improvements - allowing to configure the circulation rules to the nth degree
Improving stability

RFCs - A statement of saying we want this bit of functionality or we (programmer,library, vendor) are going to implement this function for the next version of Koha. Sometimes the RFCs are wouldn't it be nice.

Not all RFCs will be implemented for 3.2. Some were just proposals, and others didn't end up being sponsored.

New acquisitions module

Developed by BibLibre
In production with one of their customers
Working on submitting their patches for 3.2
Review and testing period

Holdings structure

Developed by LibLime, sponsored by WALDO
Introduces "summary" records to Koha
Entering testing by WALDO
You will be able to load them and export then and in theory you can pretend that the 852 tag doesn't exist.

WALDO circulation features

Proxy patrons
Fines thresholds
Callslips, similar to request system
Recalls, if someone has an item on loan and a faculty member has it the recall will say return it at once
Hourly loans, meant to support having an item checked out for hours instead of days
Email checkout slips

Other circulation features

Calculate fines in days debarred, developed by BibLibre for 3.0 and will makes it way to 3.2 soon
Placing hold on multiple items, introduced by Alloy Computing (yay HCL and Alloy Computing)
Additional hold request improvements sponsored by NEKLS
Course reserves

OPAC enhancements

Support for enhanced content providers, Syndetics, LibraryThing, Babeltheque
Tag multiple items (yay HCL and Alloy Computing)

Cataloging

biblios.net marc record editor integration
Improved browse indexes
ISBN13 normalization (sponsored by PISD) - meaning if you only have 10 digit isbn it will compute isbn13 and visa versa
Item bulk status change, BibLibre working on this and Liblime also working on a global change
Brief records
Record maintenance
Deleted records - ability to still be able to search for deleted items and bibs in a specific context

Serials (for a non-library person like myself, what are Serials?)

General improvements to serials display and predication pattern management
More control over display of recently checked in issues (WALDO)

Administration

Improved system preference editor (Jesse Weaver, developer at small library)

Reporting

Improvements to the guided reports, mainly the ability to add placeholder mechanism. It will let you put in a parameter at the time of running the report instead of having to edit the SQL everytime.

Miscellaneous

Granular permissions (new acquisitions module already implements some of these)
Internet Explorer compatibility improvements (WALDO)
Improvements to overdue report (PISD)
Improved OAI-PMH server (Tamil)
URL checker (Tamil)

Galen, "There will be at least one cool new thing coming from the KohaCon development meeting this weekend"

Timeline
Not something that is finalized. The currently goal for 3.2 will likely be late summer or early autumn. 3.1 release for testing in early summer.

The tip of the Koha bleeding edge is always and odd release number following the Linux kernel version numbering system.

Stimulus package and Koha

2009-04-17T08:20:00.000-07:00

It was mentioned that it might be possible to apply for Tech Stimulus package money to use for Koha development and expenses. It seems like it would be a great use of the money to go towards developing a free open source solution, which ultimately saves money in the future.

Is this something we can investigate further?

Customize your OPAC

2009-04-17T06:30:00.000-07:00

presented by Owen Leonard

Owen works at Nelsonville Public Library and says that he wouldn't do it again. Although you can customize the templates very easily, it is a huge pain to have to maintain the templates.

Changing the stylesheet is the best way to customize the OPAC because it doesn't need to be maintained each time you upgrade.

Owen recommends a site called Listomatic, it gives you css and html code for many different types of menus that exist.

opaccolorcss = file name of customize css file
opacusercss = embedded in-line instead of a separate file

Tools that Owen recommends for Web Development:

Firebug - Firefox extension (must have)
Jquery - javascript language

Link to Owen's blog - http://www.myacpl.org/koha/

Koha Integration: RFID, SIP2, LDAP

2009-04-16T14:08:00.000-07:00

presented by Joe Atzberger

LDAP
List of LDAP Tools
1. Apache Directory Server & Studio (client) http://directory.apache.org
- Newer than openldap and more stable
- Runs on OSX, Win32, and linux
- Open source
2. OpenLDAP - http://www.openldap.org
- Includes command line tools

Koha LDAP does not go grab all your users as a "dump". That is what IMPORT is for. Instead it updates when they try to login.

Main server configuration goes in koha-conf.xml file. The line looks similar to:

ldap://auth.example.com:389
dc=example,dc=com really base, but base tag not allowed
cn=Admin,dc=example,dc=com
example

Bind-as-auth has been hacked into Koha, but not done cleanly enough to import into main Koha.

There are two options for replicate and update, which can sync the information between LDAP and Koha (very cool!!!)

Know your own Schema, example used was inetOrgPerson.

You can define data Koha cares about by using elements in koha-conf.xml.

CPL (this makes the default branch CPL)

3 kinds of required data include things that are required by the database, login, and us.

Database
- surname
- address
- city

Login
- userid

SIP2
Unlike LDAP, SIP2 runs as a totally different server process, normally on a completely different server.

Extra dependecies - UNIVERSAL::require and Net::Server:Prefork

Now a well-documented setup process in Appendix to the Koha Manual. Sipconfig.xml file contains info about the SIP server. In the Sipconfig.xml you can specify port, recommends using telnet so not to expose raw protocol to outside world. Make sure the user that SIP uses actually has correct permission to perform circulate functions.

perl -I./ ./SIPServer.pm /home/koha/sipconfig/Sipconfig.xml (example command of SIP server running)

Doesn't support things that are specified at the item level, such as, item level holds and all features that were added to Koha after the SIP implementation. Almost all 3M hardware, requires and extension to the SIP2 implementation and that is also not yet supported, but Joe believes that NEKLS will be sponsoring that change.

There was some question about can you use SIP over SSL. Joe mentioned that when the SIP specifications were written they were assuming that you were going to use a serial connection or atleast be on the same network as the SIP server. Apparently, it is possible to setup a Secure SSH tunnel and use SIP over that connection.

Koha Disaster Recovery

2009-04-16T11:31:00.000-07:00

presented by Clay Fouts

Clay's primary job is to maintain the centralized repository for the Liblime hosted Koha installations.

Disaster is inevitable.... people will make mistakes, hardware will fail, malicious infiltrators can gain access. What is important is to understand how to minimize downtime, reduce frequency of failures and how to not lose important data.

How is data stored in Koha?

Koha source files and related configuration files
Perl dependencies
Zebra index content
MySQL database contents

There are a number of trade-offs you have to make when dealing with disaster recovery, such as, speed, expense, reliability, flexibility. For example, it is possible to get 99.99% up time, however, the expense of setting up a system to support this kind of up time will cost lots and lots of money because you will need multiple redundant servers, with raid hard drives and built-in automatic failover.

Storage Media
Disk(Hard Drive) - Fastest way to move data, but not very portable and still expensive compared to other options
DVD/CD - Extremely cheap and portable, but doesn't hold lots of data.
Tape - Portable and supports more data then CD/DVD, but not as much as most hard drives. Very expensive...
Cloud - unlimited capacity, redundant, so you will not lose data. Nice because you will not have to maintain another piece of hardware. Limiting capacity is network bandwidth.

Simply making backups is NOT sufficient, you need to make sure that the backups actually work and that you are able to restore completely from them. Verify that the media doesn't degrade, that it is secure, and accessible. If you store the data off site will it be accessible in the middle of the night if you need to do recovery?

Where to Store the data?
Onsite - fast, cheap, easy access, but if the place burns, you lose everything.
Offsite - Still have backups if your place burns, but it is difficult, expensive and slow access if you ever need them.
Cloud - can be the best of both if your network supports it. Most providers have redundant disks stored at multiple facilities spread across the world.

Optimal strategy is some kind of combination, for example, storing onsite if you need quick backups, but also storing the data offsite incase of a disaster.

MySQL Data Backups (most important)
Most sensitive and impossible to rebuild it manually because you don't know who has checked out what, who owes what fines, etc. Should be backed up frequently using multiple methods. Recommends using MySQL's bin logs to back up data.
Logical using tools like mysqldump. Provides the ability to rebuild a database from raw SQL statements. Very portable and can even be moved to another platform. Slow to back up and restore. Database is inaccessible during backup operation.
Binary using tools like cp, dd, LVM snapshots. Faster than logical backups. Still blocks access during the backup, but for less time. LVM snapshots can be taken almost instantaneously. Need to script so that you prevent access to database while backup is taken because it does not do it inherently otherwise you will get corruption of data. Binary files and disk partitions are much larger than logical SQL backups. The backups are difficult to verify and less portable.
Mirroring using master/slave replication. Fatest because it's ongoing. The replication server can be used as a hot spare in case primary server goes down. Very useful in combination with logical and binary backups. Has other uses, for example, reports in which you can have the reports all generated based on the slave server and it won't degrade the performace of the master server. Can be useful in combination with logical and binary backups because you can do backups based on the slave, so it never locks up the master server. Can introduce inconsistency in regards to storing timestamps, etc. To combat the inconsistency is make sure that the master server is not overloaded and prevent network latency.

Code base backup
Useful if you have customized the data.

Zebra Index backup
Huge database and not unique, no data loss if not backed up, but rebuilding will increase recovery time. Could take hours to rebuild the indexes if you have lots of data. Make sure that you are not adding to the index while the backups are going on.

Public Library Lunch Group

2009-04-16T11:30:00.000-07:00

We walked over to the mall across the street from the conference center to have lunch (very nice mall by the way). After finally getting my food, I was fascinated to hear some of the stories about the other public libraries using Koha and migrating to Koha.

I talked with two ladies from East Brunswick Public Library and was fascinated to hear very similar concerns to our own. They are currently using Horizon 7.4 and plan to go live on Koha in May (?). They mentioned that they plan to use Horizon to do cataloging and acquisitions since they found it difficult to currently use Koha to perform these tasks.

Several people at the table also expressed concerns about not knowing what all these different vendors were doing and how do we know we are all not paying for the same feature? We would like the vendors to communicate more with the community in terms of what features are being developed.

SQL Structure

2009-04-16T09:16:00.000-07:00

presented by Paul Poulain

authorised_values table used to define lists available in a lot of places. Paul, "the swiss knife tool in Koha"

Don't try to fill the biblios and items tables directly, use bulkmarcimport.pl tool or the API (more complex).

Koha is setup behind the scenes to handle both UNIMARC and MARC21. Lots of data in MARC21, very few important and very frequently used like title, author, call no.

History

Koha 1.x was not dealing with MARC.
Koha 1.x implemented something like FRBR before FRBR existed.
Koha 2.0 implemented MARC but relied on the previous 3 tables

Biblio and Biblioitems tables
Raw MARC data is stored in biblioitems table in the marcxml field. Some information is stored in a clearly named field in biblio table. The biblioitems table that contains information that is the root information of the edition.

In Koha 3.0 you should have a 1 to 1 correlation between the biblio and biblioitems table. The two tables could be merged into one, but has not been implemented yet. Koha takes care of translating the MARC database into the clearly named field, that is why you must use the import tool or the API.

The biblioitems table should be called biblioeditions, but no developer has renamed it yet.

Which table is used..

On results lists, MARC detail from marcxml
On standard biblio detail screen, if you have XSLT the data is taken from marcxml (?)
For UNIMARC detail screens, the biblio and biblioitems tables are used.
If just a few key fields, it is grabbed from biblio and biblioitems tables

If you crash Zebra database, everything is stored in the MySQL table.

Items table contains a row for each representation of the physical item in the library. All information is contained in the biblioitems.marcxml field, however, just like biblio table contains decoded information, the items table also contains some specific decoded fields.

In Koha the primary key for biblio data is the biblio#.

Borrower table
Surname is the only mandatory field in the borrower table. Personal information, first name, title, other names, intials, dateofbirth, address, street number, street type, address, address2, zip code, city , phone, email are all examples of fields in the borrower table.

The primary key for borrower table is the borrower#. The password in Koha is never stored in plain text and only stored in the md5 hash form.

Some free to use fields are sort1 and sort2, linked to authorised_valules. Borrower_attributes table supports extended attributes.

Issues table (checkouts/checkins)
All current issues are stored in issues table. All previous/finished issues are stored in old_issues table. They both have the same structure.

Four important fields, borrowernumber, itemnumber, date_due, issuedate

History of Koha

2009-04-16T06:32:00.000-07:00

presented by Chris Cormack and Paul Poulain

Chris is one of the original developers of Koha in 1999. There was no suitable responses to the RFP as they needed an application that worked over slow connections. Using a web based application was a natural choice because it works well over slow connections. There was no open source solutions at the time and open source was a natural choice because neither HLT or Katipo wanted to be a vendor. Developed in rapid prototype with lots of refining.

It's just a database... but with lots and lots of rules.

History of Koha: http://stats.workbuffer.org/history.html

When they first released version 1.0 of Koha, the first copy was downloaded in 20 minutes. In 2002, Paul Poulain got involved and Owen Leonard from Nelsonville Public Library.

Chris soon realized that when working with libraries, standards meant "standards" and he often had to ask which MARC.

In 2003, the community continues to grow, more developers, more translations. NPL went live in 2003. The NZ prime minister mentioned Koha in a speech.

In 2004, the first koha docs were written at www.kohadocs.org and first time a non-technical person contributed to Koha.

In 2005, Liblime got on board with Koha to support U.S. libraries. Paul formed a partnership with Henri Damien Laurent to meet the demand for Koha support. Decided to go with Zebra for full-text searching. New www.koha.org site released.

In 2006, Chris presented about Koha at Linux Conference Australia and Linus was in the audience! First, KohaCon held in Paris.

In 2007, lots of talks with Liblime about hiring Chris and several others. Started Koha Days to fix features that no one was sponsoring, but needed to be done. Koha won a NZ Open Source Award, despite being in the same category with the guy that wrote Ruby on Rails.

In 2008, Nicole did huge amounts of work on the Koha 3.0 manual. Paul and Henri formed Biblibre. Chris left Liblime and now works for Catalyst, but is the translation manager. Chris says, "2008 should be called the year of India", as major libraries including Dehli Public Library switched to Koha and several major companies to support Koha.