Configuring Network Interfaces on Linux

Most of the Linux distros are derivative from one of these mainstream distributions

1. RedHat e.g. ScientificLinux, CentOS, Rocks, Fedora
2. Debian e.g. Ubuntu, Linux Mint

 Despite their common root (Linux) the way how they configure devices and services varies dramatically.
On RedHat systems, the main configuration files are /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-ethN. Here you can find data and syntax of these files.
On Debian, the main file is /etc/network/interfaces. How to populate that file can be found here.
When modifications have been done, the way how the changes are taken is as follows

1. On RedHat run sudo /etc/init.d/network restart
2. On Debian-based run sudo /etc/init.d/networking restart

 When major changes are performed additional steps could be required, for instance, when a new network interface is "inserted".

Undergrad project proposal

This will be a periodic blog entry where different proposals for grad and undergrad projects will be post.

Last week, I attended the Open Science Grid All-hands meeting held on Lincoln, Nebraska. There, I had the opportunity to talk with José Caballero and he later sent me an e-mail where a couple of projects were described. I won't share the e-mail's content but a summary of my better understanding of the problems to be need addressed.

But what is grid computing? Well, grid computing is a large scale computational platform where resources are scattered geographically and shared by institutions around the globe following secure standardize mechanisms. Storage and processing are the resources more frequently shared by grid users. Links: www.gridcafe.org, www.globus.org.

Take care that I bolded three words: secure, institutions and processing. Lets see a detailed view of them

• Institutions. Computational resources are deployed or installed in different centers or buildings. People with proper rights could access these resources according to policies established by owners of those resources.
• Secure. Information and physical resources are shared in a secure way. Following well-known authentication mechanisms, users and resources are identified in such a way that secure interactions can be performed between them. http://gdp.globus.org/gt3-tutorial/multiplehtml/ch10s04.html
• Processing. Any resource which is attached to a computational device is a good candidate for being shared in a grid platform. In particular, some institutions have hundreds or even thousands of cores which are underutilized most of the time. Grid computing has defined and implemented protocolos to leverage the sharing of processing units amongst users belonging to digital communities, a.k.a. Virtual Organizations, VO. Check this picture.

What is the problem with job submission in a grid environment? It's not an easy task for end users

• A user should get grid certificates, X.509
• He/she needs to learn about basic basic command for interoperating with condor-based clusters.
• Mechanisms for prioritizing the jobs execution and resource brokering are many time hard-coded in the grid meta-schedulers.

A software solution to tackle all these problems is barely found. Some solutions partly solved this problem, e.g. glideinWMS (1st bullet) and PanDA (2nd and 3rd bullets). The main problem with PanDA is that it is highly attached to ATLAS. ATLAS is a USA computational infrastructure where different research projects are run. Different VOs need badly a software tool able to support all these features.

The main goal of this project is create a web application highly customizable where different features can be changed "on the fly", for instance:

• Scheduling algorithms
• Priority policies

The web application should employ any SQL database as back-end. That database stores the representation of a job. For instance, PanDA stores the following information to represent a job

• Binary filename. This filename is the actual program that should be remotely executed. This filename could be a full path directory or a URL.
• Argument list.
• Hash list. Some samples for keys in nodes in this hash list can be
• siteid, a name to represent a queue where this job can be queued
• queue, a compute element's name

The web application should support the monitoring of the jobs: job owner, job state, etc.

Additional notes:

• The web application can run in a machine different from the machine where the job runs.
• cURL is the tool to provide secure interaction amongst the web application and the execution machine.

Focus follows mouse for Unity

Something that I love from X window systems is the possibility to activate a window with no click on it. However, Unity and Gnome 3 by default do not make accessible this feature. I googled how to activate the "focus follows mouse" feature and I found this link. I typed these commands and now the feature is back ;-)

gconftool-2 --type string --set /apps/metacity/general/focus_mode mouse
gconftool-2 --type boolean --set /apps/metacity/general/auto_raise false

"apt-get upgrade" "packages have been kept back"

I have deactivated the automatic upgrade process in my Ubuntu box then every one or two week I execute the following steps,

sudo apt-get update
sudo apt-get upgrade

However, today during the upgrade process I got this message:

Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages have been kept back:
  linux-generic linux-headers-generic linux-image-generic
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

I didn't understand why this happened so I googled and I found this link where it explains that "the packages that have newer versions available, and install any new dependencies which are required to do that." Then for fixing this, just run the "apt-get dist-upgrade" command.

How to find the window id in xorg

Execute "xwininfo", that's it.

Preseeding files for automatic installation of Linux systems

Days ago, I found an interesting link which explains how to install Linux systems answering no questions. This procedure doesn't require any particular CD, you can use your regular installation CD along with a text file where answers to the usual questions for installing a Linux system are responded.

The original link can be found here and additional information about content of preseeding files can be obtained from here and here.

This procedure is a blessing when you frequently install Linux systems.

This procedure was initially implemented for RedHat systems but it was later extended to Ubuntu systems. The preseeding file and variables defined in it are release dependent, for instance a preseeding file for lucid may not work oneiric.

Finally, if you wan additional info, this kind of procedure is also referred as "preconfiguration file", "preseeding file" and "installcdcustomization".

Error in API call to delete - Vagrant

These days, I'm heavily working with Vagrant and Chef. During my working sessions, I created several scripts then run "vagrant up", scripts fail then "vagrant destroy" then do some modifications, run "vagrant up" again and this loop occurs for many times. Suddenly, "vagrant up" doesn't run but arises this message:

[node1] Destroying VM and associated drives...
/var/lib/gems/1.8/gems/virtualbox-0.9.1/lib/virtualbox/com/implementer/ffi.rb:106:in `call_and_check': Error in API call to delete: 2147944126 (VirtualBox::Exceptions::COMException)
    from /var/lib/gems/1.8/gems/virtualbox-0.9.1/lib/virtualbox/com/implementer/ffi.rb:80:in `call_vtbl_function'

Looking into log files (${HOME}/.VirtualBox), I found that for an unknown reason "vagrant destroy" can not delete the directory which contains the disk attached to the virtual machine.  Solution: erase that directory by hand. Problem solved!

extracting audio content from video files with open source tools

These days, I have downloaded many music videos and I have enjoyed them but I don't only want to watch them but also listen them. I looked in Internet and found this interesting link which give many tips about how to transform and extract information from multimedia files using ffmpeg.

Then, I created this bash script which extracts audio content from music videos.

#!/bin/bash
OUTPUT=""
if [ $# = 1 ] ; then
    OUTPUT=${1%.*}.mp3
elif [ $# = 2 ] ; then
    OUTPUT=${2}
else
    echo "Usage:"
    echo "     ${0} video_filename output.mp3"
    exit
fi
/usr/bin/ffmpeg -i "${1}" -vn -ar 44100 -ac 2 -ab 192 -f mp3 "${OUTPUT}"

Cut and paste it into your preferred editor (VI). This link explains how to retrieve just the file name without its extension.

Getting high resolution images from PDF files

Today I was preparing a lecture for Thursday (I guess) and I found very illustrative images from a PDF book but I got some problems for extracting them from the book.

• Acroread (for Linux) didn't keep the image in the clipboard 
• The "convert" tool (from Imagemagick) by default, made a poor conversion from PDF to any format

I searched in Internet and I found these two links that gave me a clue

• http://ardvaark.net/useful-pdf-imagemagick-recipes
• http://www.imagemagick.org/discourse-server/viewtopic.php?f=1&t=11476

For my needs "convert -density 300 pdffile.pdf image.png" did the trick. The image.png file gave me an excellent resolution. From that point, I used gimp for cropping the region that I need.

Post-installation commands...

I have been a big fan of Fedora but those days I gave a try to Ubuntu 11.04 (Beta2). I like the way how Ubuntu manages packages since it looks very clean and stable although it's a Beta [2] release.

I was playing around with new packages and some updates ('apt-get dist-upgrade') which provoke a unstable system so a couple re-installations have been necessary. After every re-installation I executed the following commands:

1. sudo software-properties-gtk, the Ubuntu's mirror for Colombia is not synchronized and the 'apt-get update' procedure throws some errors. Through this command, I could change the Colombia Mirror for the 'Main Server' option.
2. gnome-window-properties, this command allows to modify the current windows behavior e.g. 'Select windows when the mouse mover over them' and Roll up window on the 'Double-click titlebar' event happens.

Give a try to Ubuntu 11.04, it's cool and fast.

PS: Fedora guys, why are you looking for a code name for Fedora 16 when Fedora 15 has not been released yet. Awkward, ah?

Ubuntu 11.04 Beta

Last week, I installed Ubuntu 11.04 Beta on my laptop. My Linux box exhibits a better performance during the login procedure.
Despise the new graphical environment is dramatically different, the learning curve is not so steep.
I'm a Latex user and some additional packages were needed for a fully operational environment

• texlive-latex-base
• latex-beamer
• texlive-fonts-recommended

OK, unexpectedly [g]vim was not installed, so vim-gtk package was also installed by hand.

Working with Web Services

"Web services" is a paradigm that allows the development of loosely coupled applications composed by several modules which interact through well-known and widely adopted protocols, e.g. SOAP and HTTP.
Web services allows to orchestrate several modules [perhaps provided by different developers] in one single process known as workflow.
A workflow is then a special kind of application which integrates software components who expose their functionality through the web service description language, WSDL.
May be you are wondering if a particular development platform is necessary for creating web service-based applications. The answer is NO. Any platform able to understand (reads and writes) SOAP messages, it is also able of interacting with any other platform to speak the web service language. Therefore, web service is a framework to leverage the integration of multiple software stacks, operating systems + programming languages.
Despise diverse technologies which have been developed to support this paradigm, this entry would focus on a widely known Java-based software stack Apache Tomcat + Axis2. Tomcat is an application container and Axis2 is the framework to process SOAP messages.

How to install tomcat + axi2
Download the most recent version of Apache Tomcat. Apache Tomcat is a Java-based application container to support different technologies such as Servlets and JSP. (Check References section)

For Axis2 is highly recommended to download the .WAR distro. The war distro provides an easy way to deploy web services.

How to write/deploy a web service
There are different ways for deploying Java web services. Right here, and only for demostration purposes, the deployment and execution of POJO-based classes is presented.

Write a usual Java class, for instance HelloWorld.class.
class HelloWorld {
  String sayHello(String name) {
     return "Hello " + name;
  }
}

Compile the file and copy the class file into the ${TOMCAT_CONTAINER_DIR}/webapps/axis2/WEB-INF/pojo directory. If 'pojo' does not exist, please create it.

[Re-]run your tomcat container and check the available services. The HelloWorld service would be listed.


References

• Apache Axis2 Web Services - An introductory tutorial is accessible in this link.
  
• Tomcat - Version used during this tutorial was Apache Tomcat 7.0.8
  
• Apache ODE - Orchestration Director Engine
  
• BPEL - Business Process Execution Language
  
• Eclipse - Helios release was used during this tutorial
  

Projects for Distributed Systems Lab

These projects are fundamental for creating a computational ecosystem in which vast majority of networking management tasks could be automatized. For portability, the services and tools must be deployed on virtual machines.

Network services

• DHCP Server
• DNS Server

Deployment services

• Fedora mirror
• Cobbler/koan
  

Services to enable remote access

• Tools to support remote monitoring via remote terminal as well as browser interface
  NOTE: wake-on LAN service would be enable in all machines at lab
  

Repository services

• Git tool

Security tool

• To search, evaluate and select a network intrusion detection tool
  

There would be two operating systems in the lab

• CentOS would be the Linux distro in which cobbler/koan service would reside
• Ubuntu server would be used to deploy DNS, DHCP and Git programs along with a network intrusion detection tool.

Repositories for both environments would be required. Only last releases would be maintained.

Text-based network monitoring tools...

I want share some projects that I found very useful when you require to monitor you network activity. Those projects are not so much sophisticated but very useful for figuring out how and who is consuming your network bandwidth.

• iptstate this program presents connections "observed" by iptables
• speedometer is a python script to show in real-time how much bandwidth is consumed by a network interface. It discriminates between download and upload bandwidth consumption. How to use it: 'speedometer -rx <network-interface> -tx <network-interface>'
  
• pktstat "displays a real-time list of active connections seen on a network interface, and how much bandwidth is being used". How to use it: 'pktstat -i <network-interface>'

<network-interface> could be wlan0, eth0, ppp0, and so on.

References

• Link - 1
  

Booting order during the deployment process... [CETA-CIEMAT]

The booting order during the deployment of a computing ecosystem which provides grid computational services using the OCD-gLite script is first working nodes(WN) then computing element (CE).
That is it because access permissions with no password is required for job submission and those permissions are set up by CE and it requires that all working nodes be up and running.

Submitting jobs to maui

For submitting jobs from the computing element

• Login as a user different of 'root'. In particular, when a computing element has been deployed with the OCD-gLite, it comes with a lot of user. Try 'ceta000'.
• Create a basic bash script
  #!/bin/bash
  /bin/hostname
  Change its permissions as follows "chmod +x script.sh"; assuming that the script has been named as 'script.sh'
• Submit the job, executing 'qsub -q ceta script.sh'. '-q ceta' indicates that this job would be enqueued to one queue named 'ceta'. 
• If everything works as it is expected, a file called 'script.sh.o0' must contain something. ;-)

This notes were taken during my stay at CETA-CIEMAT and the information provided here is only relevant for usage scenarios experimented there.

Building dependencies for a Fedora package...

These days, I was "engaged" with a project named libguestfs. My box runs Fedora 13 and current version is Fedora 14. Therefore some packages are oldie and particular software requirements are necessary. For installing 'libguestfs' in my system, considering all the software ecosystem that it needs, this yum command is very useful.

# yum-builddep libguestfs

It, as its name suggests, prepares all the dependencies for the libguestfs package.

When you need to install a package which requires a lot of dependencies and you want avoid the cumbersome process of install one by one dependency, then try this command first.

Some additional web links relevant to Condor

• An interesting link describing how to work with Java and Condor.
• Different examples with their corresponding submission files.
• Other interesting site describing how to use Condor.
• Another interesting link for rookie Condor users.
  

Enabling the "Parallel" universe in Condor

If you have used our script (check the last reference) for deploying Condor then you need modify your local configuration files (condor_config.local) at master and worker nodes as follows:

Master:

UNUSED_CLAIM_TIMEOUT = 0
MPI_CONDOR_RSH_PATH = $(LIBEXEC)
ALTERNATE_STARTER_2 = $(SBIN)/condor_starter
STARTER_2_IS_DC = TRUE
SHADOW_MPI = $(SBIN)/condor_shadow

Worker Nodes:

DedicatedScheduler = "DedicatedScheduler@PASTE_SUBMIT_NODE_NAME_HERE" 
STARTD_ATTRS = $(STARTD_ATTRS), DedicatedScheduler
SUSPEND = False
CONTINUE = True
PREEMPT = False
KILL = False
WANT_SUSPEND = False
WANT_VACATE = False
RANK = Scheduler =?= $(DedicatedScheduler)
MPI_CONDOR_RSH_PATH = $(LIBEXEC)
CONDOR_SSHD = /usr/sbin/sshd
CONDOR_SSH_KEYGEN = /usr/bin/ssh-keygen
STARTD_EXPRS = $(STARTD_EXPRS), DedicatedScheduler

# Dedicated Node.

START=TRUE

In addition, I attach a submission file for running parallel jobs.

should_transfer_files = Yes
when_to_transfer_output = ON_EXIT_OR_EVICT
universe = parallel
executable = /bin/hostname
+ParallelShutdownPolicy = "WAIT_FOR_ALL"
machine_count = 3
log        = hostLog
Output     = hostOut.$(Node).$(Process)
Error      = hostErr.$(Node).$(Process)
queue

The boldfaced line forces that Condor will wait until every node in the parallel job has completed.

References

• MPI applications within Condor
• An easy way to deploy Condor
  

My journey with libguestfs

When I work with Virtual Machines (VMs) many times I define/create a template hard disk (HD). This template could be later attached to a VM and then modified for attending specific needs. For instance, I have created an Ubuntu-based VM with just the OpenSSH service, ready to go. On top of it, I  can deploy different services and applications, such as:

• web servers
• monitoring tools
• hacking tools
• and so on

One of the hardest part of all this process, regards with modifying files inside the filesystem. One option is attach the hard disk to one VM, boot the VM up, log into the running VM and modify the required files. Very handy, uh?

I found the 'libguestfs' project. It provides a set of tools that allows you to view and modify files inside VMs. Check the project home page for more details and features.

I'm installing it on my Fedora 13 box. Since I would love to do some hacks in Java, executed the following command 'yum install libguestfs-java-devel'. Around 19 packages were necessary for installing the aforementioned package, so I accepted and voila!.

Next entry, I will talk about how to hack some virtual hard disks!