"apt-get upgrade" "packages have been kept back"

I have deactivated the automatic upgrade process in my Ubuntu box then every one or two week I execute the following steps,

sudo apt-get update
sudo apt-get upgrade

However, today during the upgrade process I got this message:

Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages have been kept back:
  linux-generic linux-headers-generic linux-image-generic
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

I didn't understand why this happened so I googled and I found this link where it explains that "the packages that have newer versions available, and install any new dependencies which are required to do that." Then for fixing this, just run the "apt-get dist-upgrade" command.

How to find the window id in xorg

Execute "xwininfo", that's it.

Preseeding files for automatic installation of Linux systems

Days ago, I found an interesting link which explains how to install Linux systems answering no questions. This procedure doesn't require any particular CD, you can use your regular installation CD along with a text file where answers to the usual questions for installing a Linux system are responded.

The original link can be found here and additional information about content of preseeding files can be obtained from here and here.

This procedure is a blessing when you frequently install Linux systems.

This procedure was initially implemented for RedHat systems but it was later extended to Ubuntu systems. The preseeding file and variables defined in it are release dependent, for instance a preseeding file for lucid may not work oneiric.

Finally, if you wan additional info, this kind of procedure is also referred as "preconfiguration file", "preseeding file" and "installcdcustomization".

Error in API call to delete - Vagrant

These days, I'm heavily working with Vagrant and Chef. During my working sessions, I created several scripts then run "vagrant up", scripts fail then "vagrant destroy" then do some modifications, run "vagrant up" again and this loop occurs for many times. Suddenly, "vagrant up" doesn't run but arises this message:

[node1] Destroying VM and associated drives...
/var/lib/gems/1.8/gems/virtualbox-0.9.1/lib/virtualbox/com/implementer/ffi.rb:106:in `call_and_check': Error in API call to delete: 2147944126 (VirtualBox::Exceptions::COMException)
    from /var/lib/gems/1.8/gems/virtualbox-0.9.1/lib/virtualbox/com/implementer/ffi.rb:80:in `call_vtbl_function'

Looking into log files (${HOME}/.VirtualBox), I found that for an unknown reason "vagrant destroy" can not delete the directory which contains the disk attached to the virtual machine.  Solution: erase that directory by hand. Problem solved!

extracting audio content from video files with open source tools

These days, I have downloaded many music videos and I have enjoyed them but I don't only want to watch them but also listen them. I looked in Internet and found this interesting link which give many tips about how to transform and extract information from multimedia files using ffmpeg.

Then, I created this bash script which extracts audio content from music videos.

#!/bin/bash
OUTPUT=""
if [ $# = 1 ] ; then
    OUTPUT=${1%.*}.mp3
elif [ $# = 2 ] ; then
    OUTPUT=${2}
else
    echo "Usage:"
    echo "     ${0} video_filename output.mp3"
    exit
fi
/usr/bin/ffmpeg -i "${1}" -vn -ar 44100 -ac 2 -ab 192 -f mp3 "${OUTPUT}"

Cut and paste it into your preferred editor (VI). This link explains how to retrieve just the file name without its extension.

Getting high resolution images from PDF files

Today I was preparing a lecture for Thursday (I guess) and I found very illustrative images from a PDF book but I got some problems for extracting them from the book.

• Acroread (for Linux) didn't keep the image in the clipboard 
• The "convert" tool (from Imagemagick) by default, made a poor conversion from PDF to any format

I searched in Internet and I found these two links that gave me a clue

• http://ardvaark.net/useful-pdf-imagemagick-recipes
• http://www.imagemagick.org/discourse-server/viewtopic.php?f=1&t=11476

For my needs "convert -density 300 pdffile.pdf image.png" did the trick. The image.png file gave me an excellent resolution. From that point, I used gimp for cropping the region that I need.

Post-installation commands...

I have been a big fan of Fedora but those days I gave a try to Ubuntu 11.04 (Beta2). I like the way how Ubuntu manages packages since it looks very clean and stable although it's a Beta [2] release.

I was playing around with new packages and some updates ('apt-get dist-upgrade') which provoke a unstable system so a couple re-installations have been necessary. After every re-installation I executed the following commands:

1. sudo software-properties-gtk, the Ubuntu's mirror for Colombia is not synchronized and the 'apt-get update' procedure throws some errors. Through this command, I could change the Colombia Mirror for the 'Main Server' option.
2. gnome-window-properties, this command allows to modify the current windows behavior e.g. 'Select windows when the mouse mover over them' and Roll up window on the 'Double-click titlebar' event happens.

Give a try to Ubuntu 11.04, it's cool and fast.

PS: Fedora guys, why are you looking for a code name for Fedora 16 when Fedora 15 has not been released yet. Awkward, ah?

Ubuntu 11.04 Beta

Last week, I installed Ubuntu 11.04 Beta on my laptop. My Linux box exhibits a better performance during the login procedure.
Despise the new graphical environment is dramatically different, the learning curve is not so steep.
I'm a Latex user and some additional packages were needed for a fully operational environment

• texlive-latex-base
• latex-beamer
• texlive-fonts-recommended

OK, unexpectedly [g]vim was not installed, so vim-gtk package was also installed by hand.

Working with Web Services

"Web services" is a paradigm that allows the development of loosely coupled applications composed by several modules which interact through well-known and widely adopted protocols, e.g. SOAP and HTTP.
Web services allows to orchestrate several modules [perhaps provided by different developers] in one single process known as workflow.
A workflow is then a special kind of application which integrates software components who expose their functionality through the web service description language, WSDL.
May be you are wondering if a particular development platform is necessary for creating web service-based applications. The answer is NO. Any platform able to understand (reads and writes) SOAP messages, it is also able of interacting with any other platform to speak the web service language. Therefore, web service is a framework to leverage the integration of multiple software stacks, operating systems + programming languages.
Despise diverse technologies which have been developed to support this paradigm, this entry would focus on a widely known Java-based software stack Apache Tomcat + Axis2. Tomcat is an application container and Axis2 is the framework to process SOAP messages.

How to install tomcat + axi2
Download the most recent version of Apache Tomcat. Apache Tomcat is a Java-based application container to support different technologies such as Servlets and JSP. (Check References section)

For Axis2 is highly recommended to download the .WAR distro. The war distro provides an easy way to deploy web services.

How to write/deploy a web service
There are different ways for deploying Java web services. Right here, and only for demostration purposes, the deployment and execution of POJO-based classes is presented.

Write a usual Java class, for instance HelloWorld.class.
class HelloWorld {
  String sayHello(String name) {
     return "Hello " + name;
  }
}

Compile the file and copy the class file into the ${TOMCAT_CONTAINER_DIR}/webapps/axis2/WEB-INF/pojo directory. If 'pojo' does not exist, please create it.

[Re-]run your tomcat container and check the available services. The HelloWorld service would be listed.


References

• Apache Axis2 Web Services - An introductory tutorial is accessible in this link.
  
• Tomcat - Version used during this tutorial was Apache Tomcat 7.0.8
  
• Apache ODE - Orchestration Director Engine
  
• BPEL - Business Process Execution Language
  
• Eclipse - Helios release was used during this tutorial
  

Projects for Distributed Systems Lab

These projects are fundamental for creating a computational ecosystem in which vast majority of networking management tasks could be automatized. For portability, the services and tools must be deployed on virtual machines.

Network services

• DHCP Server
• DNS Server

Deployment services

• Fedora mirror
• Cobbler/koan
  

Services to enable remote access

• Tools to support remote monitoring via remote terminal as well as browser interface
  NOTE: wake-on LAN service would be enable in all machines at lab
  

Repository services

• Git tool

Security tool

• To search, evaluate and select a network intrusion detection tool
  

There would be two operating systems in the lab

• CentOS would be the Linux distro in which cobbler/koan service would reside
• Ubuntu server would be used to deploy DNS, DHCP and Git programs along with a network intrusion detection tool.

Repositories for both environments would be required. Only last releases would be maintained.

Text-based network monitoring tools...

I want share some projects that I found very useful when you require to monitor you network activity. Those projects are not so much sophisticated but very useful for figuring out how and who is consuming your network bandwidth.

• iptstate this program presents connections "observed" by iptables
• speedometer is a python script to show in real-time how much bandwidth is consumed by a network interface. It discriminates between download and upload bandwidth consumption. How to use it: 'speedometer -rx <network-interface> -tx <network-interface>'
  
• pktstat "displays a real-time list of active connections seen on a network interface, and how much bandwidth is being used". How to use it: 'pktstat -i <network-interface>'

<network-interface> could be wlan0, eth0, ppp0, and so on.

References

• Link - 1
  

Booting order during the deployment process... [CETA-CIEMAT]

The booting order during the deployment of a computing ecosystem which provides grid computational services using the OCD-gLite script is first working nodes(WN) then computing element (CE).
That is it because access permissions with no password is required for job submission and those permissions are set up by CE and it requires that all working nodes be up and running.

Submitting jobs to maui

For submitting jobs from the computing element

• Login as a user different of 'root'. In particular, when a computing element has been deployed with the OCD-gLite, it comes with a lot of user. Try 'ceta000'.
• Create a basic bash script
  #!/bin/bash
  /bin/hostname
  Change its permissions as follows "chmod +x script.sh"; assuming that the script has been named as 'script.sh'
• Submit the job, executing 'qsub -q ceta script.sh'. '-q ceta' indicates that this job would be enqueued to one queue named 'ceta'. 
• If everything works as it is expected, a file called 'script.sh.o0' must contain something. ;-)

This notes were taken during my stay at CETA-CIEMAT and the information provided here is only relevant for usage scenarios experimented there.

Building dependencies for a Fedora package...

These days, I was "engaged" with a project named libguestfs. My box runs Fedora 13 and current version is Fedora 14. Therefore some packages are oldie and particular software requirements are necessary. For installing 'libguestfs' in my system, considering all the software ecosystem that it needs, this yum command is very useful.

# yum-builddep libguestfs

It, as its name suggests, prepares all the dependencies for the libguestfs package.

When you need to install a package which requires a lot of dependencies and you want avoid the cumbersome process of install one by one dependency, then try this command first.

Some additional web links relevant to Condor

• An interesting link describing how to work with Java and Condor.
• Different examples with their corresponding submission files.
• Other interesting site describing how to use Condor.
• Another interesting link for rookie Condor users.
  

Enabling the "Parallel" universe in Condor

If you have used our script (check the last reference) for deploying Condor then you need modify your local configuration files (condor_config.local) at master and worker nodes as follows:

Master:

UNUSED_CLAIM_TIMEOUT = 0
MPI_CONDOR_RSH_PATH = $(LIBEXEC)
ALTERNATE_STARTER_2 = $(SBIN)/condor_starter
STARTER_2_IS_DC = TRUE
SHADOW_MPI = $(SBIN)/condor_shadow

Worker Nodes:

DedicatedScheduler = "DedicatedScheduler@PASTE_SUBMIT_NODE_NAME_HERE" 
STARTD_ATTRS = $(STARTD_ATTRS), DedicatedScheduler
SUSPEND = False
CONTINUE = True
PREEMPT = False
KILL = False
WANT_SUSPEND = False
WANT_VACATE = False
RANK = Scheduler =?= $(DedicatedScheduler)
MPI_CONDOR_RSH_PATH = $(LIBEXEC)
CONDOR_SSHD = /usr/sbin/sshd
CONDOR_SSH_KEYGEN = /usr/bin/ssh-keygen
STARTD_EXPRS = $(STARTD_EXPRS), DedicatedScheduler

# Dedicated Node.

START=TRUE

In addition, I attach a submission file for running parallel jobs.

should_transfer_files = Yes
when_to_transfer_output = ON_EXIT_OR_EVICT
universe = parallel
executable = /bin/hostname
+ParallelShutdownPolicy = "WAIT_FOR_ALL"
machine_count = 3
log        = hostLog
Output     = hostOut.$(Node).$(Process)
Error      = hostErr.$(Node).$(Process)
queue

The boldfaced line forces that Condor will wait until every node in the parallel job has completed.

References

• MPI applications within Condor
• An easy way to deploy Condor
  

My journey with libguestfs

When I work with Virtual Machines (VMs) many times I define/create a template hard disk (HD). This template could be later attached to a VM and then modified for attending specific needs. For instance, I have created an Ubuntu-based VM with just the OpenSSH service, ready to go. On top of it, I  can deploy different services and applications, such as:

• web servers
• monitoring tools
• hacking tools
• and so on

One of the hardest part of all this process, regards with modifying files inside the filesystem. One option is attach the hard disk to one VM, boot the VM up, log into the running VM and modify the required files. Very handy, uh?

I found the 'libguestfs' project. It provides a set of tools that allows you to view and modify files inside VMs. Check the project home page for more details and features.

I'm installing it on my Fedora 13 box. Since I would love to do some hacks in Java, executed the following command 'yum install libguestfs-java-devel'. Around 19 packages were necessary for installing the aforementioned package, so I accepted and voila!.

Next entry, I will talk about how to hack some virtual hard disks!

Open in terminal - Nautilus

I'm a user of the Gnome window manager therefore I use its file manager, Nautilus.
Sometimes, I'm looking for a specific directory and need to start working there from the command line.
By default, Nautilus doesn't provide any tool for doing that but there is a package named 'nautilus-open-terminal' to enable that feature.
For Fedora systems run: sudo yum install nautilus-open-terminal.

For using xterm instead of terminal, check the second item in the 'References' section.

References:

• Nautilus-open-terminal: quick launch
• Using xterm
  

dos2unix in Ubuntu

I installed a JeOS appliance and it lacks of the 'dos2unix' utility. I found this site which explains how to install the aforementioned utility, aptitude install tofrodos.

Looks like this and other utilities are not installed in the last Ubuntu release, 10.04.

Many-task applications and divisible load theory

Recently, I found a paper about a new kind of applications named 'Many-task applications'. Those applications fill a gap between high performance computing and high throughput computing. Better say, those applications regard with bulk of tasks may be static or dynamic, homogeneous or heterogeneous, loosely coupled or tightly coupled, [1].
Divisible load theory offers another approach to the scheduling problem. DLT, for short, has been widely studied and applied to different distributed and large scale scenarios with important restrictions about the nature of the applications to be scheduled. However, DLT provides a close and optimal solution when some restrictions are observed.
It's important, then, to analyse the nature and characteristics of many-task applications in such a way that DLT approach can be successfully applied under circumvented  circumstances.


[1] Many-task computing for grids and supercomputers. I. Raicu, I. Foster, Y. Zhao, 2008.
[2] Scheduling many-task workloads on supercomputers; dealing with trailing tasks. T. Armstrong, Z. Zhang, D. Katz, M. Wilde, I. Foster, 2010.

Working with Qemu and GNS3

QEMU is an open source machine emulator. It is used by GNS3 to emulate PCs into network topologies defined by the user.
Why do we need a PC into a network topology? Linux-based PCs can be used as firewalls and routers. For our purposes we are interested in to deploy a virtual machine with firewall and router capabilities.
GNS3, our network simulation tool, exhibits valuable features but it lacks of consolidated documentation. As follow, I will provide some links for deploying a PC QEMU host in GNS3.

• Downloading and configuring a QEMU-based image. The configuration steps are performed into GNS3.
• Some tricks for running the terminal of the PC. The prior link suggests to modify some flags for configuring the VM image into GNS3, however the author of this link suggests no modify any flag if you require to access via terminal the PC host in GNS3 (Of course, we need it). I tested and worked BUT looks like there are connections problems with the deployed PC with other network equipments in the network topology.
  
• Download the more recent version of GNS3. I'm working with GNS3 version 0.7.3 but the link above suggest to work with the development version of GNS3. 

Stay tuned!

focus-follow-mouse

A desirable and useful behaviour under Linux GUIs is to get the focus of a window in which the mouse pointer is over. In Gnome, this behaviour is not enable by default, in fact, it isn't even configurable with the default administration/configuration tools. Two steps are then required to enable this feature

1. Install the "control-center-extra" package. As root, execute "yum install control-center-extra"
2. When the package is successfully installed, go to "System -> Preferences -> Window" and select the item "Select windows when the mouse moves over them"

References: Fedora forum

Setting up a Wireshark appliance

During this course we will use Wireshark for capturing network packets. Then, we would deploy an Ubuntu Linux based system in which install this sniffer tool.

When Linux appliance is up and running, as root user, type 'apt-get install wireshark'. Answer "Y" and then "ENTER" or just "ENTER". A bunch of packages start to download.

A note aside of this. If you have any window manager in your system, invoke the 'tasksel' command and select the "Ubuntu Desktop" option. This option would install the packages necessary for deploying a GUI that would ease your work with Wireshark.

Note: If you're experimenting problems with the screen resolution, first install the VirtualBox guest additions.

Preparing my network lab

This entry is two fold. First, provides me a record about what tools are required and how they are must be orchestrated for having a practical networking course. Second, gives a starting point to my alumni for setting up their own network lab.

Since Universidad del Valle, as many other universities, has limited resources, the networking laboratories would be executed on top of virtual environments and using open source software.

Well, let us start. As a virtualization tool I recommend VirtualBox. I know of other open source virtualization tools but I find VirtualBox particularly stable and easy to work with. VirtualBox allows the easy deployment of virtual machines running different operating systems (Windows NT, 2000, XP, Linux, Solaris, Mac OS X) that run on top of a similar number of host operating systems.

The selected guest operating system for the virtual machines(VM) is Ubuntu Server Edition. The main reason of this selection is because Ubuntu provides a JeOS (Just Enough Operating System) version of its server operating system. JeOS is characterized because it only installs the minimal set of components(operating system + system tools) that allows to barely run a Linux system.

Up to know, only the tools necessary for running virtual machines are mentioned however one interesting topic in computer networks is traffic analysis. For this task, Wireshark is the selected tool since it is the most widely used, documented and known sniffer at this time. Wireshark allows to analyse and filter network traffic by port, protocol, IP source/destination address, among others fields found in network packages/frames.

Given these tools, your first homework is:

1. Install VirtualBox
2. Download the current Ubuntu Server Edition
3. Create two Ubuntu-based virtual machines. On one machine you would deploy your preferred window manager and you would also install wireshark. The second machine, no GUI or network service would be installed but SSH.
   

That's all.

Cluster or Cloud???

Recently, Universidad del Valle acquired a bunch of machines. I'm in charge of determining the best software solution for those machines. The acquired equipment is 10 BL645C G6 HP worker nodes and one DL385 G6 HP head node. Those computing elements come along with a storage solution with 4 TB of capacity.

This equipment will be dedicated to different research activities at Engineering Faculty. Initially, I decided to install a cluster-based operating system, e.g. Rocks; however with new open source cloud solutions I'm not sure what is the "best" choice. On one hand, OpenStack looks a strong candidate because it is totally open but I'm not sure how many troubles could arise during the deployment of that solution. On the other hand, Eucalyptus is a strong candidate but it could show restrictions on the community edition.

I would appreciate hear about your experience and any advice related with what technology is best suitable for attending the new coming computational demands.

Thanks so much for your help,

Enabling network interfaces in a cloned Ubuntu-based appliance

Ubuntu is a widely used/deployed Linux-based distro. In fact, according to Linux Format (February - 2011), Ubuntu is the third most popular desktop OS.
However, network interfaces of an Ubuntu-based appliance exhibits an annoying behaviour when that appliance is executed on top of a different hardware from it was installed. This behaviour is also experimented when a virtual disk image is cloned and then used in a fresh virtual machine.
This short recipe shows how to overcome this awkward behaviour.

1. Be sure that the operating system is unaware of any network interface. Log into the appliance and execute the 'ifconfig' command. The image below shows only the loopback device.
   
   
   
2. Log into the 'root' account.
   
3. Execute the following command '> /etc/udev/rules.d/70-persistent-net.rules'. This command empties the '70-persistent-net.rules' file.
   
4. Now reboot your appliance and execute the 'ifconfig' command, again. You must see an output similar to the image below.