Recently I have been building out a SLES 11 environment and ran into some troubles with authentication with Likewise. I could join servers to the domain and everything would work great. Then some time later I would go to log in to the server only to get an "access denied" error. Upon logging in as root on the console and running "domainjoin-cli query", I would get this output:

someserver:~ # domainjoin-cli query

Error: Lsass Error [code 0x00080047]

40022 (0x9C56) LW_ERROR_PASSWORD_MISMATCH - The password is incorrect for the given username

When searching for "LW_ERROR_PASSWORD_MISMATCH", there are hardly any results. Most of the results I did get were just links to source code where that string was used as a constant.

After quite a bit of log hunting and deduction, I was finally able to figure it out and correct the error. Read on for the solution.

One part of my new build-out is configuring Samba on every host by default. It turns out that Samba was changing the machine account password every 7 days. Likewise and Samba use different databases for this information, and thus Likewise would have an invalid machine account password after Samba changed it.

The fix was to edit smb.conf on each system and add "machine password timeout = 0" under the "[global]" section. Then I simply restarted the smbd and winbindd daemons and re-joined the server to the domain.

The hard part was determining why the machine account password was changing in the first place. After digging in the Event Viewer on the domain controller for awhile I found the events involving the machine password change in the Windows Security log, under task category "Computer Account Management".

After figuring out what events related to the join breaking I was able to see that they were all 7 days apart. After then searching for "machine account password 7 days" I was able to find a reference to the samba configuration parameter for "machine password timeout". The default value for that parameter is, you guessed it, 7 days!

Read on for the download link and usage instructions.

UPDATE 2011-10-02: I am focusing on MineCraft for the next couple weeks. Hopefully I will have 2.0 released in this timeframe! Also the code has been moved to Git, and can be found at: http://git.thogan.com/mineplump/

Let me know if you are interested in contributing!

IMPORTANT COMPATIBILITY NOTE: I have not tested the old MinePlump available here since 1.5. Use at your own risk, BACKUP YOUR WORLDS!

UPDATE 2011-06-05: I have made a lot of progress on a major update to MinePlump. Wiring all the new code together has been delayed though, as I have been in the middle of moving. Here is a progress update:

1. Identify veins accurately. DONE! The logic and in-memory data structures are extremely accurate now! Deposits can have irregular shapes, overlap, and exist across chunk boundaries!
2. Plump or sprinkle options. DONE! Mineral quantity and frequency can be scaled bi-directionally now! These values can be set explicitly or as a multiple or factor of the number of minerals already in the map.
3. Post-plump analysis. DONE! You can get a graph of minerals by level by type, minerals by frequency per chunk, and simulated mining analysis! The simulated mining is epic, it will pick a few places in the world, simulate row mining at common depths, and report yields by mineral per 10 minutes!
4. Parameters tunable by mineral type. DONE! Each mineral has it's own set of parameters. AND these get saved with the plump log now, and can be exported as XML and loaded for use on another map!
5. Clear and start over. DONE! Actually this feature is a by-product of the way the new plumper works. It will either use existing mineral deposit coordinates as a starting point or pick coordinates at random. It was simple to add a clearing pass that removed everything first.

So, why isn't the update posted yet if all this is done!? Sorry, the boring parts still need to be tied up. Getting the GUI to invoke the whole process, error handling, making sure the cache flushes updates properly, other stuff like that. Feel free to check-out the source from subversion and peep all the new features, but no guarantees on what will happen if you run the program from trunk .

UPDATE 2011-05-06: I have a nice quiet weekend with nothing going on coming up. I will be taking the opportunity to make some big updates to MinePlump:

1. Identify veins more accurately, to reduce the number of large deposits that conservative plumping creates (2x2x2 still creates very large deposits sometimes)
2. Plump or sprinkle option, make veins larger or sprinkle more veins of target size around the underworld.
3. Post-plump analysis. Because I am likely to mess up the algorithm somewhere, and because it would take more time to make perfect than to do this other thing, I will be writing an analyzer that verifies that the map's mineral deposits now correspond with your input parameters. If it finds errors in this second pass it will attempt to correct them.
4. Parameters tunable per mineral type! Also, I will save the parameters in the plump log so that when you open the same world again your params come back.
5. If I have time left, I will add a underworld clear and re-create function.

UPDATE 2011-04-22: The beta compatible update is posted and I have tested it with 1.5_01! Come and get it!

UPDATE 2011-03-21: MinecraftForum user crackedEgg (Michael Sheppard) has updated MinePlump to support the new region format! I will be compiling his updates and updating the binaries as soon as I can.  Thanks Michael!

Downloads

BEFORE YOU RUN MINEPLUMP BACKUP YOUR SAVE FILES!

There is no undo, other than copying back your save data from a backup.

Launch MinePlump with Java Web-Start - MinePlump released under BSD license, see below.

Get the latest source code from Git: http://git.thogan.com/mineplump/

See what's going on in the code repository here: MinePlump Gitweb

The NBT I/O code I took from here: http://www.minecraftwiki.net/wiki/NBT_class.  My code is hereby released under the BSD license.  This means I don't care what you do with it, and I am not liable for any damages that arise from your use of it.  See the full license at the end of the post.

Contributors

• crackedEgg (Michael Sheppard) - Beta support, bug fixes

Bugs

• Mac users have reported some issues. Unfortunately I do not have a Mac to test on, so if a Mac user wants to grab the source and help me sort this out that would be great! Several Mac users have reported no problems, others have reported that it does not start.
• Blocks change to minerals when they are visible in some instances, even if the flip visible blocks options is deselected. The alogrithm that examines a block to see if it is visible only looks for air in adjacent blocks. Torches, minecart track, redstone wire, and other blocks that allow adjacent blocks to remain visible are not detected properly.

How To Use

1. Go make a backup copy of your Minecraft saves directory.
2. No really, go make a backup, you cannot un-plump.
3. Click the Open button. MinePlump will try to show you your saves directory automatically, but this has not been tested yet on non-windows platforms.
4. Select the level.dat file from the world you want to plump.
5. After the world loads, the path to the world's directory and the number of chunks it contains will be displayed. Also, all the plumping options will activate.
6. On the left, select the resources your want plumped.  Then select whether visible blocks should be included in the plumped veins.
7. In the middle, set the order of the plumping.  Minerals listed at the top get first dibs on turning non-mineral blocks into new mineral blocks.
8. On the right, set the target vein size.  To figure out what a vein will yield at a give size, calculate width * depth * height.
9. Set the minimum depth.  Only veins at or below this map level will be plumped.  The default is 45, safely underground for most cases.
10. Click the "Plumpify!" button!

After the progress bar fills, statistics regarding the operation will be displayed in the text box at the bottom of the window. Either that, or a bunch of dialog boxes will pop up reporting exceptions, but I have almost all of those ironed out now.

Remember this is really new, it doesn't bend over backwards to make sure your parameters make sense. Some inputs will destroy a map, PRACTICE ON WORLDS YOU DO NOT CARE ABOUT! BACKUP YOUR WORLDS!

License

MinePlump

Copyright (c) 2010 - 2011, Thaddeus J. Hogan
All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
• Neither the name of the Thaddeus J. Hogan nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

There is a patch in Xen 4.0 unstable that fixes the 'nomce' boot option so that it works properly.  Unfortunately this change has not been back-ported to Xen 3.4 by most of the Linux distros yet.

I have applied the fix to Xen 3.4.1 for OpenSUSE 11.2 and built RPMs for the affected packages.  This did the trick and my AMD 6100 series systems are now running Xen just fine with the 'nomce' boot option.  Read on for more details and the link to the RPMs with the fix.

Xen Panic Output

If you have an AMD 6100 series CPU and are getting the following output, then you are probably running into the MCE bug:

(XEN) Xen BUG at amd_nonfatal.c:165
(XEN) ----[ Xen-3.4.2  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff828c801778f9>] mce_amd_work_fn+0x1d9/0x1f0
(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
(XEN) rax: 0000000000000ffe   rbx: ffff828c8024ff28   rcx: 0000000000000000
(XEN) rdx: c0080ffe01000000   rsi: 0000000000000413   rdi: 0000000000000000
(XEN) rbp: 000000025f13f8e0   rsp: ffff828c8024fe60   r8:  ffff828c8028f800
(XEN) r9:  0000000000000000   r10: 0000000000000005   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: ffff828c80177720   r14: ffff83081fd7b190
(XEN) r15: ffff83081fd7b190   cr0: 000000008005003b   cr4: 00000000000006f0
(XEN) cr3: 00000004ca4a6000   cr2: 000000000083c770
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff828c8024fe60:
(XEN)    0000000000000000 c0080ffe01000000 ffff828c80221180 ffff828c8011a12c
(XEN)    ffff8300dfc2c060 ffff828c80221180 ffff83081fd7b198 ffff828c8011a20d
(XEN)    000000024ab06880 0000000000000000 ffff828c8024ff28 ffff828c80267900
(XEN)    ffff828c80266900 0000000000000000 ffff828c80221100 ffff828c801185b8
(XEN)    000000000000e008 ffff828c8024ff28 ffff828c80266900 ffff828c802215b0
(XEN)    000000025e3b7f20 ffff828c80138fcc 0000000000000000 ffff8300dfafc000
(XEN)    ffff8300dfc2c000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000246
(XEN)    0000000000000008 00000000ffff8e54 0000000000000054 0000000000000000
(XEN)    ffffffff802053aa 0000000000000001 0000000000000000 0000000000000001
(XEN)    0000010000000000 ffffffff802053aa 000000000000e033 0000000000000246
(XEN)    ffffffff80511f50 000000000000e02b 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff8300dfafc000
(XEN) Xen call trace:
(XEN)    [<ffff828c801778f9>] mce_amd_work_fn+0x1d9/0x1f0
(XEN)    [<ffff828c8011a12c>] execute_timer+0x2c/0x50
(XEN)    [<ffff828c8011a20d>] timer_softirq_action+0xbd/0x2e0
(XEN)    [<ffff828c801185b8>] do_softirq+0x58/0x80
(XEN)    [<ffff828c80138fcc>] idle_loop+0x4c/0xa0
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at amd_nonfatal.c:165
(XEN) ****************************************

Xen Community Discussion

Since these CPUs are just starting to hit the market, I was not able to find a lot of details on this issue.  The only real discussion I was able to find was this thread on the [Xen-devel] mailing list:

[Xen-devel] (XEN) Xen BUG at amd_nonfatal.c:165 on a new amd g34 board

There is a patch attached in the thread and it works.  For those not wanting to get into building the package from source I have created some patched RPMs for all the platforms I currently have access to.

Binary Package With Fix

I have built RPMs for the current stable version of OpenSUSE (11.2) and will be building packages for SLES 11 shortly.  I will then be back-porting to OpenSUSE 11.1 and posting those RPMs within a week.  Maybe I will do OpenSUSE 11.0 and SLES 10.  Leave a comment if you need one of these platforms and, if there is demand, I will build packages for them.

Using The Patch

Untar the archive with the proper RPMs for your platform.  Install the package with 'zypper', it will fetch any dependencies required from your configured repositories:

# zypper install xen-3.4.1_20360_04-2.1.x86_64.rpm \
    xen-tools-3.4.1_20360_04-2.1.x86_64.rpm \
    xen-libs-3.4.1_20360_04-2.1.x86_64.rpm

After you have installed the patched RPMs, you need to add the 'nomce' boot option to your Xen entry in GRUB.  It should be on the line that reads 'kernel /xen.gz ...'  Below is an example from one of my patched hosts:

title Xen -- openSUSE 11.2 - 2.6.31.12-0.2
 root (hd0,0)
 kernel /xen.gz nomce noreboot com2=115200,8n1 console=com2
 module /vmlinuz-2.6.31.12-0.2-xen root=/dev/rootvg/rootlv
 module /initrd-2.6.31.12-0.2-xen

OpenSUSE 11.2 Packages

These RPMs are based on the latest (as of 2010-09-06) source RPM package for 'xen'.  I have artificially inflated the version number so that it will be applied as an upgrade to an existing install if necessary.  This may interfere with future official OpenSUSE updates to this package, and if so, will need to be removed manually and re-installed.

I will try to maintain up-to-date versions of these RPMs as long as official OpenSUSE updates come out that do not include the patch.  Check back often as I will setup a mailing list this week to handle notifications on when I update these packages.

Simply install the RPMs included with 'zypper', it will properly fetch the dependencies from your other repositories.

Xen 3.4.1 for OpenSUSE 11.2 with 'nomce' option fix.

Novell SUSE Linux Enterprise Server 11 Packages

My day job runs SLES 11, so I will have an opportunity to build SLES 11 packages with the fix this week.  Check back after 2010-09-09 for these packages.

UPDATE: Things have been busy at work and I have not gotten around to building a SLES 11 package.  I am probably not going to build one unless someone asks for it, so if you need it, leave a comment!

I have been lazy about updating and have a lot of content to add in the coming weeks.  I really liked the WordPress post editor, and considering that I will be doing a lot of writing soon, that was important.  Also I just like to try out new stuff, so here we are!

This turned out to be because of my DNS configuration.  The cause of the problem surprised me, because it was so decoupled from the mail setup.  Hit the jump for the full story and solution.

My mail setup consists of three sites.  My primary site is a colocation facility, where my primary MX (farmx.thogan.com) and my main mail server (mail1.thogan.com) reside.  The second site is a collection of virtual servers in the Rackspace Cloud.  My backup MX resides here (cloudmx.thogan.com).  The third site is my home, where my backup mail server resides (mail2.near.lan).

The three sites are all have their own private networks in addition to their public addresses.  These private networks are all joined via VPN.  For the purposes of this problem, the type of link dosen't really matter.  In my case it is a VPN, but this applies to any multi-site mail configuration.

So, back to my current situation.  My primary site is offline while I move the server to a new colocation facility.  Mail is being delievered through my backup MX at the cloud site, and from there is being relayed to my backup mail server at my home site. Two days after taking my primary site offline for the move, my router at home died due to hardware failure.  Now to only operating site is the cloud site, with my backup MX.  I noticed the failure of the home site while I was at work, and expected that it would not be a problem.  Mail should have just queued up at the backup MX until I fixed the router at home, re-established the VPN connection, and flushed the mail queue.

Once I fixed the router, I logged into the backup MX and ran `postqueue -p`.  Surprise!  Nothing.  I examined the mail logs and saw that mail was not being deferred, but rather was bounced all day long!  But, that's not how it was supposed to work!

Turns out the problem was DNS.  Each site had a DNS server that would resolve names for the internal domain names (near.lan, far.lan, cloud.lan).  When I configured the near.lan zone on the cloud site's DNS server, I set it up to forward the queries to the home site's DNS server.  It was not a slave for that domain, just a forwarder.  And there was my mistake.

When mail would come into the backup MX at the cloud site, it would attempt to deliver it to mail2.near.lan.  When postfix went to lookup mail2.near.lan in DNS, Bind would attempt to query to home site's DNS server for the name.  When the VPN link went down, the cloud site's DNS server responded to the queries for mail2.near.lan with NXDOMAIN.  So postfix bounced the message instead of deferring it, as it would if it had resolved the name and received a connection timeout trying to contact the home site mail server.

The moral of the story is, in a multi-site setup, internal domains should be slave zones on remote DNS server, NOT forwarded zones!  If I had setup near.lan as a slave zone in the cloud site's DNS server, then postfix would have been able to resolve the name, failed to connect to that address, and deferred the message.

BEFORE YOU READ THIS: Read the XenNetworking page at Xen Wiki.  (http://wiki.xensource.com/xenwiki/XenNetworking).  Go through the whole thing, as it hammers out the basics of how networking in Xen works and will make the rest of this document make sense.  What I am presenting here is an example based explanation of the concepts, but not concepts themselves.

PLATFORM: This document is Debian oriented.  Specifically I am using Debian 5 (Lenny) and Xen 3.2.  It should apply to other Debian based distributions fairly well but I have not tested it.  If you have the time and the knowledge, please consider porting this document to other distributions such as the RedHat, CentOS, Fedora family!  (I probably will not be running Xen on those platforms anytime soon)

INSTALLATION: I am assuming that you have successfully installed Xen and booted the Hypervisor and dom0.

CONVENTIONS: I quote literal commands and file content if they are used within a sentence.  The quotes are not part of the file content unless specified.

Default Configuration

The default bridging configuration is defined in /etc/xen/xend-config.sxp using just the line "(network-script network-bridge)".  If you boot dom0 with eth0 configured as usual and the default bridging line uncommented in xend-config.sxp then you should have network connectivity as defined for eth0 in /etc/network/interfaces.

NOTE! The default xend-config.sxp states that the default bridge name is "xenbr0".  This was not the case after I enabled the default bridging config.  The physical interface eth0 was renamed to peth0 and the bridge was given the name eth0.  This is not consistent with how bridging is explained in the XenNetworking wiki page or in the configuration file.

From the config file:

# Your default ethernet device is used as the outgoing interface, by default.
# To use a different one (e.g. eth1) use
#
# (network-script 'network-bridge netdev=eth1')
#
# The bridge is named xenbr0, by default.  To rename the bridge, use
#
# (network-script 'network-bridge bridge=<name>')

Do an "ifconfig -a" and review the interfaces that now exist on the system.  You should see what appears to be your original interface(s) and then several veth and vif interfaces:

xen0:~# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:1e:2a:cc:60:c5
          inet addr:192.168.1.4  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:2aff:fecc:60c5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:101 errors:0 dropped:0 overruns:0 frame:0
          TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:13517 (13.2 KiB)  TX bytes:7115 (6.9 KiB)

eth1      Link encap:Ethernet  HWaddr 00:0c:76:ad:c5:f5
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:23 Base address:0xc000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:560 (560.0 B)  TX bytes:560 (560.0 B)

peth0     Link encap:Ethernet  HWaddr 00:1e:2a:cc:60:c5
          inet6 addr: fe80::21e:2aff:fecc:60c5/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:173 errors:0 dropped:0 overruns:0 frame:0
          TX packets:50 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:25945 (25.3 KiB)  TX bytes:7655 (7.4 KiB)
          Interrupt:19 Base address:0x4f00

veth0     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

veth1     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

veth2     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

veth3     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

vif0.0    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

vif0.1    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

vif0.2    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

vif0.3    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

What Exactly Happened?

In Short

When Xend started it did the following:

1. Renamed eth0 to peth0
2. Created a bridge called eth0
3. Added peth0 to the bridge called eth0
4. Ran "ifup" using the bridge name as the interface name.

The Whole Story

As it is explained in the XenNetworking page, a bridge called xenbr0 is created.  The interface vif0.0 is added to the bridge and then veth0 is renamed to eth0.  This was not the case with my Debian 5 / Xen 3.2 setup, so let's take a look at what we actually have:

xen0:~# brctl show
bridge name     bridge id               STP enabled     interfaces
eth0            8000.001e2acc60c5       no              peth0
xen0:~# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:1e:2a:cc:60:c5
          inet addr:192.168.1.4  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:2aff:fecc:60c5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:923 errors:0 dropped:0 overruns:0 frame:0
          TX packets:199 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:147869 (144.4 KiB)  TX bytes:35249 (34.4 KiB)

So, the bridge is called eth0, and it has the ip configuration that was specified for eth0 in the /etc/network/interfaces file:

allow-hotplug eth0
auto eth0
iface eth0 inet static
        address 192.168.1.4
        netmask 255.255.255.0
        gateway 192.168.1.1

How did the bridge get the IP information for eth0 exactly?  It seems that xen runs the operating system's own "ifup" script to reconfigure the bridge with the ip information that the physical interface originally had.  We can test this by commenting out the line "auto eth0" and rebooting.

After having commented out the "auto eth0" line and rebooting, one would expect that the eth0 interface would not be up after the boot completes.  However you will find that after Xend starts it comes up and is a bridge.

So now we know that Xend runs the ifup command for the bridge interface after it renames the physical interface by prefixing it with a 'p' and creates the bridge.

Renaming The Bridge

Make sure you do this from a VGA or serial console because it will break your network connectivity.

The next step for examining how the bridge works is renaming it.  Edit the /etc/xen/xend-config.sxp file as follows:

Change: (network-script network-bridge)
To: (network-script 'network-bridge bridge=abcd0 netdev=eth0')

Then reboot.  After you get to the login prompt, you should see this error message from the boot sequence when Xend started: "Ignoring unknown interface abcd0=abcd0".  Login and test for network connectivity, it should be broken.  Run the route command, there should be no routes.

Explanation Of Changes: The "bridge=abcd0" parameter specifies the name of the bridge interface.  The "netdev=eth0" parameter specifies the name of the physical interface to use for the bridge.  You must specify the netdev parameter because otherwise Xend will assume that the physical device name is the same as the bridge name and attempt to create pabcd0 as the physical interface.

There will now be a bridge interface called abcd0.  It will have the IP information copied from eth0 when it was renamed peth0, but it is not up.

abcd0      Link encap:Ethernet  HWaddr 00:1e:2a:cc:60:c5
          inet addr:192.168.1.4  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:2aff:fecc:60c5/64 Scope:Link
          BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:923 errors:0 dropped:0 overruns:0 frame:0
          TX packets:199 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:147869 (144.4 KiB)  TX bytes:35249 (34.4 KiB)

Manually up the interface with the command "ifconfig abcd0 up".  Test for connectivity again, it should be restored.  Test against a local network host because there will not be a default gateway defined.

Configuring The Bridge Properly At Boot

Now that we have renamed the bridge and demonstrated that it does not get brought up or have routes for it added by Xen, we need to go about configuring the /etc/network/interfaces file to handle the ifup that Xend calls for the bridge.

In /etc/network/interfaces, change the interface name from eth0 to abcd0 so that the section for that interface looks like this:

iface abcd0 inet static
        address 192.168.1.4
        netmask 255.255.255.0
        gateway 192.168.1.1

Reboot the server and verify network connectivity.

Why Rename The Bridge?

First of all, I think that giving the bridge the same name as the physical interface is confusing, especially when you are first presented with all the new interfaces.  Also when you want to add additional bridges, let's say to a dummy interface to create a private network, giving the bridges their own naming nomencalture helps maintain consistency when building bridges for different interface types.

Secondly, the default configuation file's comments are not correct about the behavior of the scripts.  Going through the exercise of renaming the bridge helps demonstrate how it actually works.

Xen Virtual Interfaces

So far we have avoided the other half of the Xen networking equation, which are the virtual interfaces.  The XenNetworking page describes an array of interconnected virtual interfaces.  There are the vif* interfaces which are all present in dom0.  Vif0.* is bound to the veth* interfaces in dom0, and vif1.*interfaces are bound to eth* in dom1, and so on.  To demonstrate how these work in dom0 in a bridging configuration, I'm going to remove the IP configuration from the bridge then attach veth0 to the bridge for network connectivity.

Update the /etc/network/interfaces file to bring up the bridge (abcd0) without an address.  Then configure the interface veth0 to have your IP information with some extra commands to manage its vif0.0 counterpart.  These changes aren't very intuitive so bear with me and I will explain it all below:

iface veth0 inet static
        pre-up ifconfig $IFACE hw ether 00:1e:2a:00:00:01
        pre-up brctl addif abcd0 vif0.0
        post-up ifconfig vif0.0 up
        address 192.168.1.4
        netmask 255.255.255.0
        gateway 192.168.1.1
        post-down brctl delif abcd0 vif0.0

iface abcd0 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        down ifconfig $IFACE down

After making these changes, reboot the server.  Make sure you are on a physical console because this will break networking again.

Let's start with the configuration of the bridge, abcd0, at the bottom of the configuration above.  The "up" line is a literal command to execute when ifup is called, and the "down" line is a literal command to call when ifdown is called.  Running ifconfig and specifying an address of 0.0.0.0 removes an address if the interface at one point was configured with one, since downing the interface does not remove the address.

The result of this configuration is that after a boot you should have the bridge interface UP and WITHOUT an address:

xen0:~# ifconfig abcd0
abcd0     Link encap:Ethernet  HWaddr 00:1e:2a:cc:60:c5
          inet6 addr: fe80::21e:2aff:fecc:60c5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:282 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:53239 (51.9 KiB)  TX bytes:594 (594.0 B)

According to the XenNetworking wiki page, the vif interfaces are where packets enter the Xen networking system, so this is where we will manipulate the connection to the physical network.  The veth interfaces are attached to the vif interfaces and are the interfaces that the OS can use for network connectivity through Xen.

In the configuration for interface veth0, there are a few lines that inject custom commands into the interface configuration process as performed by ifup and ifdown.   Let's start with the line "pre-up ifconfig $IFACE hw ether 00:1e:2a:00:00:01".  This assigns a MAC address to the veth0 interface BEFORE the interface is configured with an IP address and actually raised.

After veth0 has a MAC address, we need to add it's counterpart vif0.0 to the bridge.  This gets the packets from the bridge into the Xen networking system for delivery to veth0.  The line "pre-up brctl addif abcd0 vif0.0" perfoms this task.

At this point there are no more pre-up commands to execute, so ifup will configure the veth0 interface with the specified parameters (address, netmask, gateway) and then bring veth0 up.  Now veth0 is configured and vif0.0 is in the bridge, BUT vif0.0 is still down.  So after everything else is configured, which is what "post-up" means, we then can manually bring up the related vif0.0 interface with this line: "post-up ifconfig vif0.0 up".

Since all of this is wrapped into the interface configuration, you should just be able to run the command "ifup veth0" to restore networking.

Now lets look at our interfaces.  The bridge will be in the same state, up and without an address.

Interface vif0.0 should also be up and have no address:

xen0:~# ifconfig vif0.0
vif0.0    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:101 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1285 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:13951 (13.6 KiB)  TX bytes:232096 (226.6 KiB)

Interface veth0 should be up and configured with an IP address:

xen0:~# ifconfig veth0
veth0     Link encap:Ethernet  HWaddr 00:1e:2a:00:00:01
          inet addr:192.168.1.4  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:2aff:fe00:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1456 errors:0 dropped:0 overruns:0 frame:0
          TX packets:126 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:268331 (262.0 KiB)  TX bytes:16937 (16.5 KiB)

And the bridge abcd0 should show that vif0.0 is a part of it now:

xen0:~# brctl show
bridge name     bridge id               STP enabled     interfaces
abcd0           8000.001e2acc60c5       no               peth0
                                                         vif0.0

Bring Up veth0 On Boot

Finally, if you want to keep the network configuration this way, you'll need a way to raise veth0 on boot after Xend starts.  So what better place than in a custom network-bridge script?

Below is a script I created at /etc/xen/scripts/network-bridge-custom.  It is similar to many multiple bridge configuration examples you will find, though it only specifies the one bridge (abcd0) at this time.  I then tweaked it to bring up veth0 after it creates the bridge.

#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=abcd0
/sbin/ifup veth0

Make sure that if you create a custom script like this that you give it the proper execute permissions.

To use the new script, reference it in /etc/xen/xend-config.sxp instead of the default network-bridge script:

Change: (network-script 'network-bridge bridge=abcd0 netdev=eth0')
To: (network-script network-bridge-custom)

Now when Xend starts it will call the custom script and raise veth0 after configuring the bridge!

Conclusion

I hope that this helps explain bridged networking in Xen in a more thorough way than a regular howto or tutorial.  This is not intended to be a practical application but instead an explanation of each part of Xen networking integration in regards to bridges.

If you see any errors or can contribute clarifications, let me know! thaddeus@thogan.com

Finally, a fresh installation of Xen 3.2 on Debian 5 seems to have different default networking behavior than described in most other documentation I have read.  Since I am just getting started with Xen I am going to assume that there were recent changes that obsoleted some documentation.

Please feel free to copy this document or any part of it for other documentation if you find it useful.  Also kindly give me the bump if you do!

It has been awhile now since I have lost my animosity toward people who destroy computers that I administer.  Mostly because I get paid money to spend half my time at work un-breaking and tidying *NIX servers.  It also means that I have a particular attraction to anything that helps me clean up the users' mess more quickly.

Enter: My home media PC.

This is a computer attached to the TV in my living room.  Everyone who uses it is 90% likely to be intoxicated and know nothing about computers.  When we get back from the bar or just party at home this thing ends up being used to troll YouTube and email until the wee hours of the morning.  It's 3:30am and your drunk friend checks his email and gets a link to download sweetvideo.exe.  "Sounds GREAT!"

So I got sick of constantly rebuilding the computer.  Linux storage server + LVM snapshots + gPXE + iSCSI boot-from-san Windows solution after the jump.

First of all, Linux was not an option for this computer.  We do not have cable, so all the Silverlight, Window Media, Netflix, and other proprietary DRMed video players have to work.

The solution was Windows XP, booting from the network off of an iSCSI target, where the exposed LUN is an LVM snapshot of a fresh, virus-free, and fully patched installation.

Storage Is What I Got

I have long been using LVM and a large disk array for general storage management.  My main storage server has two volume groups: rootvg - a pair of mirrored 750 GB drives for the OS, and datavg - a 4.2 TB storage volume.  The 750 GB drives are of that size because that is what I had on hand.  In GOOD ADMIN STYLE, I used only what I needed of them in my system LVs.  I expected that maybe later I could use the rest of the space for something else.

So with 600 GB of free space on a mirrored volume group, I set out to find a way to install Windows on it, export the volume, and boot across the network on it.

gPXE and iSCSI

The key to everything I have running now is gPXE (http://etherboot.org/wiki/index.php).  This sweet PXE program has the key ability of attaching to an iSCSI device at boot time.  The HOW-TOs on the site are flawless.  Just make sure to pay attention to the special instruction for Windows XP (as opposed to 2003) that involves running a script to enable iSCSI booting.  (Relevant Page)

I followed the HOW-TO instructions and installed a Windows XP then converted it to iSCSI boot and transfered the disk image to a logical volume on my storage server's "rootvg".

IMPORTANT NOTE! In my first attempt I installed Windows XP and applied all available patches, SP3, and IE8.  Then installed AVG and several other goodies.  I NEVER GOT THIS TO iSCSI BOOT!  It only worked when I converted a fresh install to iSCSI boot then applied all patches and goodies.

iSCSI, LVM Snapshot, and DCHP Config

My storage server is running Ubuntu 9.04.  The relevant package for using it as an iSCSI target that I installed is "iscsitarget".

After installing Windows and patching it, plus installing relevant software, I made an LVM snapshot:

  --- Logical volume ---
  LV Name                /dev/rootvg/medialv
  VG Name                rootvg
  LV UUID                p2Kqzg-aSn4-31TD-2CsU-1jeX-ugQ5-E0ZfNU
  LV Write Access        read/write
  LV snapshot status     source of
                         /dev/rootvg/mediasnap [active]
  LV Status              available
  # open                 0
  LV Size                32.00 GB
  Current LE             8192
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:6

  --- Logical volume ---
  LV Name                /dev/rootvg/mediasnap
  VG Name                rootvg
  LV UUID                pB7ftO-9BAv-i6eA-N91E-PmYg-BIXN-0Bc30y
  LV Write Access        read/write
  LV snapshot status     active destination for /dev/rootvg/medialv
  LV Status              available
  # open                 1
  LV Size                32.00 GB
  Current LE             8192
  COW-table size         32.00 GB
  COW-table LE           8192
  Allocated to snapshot  2.64%
  Snapshot chunk size    4.00 KB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:7

The snapshot is the same size as the original volume.  This allows unlimited modification to the snapshot volume.  You can probably get away with making the COW table 10% or so of the original volume size, but I had the space available.

I then exposed this volume via iSCSI.  Here is the relevant content in my ietd.conf file:

Target iqn.2009-01.lan.near:media.host0
        Lun 0 Path=/dev/rootvg/mediasnap,Type=blockio

I built the gPXE image for PXE chainloading per the site's instructions and then configured my DHCP server to shell out this image then instruct gPXE to iSCSI boot the Windows XP install.

Here is the relvent section of my dhcpd.conf file:

host media {
        hardware ethernet 00:19:21:a6:70:ef;
        fixed-address 192.168.5.202;
        option host-name "media";

        if exists user-class and option user-class = "gPXE" {
                filename "";
                option root-path "iscsi:192.168.5.3::::iqn.2009-01.lan.near:media.host0";
        } else {
                filename "undionly.kpxe";
        }
}

And Now, Simple Recovery

Now the media PC boots Windows XP off of an iSCSI target, which is a snapshot of a clean install.  I no longer need to take potentially restrictive measures to protect that PC (those that may limit functionality).  If I need to rebuild, I remove the "mediasnap" logical volume snapshot and recreate it.  Then reboot the media PC.  All done.  Five minutes later I have a clean media PC again!

Windows 7

I also have installed Windows 7 against an iSCSI target for use on my desktop for games.  Windows 7 is much easier to install on iSCSI, as it supports installation directly onto the target and requires no fiddling to get it to boot.  Just keep in mind that with gPXE you will need to add the "gpxe.keep-san" option to your DHCP config in order to install directly onto the iSCSI target.  This will involve adding configuration for the "gpxe" namespace to your DHCP configuration. (I will link to details in an edit soon).

EDIT: Here is the link. (http://www.etherboot.org/wiki/dhcpd) You will need to add the gpxe specific DHCP option definitions to your DHCP config to use those option names later in the config file.

I was in the process of shutting down all of my virtual hosts because the GUI hung while I was editing the properties of one.  When this happens you can no longer modify the properties of any VM without shutting them all down and killing off all VirtualBox processes.  (VirtualBox bug #443)

This sequence of VirtualBox hanging while creating a new VM, then having to restart everything happens every time I create a VM.  This is not something that only happens occasionally.  Becoming more frustrated every time this happens I took the risk of upgrading to VirtualBox 2.2.0.

Sadly, I started having problems with virtual hosts that would only stay up a few minutes, if even boot, then hang.  Also, these hung virtual hosts would enter an inconsistent state and a full cycle of kill off all VirtualBox processes and restart everything was required.

Thankfully the bug regarding this issue was easy to find.  I guess it is a problem with the PCNet virtual NICs.  This is documented here http://www.virtualbox.org/ticket/3676 (VirtualBox bug #3676).

I shut down everything again and changed all of my virtual hosts to use the Intel PRO/1000 MT Server virtual NIC.  This worked allright but the Smoothwall virtual host didn't like it.  I tried again with everything switched to the Intel PRO/1000 MTDesktop virtual NIC.  Bingo, everything was happy and stayed up just fine.

Despite my disappointment with the upgrade, it would seem that the problem with the GUI hanging everytime I try to create a virtual host is now resolved.  When I accidently closed the SSH session carrying the VirtualBox (2.2.0) remote X window, all virtual hosts had to be restarted.  But I have yet to have the client just hang on me like it used to.

I thought I would have a nice long article to write about this.  Something complete and detailed to fill the void of information I found when looking for instructions myself.  Now I think I understand why there was nothing to be found on this topic; there really is nothing to it.

The build platform was an HP DL380 with Q-Logic 24xx cards connected to an EMC CLARiiON array.  I'm not sure what model, I just ask our storage engineer to give me LUNs and LUNs I get, but I do know it is a CLARiiON.  I asked our storage engineer for 20 GB and four paths, two HBAs each with access to two storage processors.

In other boot from SAN builds we usually have to expose only one path to the system during the install because we have had problems with the installer / LVM getting confused when there are multiple active paths.  Just for reaffirmation, I tried the Ubuntu install with all paths exposed.  The install was successful and there was no manual intervention required to make it through the first boot.

After the install completed I did an 'apt-get dist-upgrade' to bring all packages and the kernel image current.  Then I did an 'apt-get install multipath-tools multipath-tools-boot' and rebooted.  Up it came with LVM referring to the multipath device for its physical volumes.  That was mundane.  There was one hiccup in the first boot after installing multipath where fsck couldn't check /dev/sda1 because it was already nabbed by multipath.  I just commented the entry out in /etc/fstab and all was well on the next reboot.  I usually leave /boot unmounted just as a paranoid safety measure, but I assume changing the /etc/fstab entry to refer to the multipath device would fix the fsck problem too.

We proceeded to copy files to the system while pulling and replacing fiber cables and all worked as planned.  Just for fun to end the day we dezoned all the LUNs on the storage processors while copying files to it and watched the running system slowly fall apart.  The really quite amazing thing though, is that we re-mapped the LUNs to the system and it came back to life without a reboot!  My scp that had stalled out started moving again and finished up and all the multipath failure messages flushed to the system logs.  I have to admit that I didn't expect the system to survive losing all of its I/O for ten minutes =D