2013-12-26

Installing ESXi with Razor

This is part #6 of the Razor series. In the last post we looked at what lies within the installer. In this post we will go more into detail of how exactly the installation process works. Fort his post we will run through the deployment of an ESXi host.

1. Installing Razor - Yes the New Version..
2. Razor - DHCP and TFTP
3. Installing the Razor client and creating a repository
4. Installers, Policies and Tags
5. Razor - What Lies Within the installer?
6. Installing ESXi with Razor

Since there has been some time from the last part of this series I will recap what you should have in your environment.

We have a repository named ESXi_5_1 for our ESXi installation.

repository

We have a policy named ESXi51 defined that will deploy ESXi to the node.

policy

And we have a tag (which is what the policy above uses) named Test_tag used to classify the node.
(this is the minimum that will allow us to install ESXi - 2vCPU's and 4GB RAM)

tag

We then power on a node. It boots up, receives an IP from the Razor server and then continues to boot to the microkernel.

node boot

In the meantime on our Razor server we see that a new node "node3" has been recognized.

razor nodes

razor nodes

Looking at the details for that node - we will initially only see the following information:

  • mac address
  • uuid
  • log location

razor nodes node3

razor node details

Looking at the log for the node:

razor nodes node3 log

razor node log

When the node is matched against a policy - it will reboot. This can be seen in two places.

1. Looking at the node information again - here you will there is far more information about the hardware. This is because the microkernel has now reported back to Razor with all the information.

razor nodes node3

after report from microkernel

You can see that the node was matched against a policy.

policy bind

The tag was also correctly recognized.

tag match

2. Looking at the log you will see additional entries. The 3rd entry shows that the node is now bound to the ESXi_5_1 policy. The 4th entry reports that the node has now rebooted - as a result of the policy.

razor nodes node3 log

node log

Back to the node. We now see the node connecting to the ESXi repo

boot to ESXi installer

Looking at the node log again on the Razor server - we will see the process of the installer.

log update

And from the node's perspective.

installing..
installing..installing..
installing..

There are two things I would like to point out here - and the information here is current only for the time of this article - because the product is constantly evolving - things could change.

Taking another look at the nodes logs we can see the following:

razor nodes node3 log

another look

The first entry gives you the URL of the kickstart file that will be used. The contents of that file will show that the password is already populated (encrypted of course) and that there is a API call to report that the kickstart is finished. This of course can be accessed through the API with a web browser .

kickstart file from the API

There are no network settings and no hostname settings defined in the kickstart. This is not supported/functioning in the current release.

The second thing you might notice in the log is that the second entry will already report "stage_done" almost immediately after the ks.cfg file is retrieved. In reality the process is far from being completed - it has only really started - because a good part of the ESXi installation is done in the %firstboot section.

The reason that this is done is because - the node needs to boot from its local disk to complete the installation (%firstboot), if not it will go into an infinite loop and never complete the next phase.

So even if you log says that the installation has finished - it will still continue through the installation process.

As a side note - there is a problem using this method when installing ESXi 5.0, wget on that busybox version does not know how to exit properly when retrieving an empty page (more info can be found here).

Here - in theory - you could pass the work over to an external broker - but currently there is no working example for ESXi.

Install Complete

This is where Razor's job finishes with the node - but there are several points that are missing - two of them for me were critical.

  1. This method does not support static IP addresses. I do not know of many people who deploy their hosts with dynamic IP's. I certainly do not.
  2. There is no support for configuring the hostname of the node - which means this needs to be taken care of in another way.

    I will say that I have found a way to circumvent both of these problems - but that will come in an additional post.

    DHCP and no hostname

I hope you have managed to stay with me until now - and managed to digest most of this. It is a lengthy process but I have tried to be as detailed as possible to explain the process better.

The next part will provide a run through of installing a Linux node and passing that over to Puppet for configuration.

2013-12-16

0-Day Deployment - My Presentation at the #UKVMUG

A few weeks ago - I gave a session about how to deploy a full environment starting from a bare rack to a fully functioning environment with deployed applications.

Firstly about the VMUG event. I was fortunate enough to be invited by the VMUG organization to present at this years VMUG. I would like to extend my thanks to the VMUG organization for their hospitality and the effort they made to help feel at home (including kosher food)

There were approximately 400-500 people at the Event, the location was very nice and the the organization of the event was very well done.

The lineup of speakers, sponsors and vendors was very impressive, the local VMUG leaders and VMUG organization did an outstanding job.

Community was a big part of the event, just the way it should be.

I got to chat a bit with Joe Baguley, the VMware CTO, EMEA - about competition, "co-opetition" and what will happen in the future. We can never be prophets, but even if only half of what we talked about does actually come to be, then amazing things will happen.

I highly enjoyed the keynotes, both from Joe Baguley, and Greg Ferro - who surprised me with the amount of information he knows about ducks, and agriculture. They are both outstanding speakers.

Back to the session. Firstly I would like to thank the brave souls that attended my session. There were not many of you, because the slot I got was the same as three other VMware speaker sessions, one by Massimo Referre, the other by Cormac Hogan, and the last was a panel with Mike Laverick and Joe Baguley.

It not uncommon that one has to stand up an environment - of 8-50 hosts and 50-250 virtual machines. But is not quite common that you have to do this - when you have no existing infrastructure at all. This was the case I was presenting. Standing an infrastructure from complete zero to a fully deployed virtual VMware environment - in the most automated fashion possible.

There are a lot considerations that you have to take into account, for example - Autodeploy is more or less out of the question - seeing that you have no vCenter environment up and running yet.

How do you get vCenter up? How to deploy VM's? How do I configured the underlying storage array, the network switches, so many questions and dependencies.

The process is not ideal, not fully automated, but it will become so - not too far in the future.

In the meantime - here is the slide deck from my presentation.

Hope to see you all at another VMUG event soon!

2013-12-12

OpenStack Israel 2013 #OpenStackIL

This week at the OpenStack Israel conference there was a impressive turnout of approximately 400 people. It was a vibrant day, lots of interesting people from all over the world, and great sessions. I even got to have lunch with Scott Herold!

Here are some of the moments from day

I delivered a session "OpenStack for VMware Admins" going over the differences, similarities and some gotchas from those who are used to VMware environments, and are looking at OpenStack

And we're back with three stellar tracks @maishsk from @cisco on #openstack for #vmware admins #openstackil pic.twitter.com/Uv5cfDiK6V

Here are pictures from the day.


Created with flickr slideshow.

I think that the Israeli team did a great job, in attracting the talent to come to the conference. I was happy that I could present and am looking forward to seeing the recordings when they come online.

My session is available online and embedded below.

2013-11-28

Razor - What Lies Within the installer?

This is part #5 of the Razor series. In the last post we looked at Policies, Tags and touched slightly on Installers. In this post we will go more into detail into what exactly the installer does.

1. Installing Razor - Yes the New Version..
2. Razor - DHCP and TFTP
3. Installing the Razor client and creating a repository
4. Installers, Policies and Tags
5. Razor - What Lies Within the installer?
6. Installing ESXi with Razor

The installers are located in the /opt/razor-server/installers directory. In this directory you will find a yaml file and folder - both with the same name.

installer location

During this post - we will go through 2 separate installers, the Redhat one and the ESXi installer. The yaml file contains the following

redhat.yaml

The contents are quite self-explanatory, os_version, label, description and lastly the boot_sequence. The boot_sequence is the set of instructions that will be passed to the microkernel, the first time (1:) - boot to the installer (boot_install), and on the other boots (default:), boot to the local disk (boot_local) and bypass the microkernel.

The folder contains the following files (remember the os_version - therefore all the files are under the 6 subfolder).

redhat folder files

These files are the steps that will be executed, and as we saw in the redhat.yaml file the first boot will go to a boot_install.erb

boot_install

This file holds the correct boot parameters for booting into the installation - including the contents of the kernel_args template which we will see shortly. There is also information echoed out to the screen such as the Installer Label, the node's url, and the repository's url.

kernel_args

The kernel_args - has the pointer to the kickstart.erb file and the instruction to boot from the network device.

kickstart

The kickstart.erb file is mostly a standard kickstart file - with a few variables.

  • url=<%= repo_url %> - will be translated to the url for the repository.
  • rootpw <%= node.root_password %> - will be the password that you defined when you created the policy
  • network --hostname <%= node.hostname %> - will be the hostname according to the pattern that you defined in the policy.

After the rest of the standard kickstart contents (in the %post section), there is some more razor stuff.

kickstart - %post

  • %post --log=/var/log/razor.log - will log the rest of the process to /var/log/razor.log
  • curl -s -o /root/razor_postinstall.sh <%= file_url("post_install") %> - the node retrieves the post_install file and saves it to /root/razor_postinstall.sh
  • # Run razor_postinstall.sh on next boot via rc.local - this section is self explanatory.
  • curl -s <%= stage_done_url("kickstart") %> - here a API call is sent to razor to announce that kickstart is finished - and therefore the subsequent boots will go to local disk (as was defined in the redhat.yaml file)

post_install

The post_install.erb (the one that was saved to /root/razor_postinstall.sh) has a few more operations that are performed.

  • # Wait for network to come up when using NetworkManager - waits for the network to come up.
  • <%= render_template("set_hostname") %> - is the piece of code that sets the hostname.
  • <%= render_template("store_ip") %> - is the code that stores the IP address and reports back to razor server.
  • <%= render_template("os_complete") %> - sends an API call to the Razor server to notify that the OS installation is now complete.

The boot_local, set_hostname, store_ip and the os_complete pieces of code are located in the
/opt/razor-server/installers/common/ directory.

common directory
boot_local

The boot_local.erb has the instructions for the OS to boot to a local disk.

set_hostname

The set_hostname.erb has all regex magic that implants the hostname
(from the <%= node.hostname %> variable) into /etc/hosts.

store_ip

The store_ip.erb - retrieves the node's IP address and then reports back to the Razor server.

os_complete

The os_complete.erb updates the MOTD on the node, and removes the lines that razor added to the /etc/rc.d/rc.local file. It then goes back to the Razor server and invokes the broker.

If we were to look at the ESXi installation we would see something similar.

vmware_esxi.yaml

The contents of vmware_esxi.yaml file - the same format as before.

vmware_esxi directory

The contents of the vmware_esxi folder is slightly different.

boot_install

The boot_install.erb file here holds the instructions on how to boot the ESXi kernel - which is slightly different than that of a traditional Linux.

pxelinux_esxi

The pxelinux_esxi.cfg.erb is the file that was called from the previous step

  • KERNEL <%= repo_url('/mboot.c32') %> - the standard ESXi boot menu.
  • APPEND -c <%= file_url('boot.cfg') %> - add in the boot.cfg file.

boot.cfg

The boot.cfg.erb file modifies the boot string to point to the kickstart file on the Razor server.

ks.cfg

The ks.cfg.erb file has the kickstart instructions for ESXi - with great thanks to William Lam

  • rootpw --iscrypted <%= … %> - decodes the password that was defined in the policy.
  • wget <%= stage_done_url("kickstart") %> - here an API call is sent to the Razor server to notify that the kickstart process has completed and that the default boot process should now be used.

The ks.cfg file can be customized according to your needs of course.

A few things that need to emphasized about the current ESXi kickstart process - it has some limitations.

  1. The network configuration is DHCP only. There is no built in mechanism to accommodate static IP addresses at the present time. I will say that there is a method - which I will explain in a future post.
  2. Hostname configuration - is not covered either, that same future post will explain how to deal with this problem as well.
  3. There is no additional reporting on the node's state - besides the fact that kickstart has completed.

That is enough for this part. Next we will look at a full deployment.

2013-11-26

Razor - Installers, Policies and Tags

This is part 4 of a series of posts on the new version of razor-server

1. Installing Razor - Yes the New Version..
2. Razor - DHCP and TFTP
3. Installing the Razor client and creating a repository
4. Installers, Policies and Tags
5. Razor - What Lies Within the installer?
6. Installing ESXi with Razor

We are almost ready to deploy our first node but in order for that to happen - we will need the following.

  • Installer - the set of commands that will be run to deploy the OS
  • Broker - the next step for the installation of the node - usually the orchestration software
  • Policy - this will contain the repository name, the installer name
  • Hostname pattern - the naming policy for your deployed nodes
  • Password - the root password for the deployed node
  • Count - the maximum number of times you would like this policy to be applied
  • Tags - How to identify the node - based on the hardware reported. In the previous post we created a single repository - ESXi_5_1. The details of the repository are below.

    razor repos "ESXi_5_1"
    razor repos

    Create a broker. In our case - this will be the noop broker - which means that nothing will happen after the node has been provisioned.
    razor create-broker --name=noop --broker-type=noop
    razor brokers
    razor brokers
    Create a tag.
    Before we go into how the tags are created - I would like to explain in a bit more detail, what the tags are used for.
    The microkernel's main purpose is to scan the hardware of the node, and report that inventory back to the Razor server. With that information we can create categories/profiles/flavors of nodes - and deploy a operating to that node.
    Let's take an example. You have a UCS Chassis. You want to deploy ESXi on the Blades that have 128GB of RAM, but on the UCS blades that have 64GB of RAM you don't want ESXi, but rather RHEL6. You also have an HP Chassis with 16 blades, and there you want to deploy ESXi on all the servers.
    If we were to verbally describe the tags they would be as follows.
    Tag Rule
    UCS_ESXi Physical server + Manufacturer is Cisco + Blade has exactly 128GB of RAM
    UCS_Redhat Physical server + Manufacturer is Cisco + Blade has exactly 64GB of RAM
    HP_ESXi Physical server + Manufacturer is HP
    VM_Ubuntu Node is a VM + 1 vCPU + amount of RAM is <= 1GB
    For our example we will be creating a test rule and it's appropriate tag. The tag definitions will be:
  • 2 processors
  • It is a virtual machine
  • Greater/equal to 4GB ram

    cat > tag1.json << __CREATE_TAG_JSON___
    {
      "name": "Test_tag",
      "rule": ["and",
    ["=", ["num", ["fact", "processorcount"]], 2],
    ["=", ["fact", "is_virtual"], "true"],
    [">=", ["num", ["fact", "memorysize_mb"]], 4096]
       ]
    }
    __CREATE_TAG_JSON___
    razor create-tag --json tag1.json
    razor tags Test_tag

    razor create-tag

    Now to create the policy.


    cat > policy.json << __CREATE_POLICY_JSON___
    {
      "name": "ESXi_5_1",
      "repo": { "name": "ESXi_5_1" },
      "installer": { "name": "vmware_esxi" },
      "broker": { "name": "noop" },
      "enabled": true,
      "hostname": "host${id}.maishsk.local",
      "root_password": "blahblahbb5",
      "max_count": "100",
      "rule_number": "100",
      "tags": [{ "name": "Test_tag"}]
    }
    __CREATE_POLICY_JSON___

    The text above is piped to the policy.json file.
    create policy.json
    Let's go into a bit more detail into each of the lines of this policy.
    name The name of the policy
    repo The name of the repository that should be used to perform the deployment
    installer The name of the installer to be used - The name that should be used is the name of the yaml file (without the extension) which can be found in the /opt/razor-server/installers/ directory.

    image
    broker Which broker should be used to continue the configuration, in our case the noop broker - which means nothing will happen.
    enabled True or false - you can create a policy and not have it active - if you choose.
    hostname The pattern that will be used to name your nodes.
    host${id}.maishsk.local (id is the number that was allocated to the node) - the rest you can customize as you please.
    root_password The root password that will be configured on the node.
    max_count The number of times the policy should be applied.
    rule_number This allows you to prioritize your policies. Lower number wins
    tag The tag that the node will be matched against.

    razor create-policy --json policy.json
    razor policies ESXi_5_1

    razor create-policy

    The steps above were split into two stages, but there is also a possibility of creating the tag and the policy in the step, all you will need to do is define the rule in the policy.json - when the tag is declared.
    Instead of this line

    "tags": [{ "name": "Test_tag"}]

    You could do the exact same thing without creating the tag beforehand with the following syntax (all should be on one line)

    "tags": [ "name": "Test_tag", "rule": ["and",["=", ["num", ["fact", "processorcount"]], 2], ["=", ["fact", "is_virtual"], "true"], [">=", ["num", ["fact", "memorysize_mb"]], 4096]]]


    The difference in the syntax above is that we defined the rule of the tag - during the creation of the policy.
    That was a lot of information to digest in this post, so I hope you were able to get it all in.
    In the next post - we will go into a little more detail about the installers and their configuration.
  • 2013-11-18

    Installing the Razor-Client and Creating a Repository

    This is part three in the Razor Series. In part #2 we saw how to configure DHCP ad TFTP and as I mentioned in a previous post, the client is now a separate component.

    1. Installing Razor - Yes the New Version..
    2. Razor - DHCP and TFTP
    3. Installing the Razor client and creating a repository
    4. Installers, Policies and Tags
    5. Razor - What Lies Within the installer?
    6. Installing ESXi with Razor

    Here is how you would go about installing the razor-client which will allow you to interact with the server.
    I installed it in my $HOME folder

    First we will clone the code from Github.

    git clone https://github.com/puppetlabs/razor-client

    git clone

    Change directory to the razor-client folder and install the required gems

    cd razor-client
    bundle install

    gems

    Next create an alias that will include the url to the razor server (substitute the correct IP address / Hostname of course), and you should add this to your profile if you want the command to be available each time you login.

    alias razor="$PWD/bin/razor -u http://127.0.0.1:8080/api"

    create alias

    As you can see it is possible to install the client on the same machine as the server. If you are doing this then then you will not need the url - because it defaults to http://localhost:8080/api

    And if you run the razor command you should see the following output

    razor

    At the moment there is nothing there so looking at the repos/brokers/policies/nodes/tags will not give you much - yet…

    razor commands

    First some terminology.

    • Nodes - Nodes are the clients that connect into the system.
    • Brokers - Brokers are responsible for handing a node off to a config management system, like Puppet or Chef.
    • Tags - A tag consists of a unique name and a rule. The rule is a characteristic present on the the node
    • Repos - A repository where the installation files of an Operating System reside
    • Policies - This is a defined set of steps that tie all the other bits and pieces in Razor together, and are what gets ultimately applied to a node.
    • Installers - This will be the set of actions needed to deploy the operating system - you can think as a set of kick start instructions.
    Creating a repo, for that you will need to provide a URL to the installation ISO.
    razor create-repo --name=ESXi_5_1 --iso-url http://webserver/VMware-VMvisor-Installer-5.1.0.update01-1065491.x86_64.iso
    create-repo
    What currently happens is that razor will return an ID for the repo and begin the import process. The file is
    extracted to /tmp and eventually moved to the repo location that we defined in config file
    (/var/lib/razor/repo-store). A new directory will be created according to the repo name. 
    import files
    (Just as a side note - the repo is extracted into the /tmp folder but removed there after - so make sure you have adequate space on your razor server - or alternatively delete the temp files after the repo is created. There is currently a bug open on this issue.)
    When you now run razor repos you should see the repository you just created.

    razor repos

    razor repos

    In the next post we will see how we create a policy and see what we can do with tags.