SmartOS "all-in-one"?

[ultrakomm]

Hi!

I'm wondering if it is possible to use SmartOS as the base for an all-in-one server, similar to napp-it ESXi/OpenIndiana/ZFS all-in-one?

Has anyone tried this? Is it possible? If not, what are the drawbacks? What are the advantages?

Best regards,
ultrakomm

 

[connorxxl]

Hello,

would be interested in this too. Currently thinking about replacing ESXi with either OmniOS or use Zones in Solaris 11.1 as much as possible, and just have a small ESXi running for "exotic" things.
Main reasons are 32gb limit of ESXi, and the need to virtualise storage instead of running OS with ZFS as base for virtualisation.

Thanks.

Regards,

Chris

 

[unhappy_mage]

I'm doing this at home. It works quite nicely, but there are a few caveats you should be aware of.

• The "zones" pool will need to service many IOPS if you have many VMs. Consider a stripe of mirrors for decent performance, or SSD.
• The VM management tools are all command-line. You can interact with KVM machines via VNC, but you need to do "vmadm info <UUID> vnc" to find out what port to use, as they're dynamically assigned.
• Project FiFo aims to fix the last problem, but it's not quite there yet as of January.

That said, I'm very happy with it. Deploying VMs from JSON config files is really slick. I use CFengine 3 to manage the VM configurations, and everything is tied to FreeIPA to allow for common Kerberos/LDAP logins.

I've considered using OmniOS to let me do more things in the global zone (configuring CIFS was quite annoying!) but that's too much work and downtime for me right now. If I had to start over I might start there.

 

[ultrakomm]

Thanks for the responses!

Could I setup SmartOS so that the "zones" pool is an SSD, and have a zone with a "storage" pool attached to it which handles all things (network) storage? Wouldn't that also solve the issue with having to configure CIFS in the global zone?

I also have two other questions related to SmartOS:

(1) What is the state of AMD support in the Illumos KVM implementation/port?

(2) Are there any plans at all to support VT-d (or IOMMU) in the Illumos KVM implementation/port?

 

[unhappy_mage]

Project FiFo aims to fix the last problem, but it's not quite there yet as of January.

I just went and retried this, and I'd like to revoke my previous statement. It's great now, my only quibble is that it doesn't allow you to just say "delegate network addressing to the real DHCP server over there". Other than that, highly recommend.

Thanks for the responses!

Could I setup SmartOS so that the "zones" pool is an SSD, and have a zone with a "storage" pool attached to it which handles all things (network) storage? Wouldn't that also solve the issue with having to configure CIFS in the global zone?

Yes, that's sort of like what I do. "zones" is on a FusionIO 80GB SSD, and all my data is on a second pool. You could easily start up a zone with the data pool delegated to it, and run Samba in that. Your zone JSON would look something like this:

{
 "zonename": "login",
 "alias": "login",
 "autoboot": true,
 "brand": "joyent",
 "image_uuid": "fdea06b0-3f24-11e2-ac50-0b645575ce9d",
 "max_physical_memory": 4096,
 "resolvers": ["192.168.0.253"],
 "nics": [
 {
 "interface": "net0",
 "nic_tag": "admin",
 "ip": "dhcp",
 "mac": "b2:76:ba:6b:59:13"
 }
 ],
 "filesystems": [
    { "type": "lofs", "source": "/export/data", "target": "/data" },
    { "type": "lofs", "source": "/export/home", "target": "/home" }
  ]
}

I also have two other questions related to SmartOS:

(1) What is the state of AMD support in the Illumos KVM implementation/port?

AMD-V is implemented currently in jclulow's fork of the kernel. You could hypothetically build that yourself, and build your own SmartOS image. I've heard rumors of a prebuilt version of this being available somewhere. I have an AMD server sitting around running a Minecraft server because this didn't work out for me, so if you happen to get this going let me know.

(2) Are there any plans at all to support VT-d (or IOMMU) in the Illumos KVM implementation/port?

Not that I'm aware of. What would you want to do with it?

 

[ultrakomm]

I just went and retried this, and I'd like to revoke my previous statement. It's great now, my only quibble is that it doesn't allow you to just say "delegate network addressing to the real DHCP server over there". Other than that, highly recommend.

Project FiFo seems nice, but I don't have any immediate use for it. I'm perfectly comfortable with the command line. I somewhat even prefer it. I might try Project FiFo, though, since it runs in a zone (mainly?) it can't hurt.

Yes, that's sort of like what I do. "zones" is on a FusionIO 80GB SSD, and all my data is on a second pool. You could easily start up a zone with the data pool delegated to it, and run Samba in that. Your zone JSON would look something like this:

...

Great. Thanks for the example zone json. I really like the json configuation of zones. All in all, SmartOS starts to feel like a much better choice than ESXi. At least for me.

AMD-V is implemented currently in jclulow's fork of the kernel. You could hypothetically build that yourself, and build your own SmartOS image. I've heard rumors of a prebuilt version of this being available somewhere. I have an AMD server sitting around running a Minecraft server because this didn't work out for me, so if you happen to get this going let me know.

Oh! Do you have any idea if/when it will be included in the "normal" SmartOS distribution?

Not that I'm aware of. What would you want to do with it?

Mainly experimentation. It's not really important. But I like the idea of being able to assign a VM a graphics card some time in the future and being able to stream games and whatnot (when technology allows for it).

Thanks for your reply.

 

[unhappy_mage]

Project FiFo seems nice, but I don't have any immediate use for it. I'm perfectly comfortable with the command line. I somewhat even prefer it. I might try Project FiFo, though, since it runs in a zone (mainly?) it can't hurt.

Yeah, that's where I am now. You install one component in the global zone of your smartos box (or on each, if you set up more than one), and then start a single zone somewhere that runs the web gui and talks to however many hypervisors you have.

Great. Thanks for the example zone json. I really like the json configuation of zones. All in all, SmartOS starts to feel like a much better choice than ESXi. At least for me.

Yeah, JSON is a much nicer solution IMO than the "wizard" solution ESXi uses. Managing through SSH is so much nicer than the Windows-only GUI thing.

Oh! Do you have any idea if/when it will be included in the "normal" SmartOS distribution?

I don't think there's really any timeline. Joyent doesn't have any AMD boxes, so they don't care to try to support it. I could try building you an ISO if you like.

Mainly experimentation. It's not really important. But I like the idea of being able to assign a VM a graphics card some time in the future and being able to stream games and whatnot (when technology allows for it).

Thanks for your reply.

Yeah, Joyent doesn't sell GPU access, so they're not interested in PCI passthrough for this sort of thing. It's one of those "well, if you want it, implement it and maybe we'll accept the code" deals. Unfortunately, in my opinion. There are other neat things you can do with that technology.

 

[unhappy_mage]

Just for fun, I built a copy of the SmartOS image with the AMD features turned on. This turns out to be surprisingly easy if you actually follow the directions, which I of course had a good bit of trouble with. I checked out vanilla SmartOS Live @178e78ef473f7a803949da0a39f618401f47bfdc, then in my configure.smartos file I swapped out the default "check out KVM" command line for one that checks out a copy of jclulow's KVM tree, merged with the fixes from the mainline illumos-KVM. Then ./configure, gmake world && gmake live. Finally, run "./tools/build_iso" and "./tools/build_usb".

Note that these images haven't been tested on Intel or AMD, but I might get to that soon on both counts. Intel is easier, but I should be able to kick everyone off the Minecraft server tomorrow during work hours

The images are available here.

 

[Stanza33]

I am trying to get a smartOS machine up and running to have a play with.

Unfortunately all my hardware is underspec'd

They have VT-x but no EPT "extended page tables" support.

So smartOS says "in soup nazi voice" ......no KVM for you

So I am restricted to zones only by the looks of it until I can aquire some later model hardwares.

or am I ?

eg

is it possible to get kvm working WITHOUT EPT support?

ie the AMD version unhappy_mage has compiled...would that give me kvm support on older xeon hardware? CPU's are E5345's etc... do AMD processors have EPT?

Thanks

 

[unhappy_mage]

is it possible to get kvm working WITHOUT EPT support?

ie the AMD version unhappy_mage has compiled...would that give me kvm support on older xeon hardware? CPU's are E5345's etc... do AMD processors have EPT?

I believe the version I posted won't require EPT. See kvm_vmx.c. It'll use it if you have it, but shouldn't require it.

 

[Stanza33]

Ok thanks,

Btw the USB image file link doesn't work

The iso link downloads

.

 

[unhappy_mage]

Btw the USB image file link doesn't work

Fixed, thanks for the heads-up.

 

[Stanza33]

Sadly kvm kernel panic'd when I gave it a go on the Socket 771 system

Haven't tried the AMD system as yet.

Will download the USB file to make booting a lil quicker

.