Constant Thinking | Technology Thoughts for the Quality Geek | by Constantin Gonzalez

ZFS is for 1337 Hax0rz

19.06.2010, updated: 03.10.2017

zfscode.jpg

The developers of ZFS are a funny bunch of people. You can tell that by watching the "ZFS: The Next Word" talk, meeting them on conferences, reading their blogs or their comments on mailing lists.

And there are also some funny parts in the ZFS source code, too. In fact, if you use ZFS, you'll have a funny joke sitting on your disk, right under your nose!

I was reminded about this particular joke while listening to Ulrich Gräf's excellent talk on ZFS internal data structures during OSDevCon 2009 (watch a video of Ulrich's talk here).

But first, we need to dig a little bit into the world of ZFS data structures.

More →


The War Between Quality and Popularity

16.06.2010, updated: 03.10.2017

Star Wars (Sorry for the pun :) )

War is waging in the galaxy. This time it's not the Rebels against the Empire, or Good vs. Evil.

No, this war is different, and it has been going on ever since products designs companies entities existed.

What I mean is the war between Quality and Popularity.

Let me explain:

(Drumroll, Roman fanfare, then dramatic Anime action trailer a la Mortal Kombat, etc.)

As of May 21st, Google officially declared war on the Apple iPhone.

Sure, there was some teasing here and there for weeks, if not months, but this is serious.

More →


A Closer Look at ZFS, Vdevs and Performance

04.06.2010, updated: 03.10.2017

vdevs.jpg

When looking at the mails and comments I get about my ZFS optimization and my RAID-Greed posts, the same type of questions tend to pop up over and over again. Here's an example from a reader email: "I was reading about ZFS on your blog and you mention that if I do a 6 drive array for example, and a single RAID-Z the speed of the slowest drive is the maximum I will be able to achieve, now I thought that ZFS would be better in terms of speed. Please let me know if there is a newer ZFS version that improved this or if it does not apply anymore." This is just an example, but the basic theme is the same for much for the reactions I see: Many people think that RAID-Z will give them always good performance and are surprised that it doesn't, thinking it's a software, an OpenSolaris or a ZFS issue.

In reality, it's just pure logic and physics, and to understand that we should look a little closer at what vdevs are in ZFS and how they work.

More →


Tipping

One of the web 2.0 startup hypes du jour is called Flattr. It's a micro-payment service that enables readers to allocate a monthly budget they can use to "flattr" blog posts and other content by clicking a simple badge. The monthly flat-rate is then distributed to the "flattred" authors as a way of appreciation.

More on that in this short video.

A lot of people are jumping on the Flattr bandwagon, and of course, rewarding creators is always goodness.

But I'm not convinced.

Why? Because IMHO there's no need for a middle man handling the "Thank You" transaction between a blogger (or other content creator) and the reader (or other content consumer).

Instead, here are ten (10) better ways to accomplish the same thing (supporting your favorite creator). They can involve money or not - you choose - and in each of them, everybody wins.

More →


NewMusicSMF.jpg

In the last couple of posts, we used DTrace to notify our media servers and perfected our script a bit.

But the script is still not ready to be used on our home servers yet: It requires manual start and stop, not quite the service oriented automatism we're used to in the Oracle Solaris world.

The next step is to wrap our DTrace script inside a Service Management Facility (SMF) service, then wrap everything into a shell script that will easily install or remove the service whenever we need it.

More →


NewMusicUpdate

A few weeks ago, we discussed using DTrace for automatically updating media servers when you upload new content.

Yesterday though, I discovered that my D script didn't work any more. I uploaded new songs to my home server, and expected the music daemon to re-scan the music directory, but nothing happened.

That teached me an important lesson about DTrace, and here's what I learned:

More →


ZFS Performance

This is a long article, but I hope you'll still find it interesting to read. Let me know if you want me to break down future long articles into multiple parts instead.

One of the most frequently asked questions around ZFS is: "How can I improve ZFS performance?".

This is not to say that ZFS performance would be bad. ZFS can be a very fast file system. ZFS is mostly self-tuning and the inherent nature of the algorithms behind ZFS help you reach better performance than most RAID-controllers and RAID-boxes - but without the expensive "controller" part.

Most of the ZFS performance problems that I see are rooted in incorrect assumptions about the hardware, or just unrealistic expectations of the laws of physics.

So let's look at ten ways to easily improve ZFS performance that everyone can implement without being a ZFS expert.

More →


During OSDevCon 2009 in Dresden, I had the honor to present on some lessons learned while putting together a ZFS Automatic-Scrubbing SMF service.

Today, Deirdre was so kind to publish the video recording of my presentation on The Oracle Solaris Video Blog.

From the description:

This talk walks you through the implementation of a simple SMF service. What sounds "simple" at first, develops a life of its own when you consider installation/de-installation, security considerations, error handling and debugging, and of course unexpected little bugs and shortcomings. Finally, we add a GUI to our service by discovering the OpenSolaris Visual Panels project. This "lessons learned" talk is intended to be a practical roundup of things to consider for developers interested in integrating with SMF.

More →


newmusicsmf.jpg

Before we continue with our Home Server Scripting Series, let's throw in a simple but useful DTrace hack.

One of the most typical uses for a home server is to serve music or videos to home entertainment equipment. In my case, I'm using the Firefly Media Server to serve music to my Roku Soundbridge and Mediatomb for videos.

The Media Server Update Problem

Whenever I upload new music or videos to my OpenSolaris home server (typically by rsync-ing my laptop home directory), both Firefly and Mediatomb need to be restarted so they detect that new files are sitting in their directories, waiting to be served.

More →


OpenSolaris ZFS Home Server Reference Design

08.04.2010, updated: 03.10.2017

Blueprint

When I blogged about my OpenSolaris Home Server a while ago, little did I know that this would become my most popular entry in my old blog!

In fact, R.G. was so kind to call my setup "perilously close to being an AMD reference design". Thanks, R.G.! Read about his final setup here. And BTW, if you're into e-guitars, check out his GEOFEX page, a great resource for guitar effects.

So let's review our reference design and discuss some modifications to better suit your needs:

More →


My empty OpenSolaris Desktop at work.

Yesterday, I cleaned up my home directory at work. I went from ca. 15 GB of data down to 1.1 GB. And I only stopped there, because I didn't want to spend too much more time cleaning up. Here's how to do it.

In the previous post of this mini-series, we looked at why it's important to have our emails and files organized, then attacked our INBOX to reach zero-message-nirvana. I'm happy to see that others are living by these principles, too. Thanks, Gregor!

Now let's look at that other dark spot in our IT lives: Our Desktop and file system. If you're like me, you see this very often, too: Cluttered desktops with so many files and folders and downloads and icons and stuff, you can barely make out the underlying desktop background.

More →


A tidy desktop with a zero email INBOX

Spring's around the corner and the Easter weekend is upon us, giving us some time to sit back, relax and do some spring cleaning!

This also applies to your data, in particular your Email folders and your home directories. In this two-part series, we'll clean up our email INBOX to zero (yes: null, nada, zip) emails, simplify email folders, then clean up our home directory file structure. That'll save us time, help us find peace of mind and make us more efficient so we can concentrate our energies on what really matters to us.

More →


Package Scripting

OpenSolaris OS comes with the Image Packaging System (IPS) for managing installation and additional software. By default, it is configured to pull packages from the

/release
repository, but of course there are many other interesting repositories with software to install from.

Chris Gerhard recommends adding at least the

/extras
repository (free, registration required) so you can easily install VirtualBox, Adobe Flash, TrueType Fonts etc.

If you have a support contract or are an Oracle/Sun employee, you may want to switch your preferred repository to the supported one. The more adventurous may want to switch to the development repository.

Then, the Software Porting Community on OpenSolaris.org has a hierarchical system for making open source software available on OpenSolaris through two repositories:

/pending
and
/contrib
.

And then there are a couple of more repositories from user groups, private people, companies offering commercial packages and so on.

One way to register package repositories with the IPS on your home server is by using the package management GUI, or by using the pkg(1) command (See also: "How to Add or Update a Publisher").

But of course we want to stick to our "Script Everything" philosophy and write ourselves a small script that takes care of all of our package configuration needs.

More →


Deduplicaed Folders Illustration

Since November 1st, 2009, when ZFS Deduplication was integrated into OpenSolaris, a lot has happened: We learned how it worked, people got to play with it, used it in production and it became part of the Oracle Sun Storage 7000 Unified Storage System.

Here's everything you need to know about ZFS Deduplication and a few links to help you dig deeper into the subject:

More →


Power Management Scripting

Last week, we looked at how essential scripting is for administering home servers (one of the 7 tips for home server bliss) and we wrote us a little script for enabling automatic snapshots.

Another thing that you'll almost certainly want to do on your OpenSolaris home server is enabling power management. This will ensure your server spends as little power as possible when idle, while still being powerful when needed.

More →


Firefox wearing the OpenSolaris Persona

Firefox 3.6 has introduced a new feature (that was in Beta for some time): Personas.

What's a Persona? From Firefox' own description: Personas are easy-to-use themes that let you personalize the look of your Firefox There's a very nicely done video on the Firefox Personas site to introduce the feature.

If you (or your friendly sysadmin) haven't upgraded to Firefox 3.6 yet, you can still install the Personas Plus Add-On to enjoy all of the Personas goodness.

More →


ScriptingHomeserver.jpg

One of my OpenSolaris Home Server Tips is to script everything. That triggered quite some interest. So let's start a short series around OpenSolaris home server scripting.

Today, we'll talk a little bit about the "why?" of home server scripting, then run into a small surprise while we write a small script that will enable/disable the OpenSolaris ZFS Auto-Snapshot Service for us.

More →


Over the weekend, I redesigned the theme of my blog. I hope you like the result!

While I'm not a web design nor a theming expert, it was still quite easy to find a nice theme, and apply some blog-friendly customizations to it. Here are a few tips to get you started in Drupal theming:

More →


A book goes electronic. Sort of.

A while ago, we tried defining ebooks, and figure out what they are, and what they aren't. Now, let's have a look at some success factors, or barriers, business implications and the (un)necessity of DRM.

More →


In a recent blog article about the future of IT admins, my MUCOSUG-Buddy Wolfgang wondered whether the new generation of self-managed, appliance-like systems like Oracle Exadata, Oracle Sun Storage 7000 and their friends from other vendors are making IT personnel redundant, or what kind of jobs IT people are supposed to be doing in the future.

This reminded me of Dan Pink's book "A Whole New Mind" (Amazon.com|co.uk|de, BooksOnBoard). Pink argues that today's "left-brainish" jobs are threatened by "abundance, automation and Asia" (the latter really meaning "outsourcing") and that today's knowledge workers need to learn how to better employ their "right-brain" and add creativity to their jobs, as a new competitive differentiator.

How does this relate to Technology or IT jobs?

More →


Welcome!

This is the blog of Constantin Gonzalez, a Solutions Architect at Amazon Web Services, with more than 25 years of IT experience.

The views expressed in this blog are my own and do not necessarily reflect the views of my current or previous employers.


Copyright © 2017 – Constantin Gonzalez – Some rights reserved.
This site uses Google Analytics, Alexa, Feedburner, Amazon Affiliate, Disqus and possibly other web tracking code. See Imprint for details and our information policy. By using this site you agree to not hold the author responsible for anything related to this site.

This page was built using Python, Jinja2, Bootstrap, Font Awesome, AWS Step Functions, and AWS Lambda. It is hosted on Amazon S3 and distributed through Amazon CloudFront.