Storage Refresh

Wed 05 March 2014

Hooray! I've finally updated my home file server. This is the first non-drive change made to the system since I built it back in 2010.

The original build allowed up to 20x SATA drives sitting on top of LSI 2068E cards and a motherboard with three 8x PCIe slots on an Intel X58 chipset. This design was to allow for enough bandwidth back to the CPU for full controller-speed (SATA II) for all 20 drives in the chassis.

The system has served me well to date, but as I expand the array I'd like to keep on top of disk density - and if the opportunity arises, I wouldn't mind something a bit quieter.

Also beyond a crazy growth rate of ~300+GiB/mo for the first year or so, my growth rate has calmed massively and I'm only sitting at around 11TiB of data all up spread over 8x 2TB disks. So maybe supporting up to 20 drives isn't necessary after all.

I had a bit long rant preparated about all the goals and issues, but honestly it boils down to:

  • The newer backplane on the Norco I had still wasn't passive, and doesn't support 4TB drives
  • The IBM ServeRAID 1015 controllers (LSI SAS2008) I bought have a buggy Option ROM that doesn't pass the extended memory map through during boot and causes GRUB to wig out

I ended up settling for a design with a chassis supporting 12x 3.5" + 2x 2.5", with passive backplanes and different card on the LSI SAS2008 chipset. Whilst the dollar-per-bay isn't as great on the new case, it's a much higher build quality and has better airflow/cabling.

And it runs great - I've rebuilt the array onto 6x 4T drives still using linux software RAID but now using btrfs as the filesystem (I'll redo this as raw disk btrfs raid when the code for that is a bit more mature).

Now that the data migration and deduplication run have completed, I can cross this off my TODO list - and decide what to do with the spare gear I have left over, enough to build 2 file servers.

Any takers?





Mon 03 March 2014

I mentioned in a recent post that I've been working with a cool piece of technology called AngularJS recently. It's been a long, long time since I've done any regular web work - but since picking up Angular, I can honestly say that building web applications has never been more enjoyable.

As a developer, I found the documentation reasonably sucky - it has a quick tutorial and jumps straight into API documentation. There's no guide on overall architecture, which makes understanding how you should be building your app a bit difficult.

What follows is my quick and dirty list of notes for developers looking at using Angular in their app:

  • Everything is a module
    • Every piece of code in AngularJS is attached to a module.
    • Modules can express dependencies on other modules.
    • Your application is a module
    • Modules can contain Services, Controllers and Directives
  • Dependency Injection
    • Angular provide dependency injection for all prototypes
    • You have two options for specify a dependency:
      • Lazily, by ensuring the name of the variable name in the signature matches the name of a service; or
      • Explicitly, by providing an array of the service names followed by the function with arbitrary variable names
  • Services
    • Services are singletons and are lazily created - so you'll need to reference it in the app if you need them to be instantiated
  • Controllers
    • Controllers are exactly what they sound like, if you need logic to glue the view model to service actions - this is where that lives
  • Filters
    • Filters can exist locally to a controller or globally on a module
    • Be wary of the built-in filter module, if you use the string comparison this is actually a substring match (this has caught me out badly before). You probably want to make a global filter using angular.equals in these cases.
  • Directives
    • If you need a common Controller/Template to render the same object in your app, chances are you can bundle it together as a directive
    • Even if you're not at the point where you need to use directives, it's worth reading the AngularJS Directives Guide to understand how the Angular parser resolves directive names (and why the documentation shows directives in camelCase even though the examples you use dash-delimited).
  • You need the extras
    • Downloading AngularJS isn't enough for single-page development, you're likely going to need the following modules:
      • angular-route.js - Provides the ngRoute module providing page routing for a single-page application
      • angular-resource.js - Provides the ngResource module allowing you to easily create services backing onto REST interfaces
  • Understand that AngularJS is javascript
    • No, really. If you build your entire site it won't be indexable by search engines unless you prerender out a static copy. Read more here

I hope this is useful to other people starting out.

White Noise

Thu 28 November 2013

I got dragged into another philosophical argument the other day regarding white space in code. It's one of those quintessential tech discussions that does nothing more than rile people up over something that, for the most part, doesn't achieve anything of value.

Generally speaking, I try and stay out of these discussions - however given I was trolling the person in this situation, I feel entitled to claim at least some small victory - someone has to.

That being said, I do have an opinion - and in the interest of adding to the pile of spew on this site that feeds back into the greater slum that is the internet - let me tell you what I think.

Let's start with my personal preference:

  • DO use tabs at the beginning of a line to indent code
  • DO NOT use tabs after the first non-tab character in any line
  • DO try and avoid indentation/formatting after code has begun
  • DO NOT insert editor-aware comments in your source to set editor options

Easy! You're well on your way to coding nirvana. I think this scheme works particularly well because:

  • It doesn't matter what size indentation you prefer - No more of this "I like my indentation to be 2 characters, not 4 or 8". Please stop. That's a presentation issue. Use an editor that supports configurable indentation widths.
  • It's semantic - The number of tabs corresponds exactly to the number of levels of indentation. You don't have to worry about whether the author prefers their tabs at 2,4,8 or 19 characters wide - nor whether they prefer milk of cream in their tea.
  • It helps filesize - Okay this is a bit of a weak argument, but using tabs reduces file size by not representing redundant information. Put your solution into a version control system and this has a slight multiplier effect.

And to cap it off, here are some common arguments as to why we should avoid TABs, and why I think you're wrong.

  • Not all editors support TAB characters properly - This has nothing to do with the file you're editing. If the specification(grammar) for your programming language treats both space and tab as whitespace, then it's whitespace. Your editor is not supporting the language you're working with, and that's your problem.
  • I prefer my indentation at x characters - I've already addressed this above. Learn to configure your editor properly if it will let you, or find a new one that will. Vim, a terminal text editor from 1991 supports just about every possible combination of tabs and spaces, so I'm confident this is quite a lot easier than you might make out.
  • My Wyse TTY is only 80 characters wide! - Brilliant! I also spend the majority of my time in 25x80 or other weird-sized, resizable terminals. Wouldn't it be terrible to have to re-format the file every time your terminal changed size? That's your editors job, and again it can reflow the text and preserve indentation if you're using a good one.

Ultimately though, arguments like this are a zero-sum game - nobody is going to be 100% happy, and there's always going to be that guy who has invested their sense of identity in their position on tab-vs-space - but you're wasting your time dealing with them anyway, right?

Pick one that makes the leasy number of people cry, stick to it and move on - we've got interesting problems to solve.

Game Engine

Sun 13 October 2013

I recently stumbled across a post by Timothee Besset about his plan for an experimental framework for low latency, high fps multiplayer games. TTimo is a games programmer who wrote QRadient and has worked at id software, so knows a bit about what he does.

His post outlines the architecture for a modern FPS game engine, which struck a chord with me. Mostly because at a high level, it's similar to an idea I've had before, a message-oriented game engine using ZeroMQ and its zero-copy message model.

It's always nice to see your idea validated, especially by someone who knows what they're talking about. I'm most definitely not a game programmer, despite having tinkered and modded a lot over the years.

I also happened upon a fantastic interview with Edmund McMillen of Team Meat. Whilst being a talk about game development - Edmunds discussion about motivation, career and dealing with people really struck home about a lot of things I have been thinking about recently. Some of those things I'm not ready to talk about, but in terms of game development, the message is clear - keep going at it, the first 30+ things you make are going to be absolute horse shit..

Combined with discussions with one of my coworkers who is writing his own WebGL MMO, Langenium, the interest has definitely been reawakened, for now.

Onwards and upwards

As a result, I've been putting a small amount of time into this project over the last three weeks. One of the biggest problems I've had working on game-related projects (or really, anything at all) is trying to do too much all at once. You get no reward and make all the wrong design decisions at once.

Instead, I've been working at this iteratively - starting with a 2D maze game and upgrading a single thing at a time towards the end goal of a 3D FPS. A high-level of the iterations are as follows:

  • Tile-based 2D maze game using Windows Forms
  • Moving from tiles to continuous coordinate space
  • Move to OpenGL render in 2D
  • Generate 3D geometry
  • Move to 3D
  • Writing Quake 3 BSP parser
  • Move to BSP geometry
  • Quake 3 BSP culling

So currently, I can load and render a Quake 3 map - moving around within the world. I've implemented visibility testing using the BSP tree, however have a bug in either that code or my BSP loader which is preventing it from working correctly.

Before I push on - I need to fix that and implement support for Vertex Buffer Objects, as I am currently making an OpenGL call for each vertex, which is very expensive and thrashes the frame rate way down. I'm currently getting 10-15FPS, which is much better than I was expecting for my simplistic engine so far.

The Quake 3 map format is super-efficient and designed so data can be slurped straight off disk into OpenGL buffers. My BSP loader takes the data and interprets it into an object graph - so currently does not do this. The current plan is to implement my own load handler to re-flatten the data internally.

Whilst this is a bit of a waste for now, I don't have my own BSP tooling and want to keep my engine from being tied to the IBSP format. This is primarily a learning exercise, and I would like to support early versions of the Valve BSP format as well before I'm done.

From here

I'll be happy with my progress when I can come up with a simple death match game with netcode, similar to Quake 3 - complete with bots. Once I'm done with the game side I want to explore the map 'compilation' side of things, BSP partitioning and portal testing.

I've never really understood that side of things before now, and conceptually it's very straightforward - which is a nice refresher. I'm hopeful that as long as I can stay motivated and get past where I'm currently stuck, I can start to play with game ideas on top of my own engine stack.

image image image image image image image image


Sun 13 October 2013

Time for another update I suppose. Last year I noticed that it seems I'm most intellectually active around April and October, which definitely seems to hold true this year.


This year, I've been trying to force myself to spend four hours a week at a coworking space. I'm a member of a local space called Spacecubed, who host the local Startup Weekend events and have become a great hub for local special interest groups.

Using the space is fantastic, mostly because its a comfortable place where I can zone out and set time aside weekly for getting things done.

Java EE 7

I've recently started foraying into Java again. Borne mostly out of need at work, where I am writing some software to manage software deployments where reliable Git integration is possible. Plus, y'know... all the cool kids are on JVM these days.

Self-learning EE has been rewarding, and it looks like it has come a long way since I last looked at it. It looks like Java 8 will also bring about a lot of long-needed language improvements as well, including Lambda expressions - something I've found makes writing fluent code really nice in C#.

Angular JS

As part of the deployment software I'm writing - I've also had the time to start learning a good Javascript framework for web applications. I've been using AngularJS with Bootstrap as a base, and it's refreshing to be able to rapidly prototype web apps that don't feel like they've been written by a programmer.

Together with Java EE and JAX-RS - this has given me a good common patern for developing web apps again, so should cut down on the hemming and hawing when I get motivated about some of my many forever projects.

Visual Effects

This has been an interest for a long time, and having recently picked up a dslr that shoots 1080p, has been something I've been toying around with.

Whilst I'm interested in the effects side of things, to keep it interesting - I really need to be able to tell a story effectively. Even a one minute narrative is stretching myself a bit thin, but it's hard to get motivated when you're camera shy.

In the meantime I'm reading some good books on scriptwriting and creative writing and will keep hammering away on this in the background.

3D Game Engine

This one I'm particularly happy and motivated about. So much so, it's going to get its own post. Bully for me.


Will is a terrible person. He should be shot.

Will Dowling
what the deuce?

This is the personal website of Will Dowling, a Systems Engineer hailing from Perth, Western Australia.