robothaus

Butter 0.2 - 'Ghostbusters'

2012-03-06T00:40:00-05:00

Last week at the Mozilla Toronto office, we celebrated as I clicked the “close milestone” button for Butter, marking the 0.2 release, codenamed “Ghostbusters”. It was also Dave’s birth-week, so we celebrated with a aptly themed cake. [Photo credit: Ben Moskowitz]

The repo has been moved to Mozilla’s github account, so feel free to clone the repo and follow progress: https://github.com/mozilla/butter

Considering the amount of work we did to roadmap Butter for the year, and the amount of time the team had to produce this release, I’m quite proud of our accomplishment. Butter already looks better, functions faster, and feels more solid as a foundation than its predecessors. This is a great start to a year of Butter, and 0.3 is just on the horizon.

Rather than delving into details about time and estimation analysis (re: robothaus.org/mozilla/butter-roadmap/) about 0.2, I’ll just boast about its exciting features and talk about what’s to come in 0.3. Besides, I’m told that my posts are notoriously long, so splitting out that content might not be such a bad strategy.

The UI

Ben and I spent a lot of time pulling apart Popcorn Maker’s UI to attempt to deliver a more pleasing experience with this refresh of Butter. We still have some work to do to accomplish the design we envisioned in our vision document, but we’ve made significant steps toward a flexible UI environment which we can fill with better functionality in the near future.

Here are a few things to look for so far:

The timeline lost a lot of unnecessary height as we decided to make more efficient use of some key areas. Now, after minimizing the iconic Butter tray, you’re still able to use the timeline scrubber and play, pause or mute the media.
Scrollbars look far less ridiculous since we wrote our own implementation. Not only do we get to decide visual aspects of the scrollbars, but also the user experience for them.
Look at how juicy these track events and track handles look now. They’re more defined, and have a colouring scheme based on a hashing of the track event type – so image is always red, attribution is always turquoise, regardless of which OS, browser, or version of Butter you’re using.

Butter isn’t just more visually pleasing though, it’s more helpful. We’ve added a highlighting system to show you what’s targetable, and what’s been targeted.

When dragging a plugin from the plugin-tray, an icon will appear under your mouse to show you what kind of object you’re going to produce when you drop. However, you’re not limited to dropping plugins on tracks anymore; you can drop them right on the target you want to affect. As you move the icon over certain objects, they will light up to tell you that you can drop content on them.

After dropping, a track event will appear on a track at a time corresponding to the position of the timeline scrubber. Clicking on the new track event will produce a similar effect: an object on the page will flash, letting you know it’s the targeted object.

This whole procedure feels a lot more natural than our previous attempts. Your intentions as a designer or film-creator are more directly realizeable.

As a result of these UI improvements, just through some preliminary testing, the software feels like it’s more reactive and less confusing. We dug out some old code that profiled very poorly, and made sure our approach stayed clean and simple. Obviously, it’s pleasing us so far, so hopefully it will test well in future releases.

The Code

Firstly, the HTML/Template situation got a lot simpler. The HTML code listing below and a bit of JavaScript is all you need to put Butter to work in your page now:

Here are some noteworthy points about this code:

There are really two important inclusions: butter.js and butter.ui.css. The components of Butter will build themselves from these two libraries.
Two separate data- attributes help Butter identify elements of your page at different stages during the editing process: data-butter and data-butter-exclude. The former is a statement of important elements on your page – like targets or media elements for Butter to consume and use – while the latter marks a tag to be excluded from an HTML export of the page.
Starting Butter is still a manual process, keeping it an opt-in feature of your page. We want developers to be able to control their individual experiences. Above, another script is included, template.js, which looks similar to this:

  Butter({
    config: "../config/default.conf",
    ready: function( butter ){
      var media = butter.media[ 0 ];

      media.listen( "mediaready", function( e ){

        var track = media.addTrack( "Track1" );

        butter.plugin.add([
          {
            name: "footnote",
            type: "footnote",
            path: "../external/popcorn-js/plugins/footnote/popcorn.footnote.js"
          },
        ], function( e ) {

          var event = track.addTrackEvent({
            type: "footnote",
            popcornOptions: {
              start: 0,
              end: 3,
              text: "test",
              target: "Area1"
            }
          });

        });

      });
    } 
  });

Here’s a brief walkthrough:

Start Butter with the Butter function, providing a url to a configuration file, and a ready callback function.
Once the ready callback has been called, a reference to the butter object is provided, and butter is almost ready to use.
Wait for the media to be available (it still has to load) by attaching a listener for mediaready to a Media object.
Let Butter know which plugins should be available in the UI by calling butter.plugin.add
Butter!

I believe there are still too many steps to get going initially, so we’ll be widdling it down over the next couple of releases. However, this code runs consistently well now, and is becoming increasingly data-driven. For instance, we use a config file instead of an inline JSON blob. Soon, we’ll be working on ways to encompass the needs of developers as utility functions, so even less code will be necessary unless you wish to abnormally customize your experience.

The Server

Jon Buckley started laying the foundation for our server infrastructure this month. The project is hilariously titled Cornfield (also available on the Mozilla github account).

For Butter, this means having the ability to connect to a server which is specifically prepared to save, load, import, publish, and share content within the framework.

Currently, Butter has the ability to communicate with this rudamentary server, but we’re still working on the API layer and UI design to make it shine. Look for more about Cornfield in future releases.

The Tests

Briefly: at crunch time last year we forgot that our test suite existed, but now we’ve paid particular attention to making sure our tests are up-to-date, and that new features don’t hinder a passing grade. We have to make sure this stays in tact, especially as Butter continues as a formal foundation for the year’s overall project.

So, if you clone our software and run the test suite in your environment (however convoluted it may be), and some tests don’t pass, file a ticket. We want to know about it.

Onward

0.2 is publicly available, and we already have more than 100 issues to tackle in 0.3, which is slated for release the first Tuesday of next month.

Again, I’m going to be gathering metrics about ticket completion and bug generation so that we can refine our development process throughout the year. Another blog post is deserved to talk about the results of that campaign.

I’m proud of our progress so far though, and I can’t wait to see what we will produce in the months to come. We’re already beginning to wonder what kind of cake best fits the Breakfast Club theme.

Popcorn: It's Roadmapping Season

2012-02-14T06:21:00-05:00

A significant portion of the Mozilla Foundation’s effort this year will contribute to Popcorn Maker’s eventual success, culminating in a 1.0 release in or around November. We’ve correctly identified that achieving this is a tall order, so we planned and roadmapped harder than I’ve ever planned and roadmapped before.

Update: The popcorn roadmapping PDF explains the entirety of the project from a high-level design perspective. Here is the lighthouse link to add your comments about what will be happening, and here is a direct link to the PDF.

Popcorn Maker - Here and Now

In the past, we treated Butter, and to some extent Popcorn.js, as means to an end – namely, Popcorn Maker. This model used to make sense: we were creating an app (Popcorn Maker), which needed an API (Butter) and underlying library (Popcorn.js) to make it all tick. With some vision from the foundation, and specifically from Ben Moskowitz and Brett Gaylor who have both blogged tirelessly on the subject, that outlook has changed significantly. And, I believe it’s for the better.

Butter

Previously, we wrote Butter, and quickly stuffed it underneath Popcorn Maker once it was "ready". Finishing the app-version of Popcorn Maker was a greater and more demanding task than maintaining Butter separately, so we hacked at it only to give us results we could immediately use. It even lost touch with its own repo as we voted to make it a git sub-tree instead of a sub-module (deserving a rather passionate rant).

As spiteful as I might sound, this approach wasn’t all wrong. There was simply no need to maintain an API that was going to be used directly by a handful of people for a specific project at an accelerated pace for a short period of time. Butter was classified as a library – the underpinnings – of its shinier cousin, Popcorn Maker. Luckily, though, as careful engineers do when they acknowledge reasonable fear and forethought, we kept Butter isolated so as to still be separated from Popcorn Maker by an API – just an ugly one.

Now, as I’ve already stated, things are a bit different. Some of the things Butter did well were identified as useful to other parties, and we don’t necessarily want to package the entire Popcorn experience inside of one app, Popcorn Maker. So, we’ve pulled out the Butter which lay unappreciated inside of Popcorn Maker, and given it its own repo again!

After a few iterations, Butter will be an SDK (the BDK, perhaps) to support the development of Popcorn-centric apps, exclusive of Popcorn Maker or not. It will retain control of editors and templates, but with an important architectural change.

With Popcorn Hacker, Scott Downe showed us that editing Popcorn directly in a page is a great feature – a feature which was downplayed by Popcorn Maker for sake of app consistency and safety. So, we’ve folded the idea back into Butter.

Butter is slimmer, and meaner. It lives directly inside the page it’s editing, so no postMessage loops are necessary. Bloat down, functionality up. With some external guidance (bookmarklet or manual insertion), it opens itself in a page (UI and all) and, with increased versatility, attempts to recongnize existing Butter or Popcorn data. When the user is done editing, Butter attempts to stay resilient and small, insisting only on HTML and JSON for storage – no external dependencies.

The result is an SDK which allows developers to create templates and apps based on the Butter library to take advantage of the UI elements and Popcorn abstraction which Butter achieves. Developing an app specifically for "making videos with contextual tidbits which popup at specific times" is as easy as building a design for the app with CSS and HTML (the template), and enabling the Butter library within it.

Also, we heard a loud cry from template and editor developers to have editors which interact directly with the template. Now, editors can live in the template, simply using the Butter API to interact with timed events.

Sounds great, right? Well, we’re only getting started: the 0.2 release is scheduled for the end of this month.

The Popcorn Software Suite (PSS)

With Butter sketched out as an broader SDK rather than a library with a struggling identity, the Popcorn software suite unfolds like this for 2012:

Butter (SDK) - The SDK which will allow developers to build apps, including Popcorn Maker. Its library will reside inside of templates which enable Butter’s tools in a customizable fashion. It includes the usual UI pieces – a timeline, a plugin list, editors, and popups – all of which are programmable with API, and have management systems to govern them.
Cornfield (Server) - The server component in the Popcorn software suite, allowing users to store their work, share templates, fork projects, and manage video content.
Popcorn Maker (Site) - A site which encompasses Butter and Cornfield with a well-designed entry-point and fancy UI. This is where content galleries, ratings, profiles, and tutorials live, and will be as accessible as possible directly from our public-facing websites.

I’m particularly excited about being able to offer Cornfield, and specifically its Video Transcoding feature. This will let users upload their videos, and have them automatically cross-browser compatible (mp4, ogv, webm), which makes creating and distributing a project very simple.

Also, there is a focus on making each component easy to use and install. In general we want to use npm as much as possible to make Cornfield and Popcorn Maker installable packages with setup scripts (or web pages, like Wordpress) to initialize them.

Technical Roadmap

Aside from Butter, there is a lot of software to imagine, design, write and maintain this year in the Popcorn project alone. In an attempt to organize the team, I created a document which illustrates the features we need to complete, categorized by milestone and project.

Here’s an excerpt:

I really tried to stay away from assigning time to features, since I could only estimate from general experience how long each would take. Instead of spending time filing down a time estimate to precision, I attempted to give each feature a reasonable rating of effort. For example, the first top-left green block represents the Droppable Regions feature. It received a rating of 3 because I considered the amount of CSS detailing, the jquery-ui interfacing, and Butter-specific scripting we would have to do to complete it as a non-trivial amount of effort.

As you might have already noticed, features don’t necessarily exist in only one milestone; they can span milestones with varying degrees of effort, displaying three important areas in the lifetime of a feature:

Research and Design
Implementation
Bugs

The first two are sometimes lumped into the same milestone, but often there is special consideration for the bugs that are a result of the implementation of a feature. These bugs are separate from Bugs in general which have their own colour and path through the graph.

My favourite part of the document is the Summary graph:

I think it’s particularly interesting to see the shape of work as the year progresses. I wasn’t surprised to see a rough bell-curve, but its height in the beginning of the year is concerning. Either estimates are wrong and will be adjusted to reshape the curve (likely), bugs will be much more significant with respect to effort than anticipated (pretty likely), or we’ll just be working very, very hard the entire year (very likely).

Note that no graph takes into consideration the number of people working on the project, so, as more people contribute, the larger the spread of the effort in each milestone, which yields higher relaxation times and lower stress levels for engineers :).

Of course, by no means is this meant to be a perfect description of how the year will occur. There are bound to be features which slide into subsequent releases or get dropped altogether. There are also certainly some estimates I have made which will prove to be completely incorrect, either negatively or positively. The document is intended to be a living one, so throughout the year I will update it to keep track of progress and to hone feature estimates – for this project and for future ones.

You can find the document, in all of its estimated glory, here.

Popcorn.js - All Grown Up

2011-12-22T13:37:00-05:00

This morning, CDOT student and Mozilla hacker, Jon Buckley discovered that the Firefox 9 Addons About Page uses Popcorn.js to augment a tutorial video about Firefox addons by adding apropos content to the page while the video plays.

Of course it’s great to see another example of Popcorn in the wild, but this is a use case we had never before considered. Using Popcorn.js to show synchronized interactive content, the page explains addons, what kinds of addons there are, and how a user can use them in Firefox.

Being Sherlock

Such an elegant use of the library deserves a little investigatory work. Since the authors of the many pieces of user-facing Firefox content do such a good job, it may not be immediately obvious that the addons about page is made of the web. Nevertheless, because it is indeed a webpage, I can right click to learn more, and open up the page in a new tab to isolate it.

Without flirting too much with a tangent about the merits of the Web Console, finding out that the page contains an instance of Popcorn is pretty easy:

Press Ctrl, Shift and K altogether to bring up the Web Console.
Type in Popcorn – which should auto-complete even before you have typed Pop – and press Enter.

Right away, without digging through HTML for a script tag, we can see that Popcorn.js is included on the page somewhere. But, we can do more than that:

After starting the video, try typing Popcorn.instances into the Web Console. It should report back with a large blob of text, which is actually an array containing Popcorn objects.
You can see the events that are going to occur during the video by typing Popcorn.instances[0].getTrackEvents(), which should return another blob – an array of Popcorn events.
Similarly, by trying commands like Popcorn.instances[0].getTrackEvents()[0].start, or Popcorn.instances[0].getTrackEvents()[0]._natives.type you can see when the first event will start, and what plugin type it is respectively. Feel free to copy & paste.

By poking around in a similar fashion, I was able to discover that the addons page makes use of the code plugin to add and remove CSS classes from elements on the page. This approach lets the video change the visibility and interactivity properties of the information bar’s constituent elements.

If this bit of detective work caught your attention, we’re only where the rabbit hole begins. In fact, there’s a whole API to discover and play with (recent documentation update courtesy of another CDOT student and Mozilla hacker, David Seifried).

Feeling the Effects

Needless to say, it was surprising seeing comparatively excessive usage stats on the popcornjs.org server, as throngs of people visited the addons about page:

Luckily, Jon Buckley, and Rick Waldron (another passionate Popcorn contributor from Bocoup) were able to advise the people managing the addons website to pull the server out of accidental DOS territory.

It's a Good Day to Render

2011-12-14T01:52:00-05:00

After what seemed like a totally unproductive and inconsequential morning, and an equally slow and frustrating afternoon, I managed to at least prototype two rather important things that have been lingering around Paladin and CubicVR for a long time.

First, Gladius had a rendering engine, but it was almost entirely useless unless you really like clearing a canvas. Second, CubicVR.js could only draw to one canvas on a page. Not “one canvas at a time”, just one canvas.

Now, the hard part of both of those problems are dealt with, and it’s no coincidence that they were both attacked in the same day.

State of Gladius

Progress on Gladius has been steady as Alan irons out the last bits of the framework for the looming 0.1 deadline (roughly, the end of December). Doing my part, last month I boasted about Gladius getting a renderer, but it was really only partially complete.

What it did: cleared the screen

What it does now: builds models from meshes and materials, clears the screen, renders those models, and has a scheduled engine task to repeat it all

Clearly, today’s work yields far better results than the prior engine. Obviously, we still lack a lot of things that one would assume a modern graphics engine would provide to game developers, but this is a big leap.

In short here’s how preparing and rendering something looks like right now:

create a Mesh resource from a script
create a Material resource from a script
create a Model component
create an Entity to hold the Model
feed the Mesh and Material to the Model
add the Model to the Entity
render!

This approach is interesting because it relies heavily on prepared scripts to build resources like Meshes and Materials. For instance, there is a procedural cube Mesh resource which will (unsurprisingly) create a Mesh that is a cube. There is also a procedural sample Material resource which will create a simple red Material (no textures yet, just colour). Then, as a game developer, it’s easy to swap out Mesh or Material resources when you want different ones for a particular Model, and it’s easy to reuse the same procedural resource to give you lots of slightly different Meshes or Materials.

Also, it may not be entirely obvious that, by using procedural resources, we have the ability to build an entire game without loading a single bitmap or otherwise baked asset. Imagine building a game that involves a simulation of a universe. Instead of forcing the user to wait precious seconds to download a large file describing the shape and state of the universe, they can download a tiny piece of code which, when used with a particular seed, creates the same universe. This same technique can be used to create not only Meshes, Textures and Materials, but even Sounds, Forces, or Terrain. A lot of demoscene groups solve size problems similarly to create the notoriously mind-boggling 64k intro category.

Just imaging stuffing this into 64k:

Some have even attempted procedurally-generated games:

Coasting on back from that tangent, Gladius is a long way off from being able to produce these kind of effects, but that’s generally what we’ve got in mind. We want devs to really be able to use their imaginations with Gladius, especially in a climate where fat downloads are often not appreciated.

Massaging CubicVR.js

The way we’ve used CubicVR.js in the past it may be surprising that its in need of some caretaking. I don’t at all mean to imply that it lacks functionality. In fact, it continues to surprise me even as I flip through test html pages fixing syntax problems I’ve introduced. However, there are a couple pieces of CubicVR.js that require some love:

The Octree implementation has drifted slightly from the progress and structure of the rest of the engine, so it needs repair. The current implementation looks a lot like Java (my fault), and there is a new, mostly complete JavaScript-inspired version called octree.js which is available to start integrating.
CubicVR can only render to one canvas on a page. Rendering to multiple canvas elements should be possible, and indeed easy.

Luckily, one of Dave Humphrey’s students has offered to help bring CubicVR.js’ Octree back to life, which he has already completed to some degree. Soon, while octree.js gets wedged in, we’ll need a few tests to compare it with CubicVR.js’ current implementation – for sanity, and for speed comparison.

Multi-

While getting Gladius’ renderer in order today, I encountered a few pieces of CubicVR.js on which I’ve had a grudge for some while. The initialization sequence and global namespace CubicVR.js uses are cumbersome when being used inside a project like Gladius, which we’re trying to keep as flexible as possible. Much of my perturbation stems from the question, “What happens if the developers wants to render to two independent canvas elements on a page?”

Gladius is an especially good example because of its qunit testbed, which uses async tests likely more often than it should. Guess what that means. You probably thought, “multiple simultaenous renders!” Since we need to test multiple engine initialization scenarios, Gladius’ testbed will try to render to one canvas multiple times, or several canvases once each. I chose to implement the latter because of the async demands, but I really sidestepped the problem by only rendering once in the entire test module, …which is bad. Our testbed will only get more complicated, so, CubicVR.js needs a multi-canvas solution fast.

Digging Into the Problem

It’s not immediately obvious what will happen if I try to do this with a current build of CubicVR.js (which I’ve actually documented):

var gl1 = CubicVR.init( canvas1 );
var gl2 = CubicVR.init( canvas2 );

In fact, we end up in something of an undefined space here when we want to continue to use both canvases. When you create a texture, to which WebGL context does it belong? If you try to find out, Firefox will likely crash like it did for me today a few dozen times while I debugged the situation.

Initially, I attempted to solve this problem in a stateful manner, inspired by the way WebGL itself works:

CubicVR.setContext(gl1,true);
var m1 = new CubicVR.Texture();
CubicVR.setContext(gl2,true);
var m2 = new CubicVR.Texture();

In this simple example, we attempt to create a Texture for each canvas. It’s clear which canvas owns which Texture, since explicitly switch “contexts”. CJ was quick to point out the pitfalls of this approach, and the problem sat unanswered for a while. Just think of the nightmare inflicted upon forgetful developers when they leave out a setContext somewhere and don’t understand why their renders are incorrect.

Proposal for a Solution

Fortunately, there was some work done to create Automated Testing which skirted on the multiple-canvas problem, but didn’t fully address it. It did, however influence the codebase in a way which was already occuring in several other spots.

CubicVR.js uses a require.js-like module structure, where modules are registered by the core, and executed before initialization to ensure the CubicVR namepsace is full of all the goodies that one would expect, like CubicVR.Mesh and CubicVR.Scene. What I hadn’t thought of before was how much this design allows the isolated initialization of each module.

In other words, why not initialize each registered module whenever a call to CubicVR.init or CubicVR.start occurs, and attach the result to a properly scoped object? So, I did just that: I forced much of the CubicVR namepsace into a closure which is instantiated by a call to CubicVR.init or CubicVR.start. My proposed solution makes the return value from a call to either function a CubicVR “Core”, which is basically the entire namespace as you would expect it. For example:

var core = CubicVR.init( canvas ),
    texture = core.Texture( "lolcat.gif" ),
    mesh = core.Mesh( catMeshDescription ),
    scene = core.Scene( 300, 300, 60 );

mesh.prepare();
var sceneObj = core.SceneObject( mesh );

scene.bind( sceneObj );

This code looks remarkably similar to current CubicVR.js code. In case you missed it, the only change is that I’ve replaced every CubicVR with core, except where completely necessary, like initialization. With this method, only a few fundamental functions are attached to the global CubicVR namespace, so many versions of CubicVR.js’ modules can exist in harmony.

The branch in which I’m doing my multi-canvas support work contains a test of the functionality in the instancing test folder, which demonstrates the ability to have CubicVR.js render to multiple canvas elements, and control them independently with respective MouseViewController objects. I thought interactivity would be particularly convincing.

A bit more work needs to be done to update the rest of the tests and samples with the syntax change I’ve proposed, and I’d like to get some more feedback from CJ, but I’d say this work puts us in the position to have multiple canvas support in the very near future of CubicVR.js.

Rendering a Conclusion…

I’m glad Paladin and Gladius are affecting CubicVR.js so positively. The focus is forcing me to address problems that are otherwise easy to ignore. More importantly, however, there is much more activity from contributors recently, many of which are involved in Dave Humphrey’s courses at Seneca. The sudden increase in bug fixes and feature additions in Paladin as a whole – constituents included – is really starting to show in the way of more stable tools.

More progress soon. Expect big things.

Gamepad Vibration - Zero to Hero

2011-12-09T23:31:00-05:00

If you’ve been paying attention to the recent surge of activity surrounding implementation of game-related browser API’s, you’ve probably already caught wind of Mozilla’s stream of cross-platform gamepad-ready try-builds, supporting libraries and working RescueFox demos, or perhaps Google’s announcement of Scott Graham’s Gamepad API and his game to show it off.

Today, I’m happy to announce that, as far as I know, the first working prototype of force-feedback (rumble) support is available to play with. I also forked Jon Buckley’s input.js to augment its test suite with vibrational goodness.

Zero → Discovery

Watching Dave Humphrey lead a class of students at Seneca College to implement mouse locking in Firefox really inspired me to do some digging for myself. It was an enlightening experience to partake in one of his classes and subsequently hack around with Jon Buckley on adding XInput support to Ted Mielczarek’s recent Firefox Gamepad API implementation.

Optimism with FF_RUMBLE and Friends

Obviously, having rumble support is strictly cooler than input-only gamepad support (which is already very cool). So, I thought about how to get it done which was ostensibly straightforward, as things usually are at first. My development system of choice being Linux, I dove into some google searches for generic rumble support and found some helpful, but scant and unsupported information. Getting operational info from my gamepad was the easy part.

unsigned long features[4];
int num_effects;

ioctl(fd, EVIOCGBIT(EV_FF, sizeof(features)), features); 
ioctl(fd, EVIOCGEFFECTS, &num_effects);

Actually causing some force feedback to occur requires two steps:

Prepare an “effect”, which is a reusable construct that tells something which supports force feedback how to react. They have different types like FF_RUMBLE, or FF_PERIODIC, which you can find out by bit-testing the features array against these constants. Once it’s created, upload it to the controller. You can upload several effects in a row and activate them at will.
When force feedback actually needs to occur (for instance, when your car hits a wall, or some reasonable physics-related facsimile thereof), create an “input event” to activate an effect, and upload it to the controller.

The C++ looks a bit like this:

//setup effect
ff_effect effect;
effect.type = FF_RUMBLE;
effect.id = -1;
effect.u.rumble.strong_magnitude = 0;
effect.u.rumble.weak_magnitude   = 0xc000;
effect.replay.length = 5000;
effect.replay.delay  = 0;

//upload effect
ioctl(fd, EVIOCSFF, &effect);

//setup input event
struct input_event play;
play.type = EV_FF;
play.code = effect.id;
play.value = 1;

//upload input event
write(fd, (const void*) &play, sizeof(play));

One thing I need to point out is that you can run out of space rather quickly on your device. My Logitech F510 can hold 15 independent effects, and it’s not entirely obvious how to reuse an effect without taking up another effect slot on your gamepad.

The key is this line:

effect.id = -1;

If the effect id is set to -1, uploading the effect will reset it to something > -1 to identify the effect. If your code sets it to -1 again, you will upload a new effect with the same properties, and the struct will get a new id. However, if you just want to update the effect’s parameters and re-upload it, don’t touch the id. An id of greater than -1 will be assumed to identify an effect that already exists.

So, we create effects, upload them, activate them, and that should be it, right? Problem solved! Mission accomplished! Gamepad is vibrating…! Well, no, not quite…

Trouble

Alas, initially, uploading effects always failed (returned -1 from ioctl). I searched around for some answers, and discovered three imporant things:

ff_test, a tiny program that tests force feedback for devices in linux, exists and comes in most Linux package managers with jscal, joystick-utils, or just by itself.
In order for O_RDWR access to be granted when opening an fd (so you can actually write to the device and not just read from it), you need to have full access to the device. Fortunately, this is easily temporarily attainable with sudo.
As far as I can tell, support for the PS3 controller with default Linux gamepad drivers won’t give you force-feedback. I’m still not entirely convinced I’m not doing something wrong, but purchasing a Logitech F510 for testing instead made me care a lot less (because it just works).

Of course, ff_test didn’t want to install cleanly on my x86_64 Arch Linux setup, but luckily, finding and compiling the ff_test source code was a tremendous help to my progress anyway. Here, I had a working piece of software that let me read how it made force-feedback work. Learning by example is incredible in these situations.

I made so much progress in finding out what I needed and what I had been doing wrong, and was understandably excited to arrive back at my desk, rip open my new F510, and plug it in, almost all in one continuous motion. Then, disappointment: still no rumble, even though the words “Rumble Support” decorated the gamepad’s box. After a bit more digging and help from Ted, I found out that you can’t actually write to /dev/input/jsXX fd’s; you have to use their /dev/input/eventXX counterpart! Using the event instead, my F510 started to come alive.

The really fun part is finding out which event device corresponds to a given js device. If you’re trying ff_test for yourself and failing, try

sudo ff_test /dev/input/eventXX

where XX is the correct device number corresponding to your gamepad. If you don’t know which one is the one you want, read dmesg or just… try them all (start with 0). But, if you want this to happen programatically, say in Firefox, you need a different, more dependable solution.

udev

Thankfully, udev will get you most of the way there. If you’re running a modern Linux distro, you should have access to udevadm which will give you a lot of useful information about HID’s. Try this:

for i in /dev/input/js*;do udevadm info --query=all --name=$i;done

Gamepads and their corresponding event devices will have an entry reading ID_INPUT_JOYSTICK=1 which is very helpful for identification. However , if you have more than one gamepad plugged in, they will all have that property. So, we need to do some matching based on another property. I decided on ID_PATH because looks like some sort of usb identifier. This might change in favour of a quicker, or less sloppy method, but here’s how I’m doing it right now:

bool
LinuxGamepadService::is_gamepad_event(struct udev_device* dev) {
  if (!mUdev.udev_device_get_property_value(dev, "ID_INPUT_JOYSTICK"))
    return false;
  
  const char* devpath = mUdev.udev_device_get_devnode(dev);
  const char kJS[] = "/dev/input/event";
  if ( !devpath || strncmp(kJS, devpath, sizeof(kJS) - 1) != 0)
    return false;

  return true;
}

struct udev_device* 
LinuxGamepadService::get_corresponding_event_device(const char* cmp_id_path) {

  struct udev_enumerate* en = mUdev.udev_enumerate_new(mUdev.udev);
  mUdev.udev_enumerate_add_match_subsystem(en, "input");
  mUdev.udev_enumerate_scan_devices(en);
  struct udev_list_entry *dev_list_entry;
  for (dev_list_entry = mUdev.udev_enumerate_get_list_entry(en);
       dev_list_entry != NULL;
       dev_list_entry = mUdev.udev_list_entry_get_next(dev_list_entry)) {
    const char* path = mUdev.udev_list_entry_get_name(dev_list_entry);
    struct udev_device* dev = mUdev.udev_device_new_from_syspath(mUdev.udev, path);
    const char* id_path = mUdev.udev_device_get_property_value(dev, "ID_INPUT_JOYSTICK");

    if (is_gamepad_event(dev) && 
        strncmp( cmp_id_path, id_path, sizeof(cmp_id_path) - 1) == 0 ) {
      return dev;
    }
  }



  return NULL;
}

const char* id_path = mUdev.udev_device_get_property_value(dev, "ID_INPUT_JOYSTICK");
struct udev_device* event = get_corresponding_event_device(id_path);

NOTE: There is definitely a bug in this code, which I will fix as soon as I feel like programming again. Try to spot it: it’s something to do with the property I’m trying to use for comparison (ID_PATH). I get the matching event device, but until I fix this bug, it will only work in some cases.

Using this technique, I was able to consistently open two associated file descriptors: /dev/input/js0 for reading, and /dev/input/event11 for writing. And guess what. My gamepad started rumbling.

DOM

Somehow, the easiest part of all of this work was getting the JavaScript Gamepad object to expose a new set of functions for vibration. Simply altering nsIDOMGamepad.idl gave me most of what I wanted, and I was able to find and copy a couple of other existing IDL files which gave me the rest.

Just to be able to set the vibration state, I needed a function in the gamepad interface like this,

void setVibration(in unsigned long duration, in unsigned long strongMagnitude, in unsigned long weakMagnitude);

which, on the implementation side, turned into C++ code like this:

NS_IMETHODIMP nsDOMGamepad::SetVibration(PRUint32 aDuration, PRUint32 aStrongMagnitude, PRUint32 aWeakMagnitude, PRUint8 aArgsc) 
{
  mGamepadService->SetGamepadVibration(mIndex, aDuration, aStrongMagnitude, aWeakMagnitude);
  return NS_OK;
}

Ted’s Gamepad API code is split up so that gamepad services are implemented on a per-OS basis, but they all inherit from a GamepadService class. So, in the GamepadService and LinuxGamepadService classes, I created SetGamepadVibration to deliver vibration state changes from Gamepad DOM objects to the service.

The only piece left was actually talking to the device.

Using `select` and `pipe`

Having done a considerable amount of detective work, learning about udev and the kernel in the process, the end felt near. I just needed to wire up the browser to push values from JavaScript to the gamepad service. The trouble was that an OS-specific service ran in its own thread to talk to devices, using select to wait for events, processing them, and sending consequent messages to the main thread. So, getting data to move in the other direction (from DOM objects to the Gamepad service; from the main thread to the service thread) would have to involve pipes.

pipes are simple communication channels which two running bodies (threads, processes, etc.) can use to send each other information. In simple terms, a pipe can be written to with write, and then read from with read, and all the data should arrive in the order it was sent. So, sending a vibration event over a pipe is as simple as writing a few pieces about the event in sequence. So, the main thread writes to the pipe like this:

write(mVibePipefds[1], &index, sizeof(index));
write(mVibePipefds[1], &duration, sizeof(duration));
write(mVibePipefds[1], &strongMag, sizeof(strongMag));
write(mVibePipefds[1], &weakMag, sizeof(weakMag));

where mVibePipefds is the pipe, mVibePipefds[1] is the end of the pipe to which we write, and &index, &duration, &strongMag, and &weakMag are the pieces of data we want to deliver to the service.

On the receiving end the code looks incredibly similar:

read(self->mVibePipefds[0], &index, sizeof(index));
read(self->mVibePipefds[0], &duration, sizeof(duration));
read(self->mVibePipefds[0], &strongMag, sizeof(strongMag));
read(self->mVibePipefds[0], &weakMag, sizeof(weakMag));

the only different being mVibePipefds[0] which is the end of the pipe from which we read.

I put together a handy function to finish the job:

void
LinuxGamepadService::SetVibration(Gamepad& gamepad, PRUint32 duration, PRUint32 weakMag, PRUint32 strongMag)
{
  if (gamepad.eventFd > -1) {

    //set up the effect
    //TODO: safety-check effect values
    gamepad.ffEffect.u.rumble.strong_magnitude = strongMag;
    gamepad.ffEffect.u.rumble.weak_magnitude = weakMag;
    gamepad.ffEffect.replay.length = duration;
    gamepad.ffEffect.replay.delay = 0;

    //upload the effect
    if (ioctl(gamepad.eventFd, EVIOCSFF, &(gamepad.ffEffect)) == -1) {
      printf("failed to set vibration :(\n");
    }

    //set up the command
    gamepad.ffInputEvent.type = EV_FF;
    gamepad.ffInputEvent.code = gamepad.ffEffect.id;
    gamepad.ffInputEvent.value = 1;

    //upload the command
    if (write(gamepad.eventFd, (const void*) &(gamepad.ffInputEvent), sizeof(gamepad.ffInputEvent)) == -1) {
      printf( "failed to activate vibration :(\n" );
    }
  } //if
}

Most of the code here is code I’ve already explained, so none of it should be surprising. I just wanted to consolidate it properly. Note, though, that it’s incomplete (as you can plainly see), and note that I’m not ever changing the id of gamepad.ffEffect, so I reuse only one effect per device.

Finally, some on-command rumbling from JavaScript.

Proof

As soon as I had this working from front to back, I had to prove it to the web. So, I filmed it:

On to the Hero Phase

Getting this work past the prototype phase is going to take some work:

Implementing cross-platform support, which means augmenting the Windows and OSX services at least.
Getting the community to help decide on a good API. There is a WebVibrator bug with which it would be useful to coordinate so that using any vibration capabilities on the web seems similar and natural. It’s also very useful to replicate something like XInput’s API for vibration, since it’s already used (a lot) in practice, and a lot of devices are being produced to support XInput specifically.

If you’d like to follow along, add yourself to the cc list on the bug which I’ve very recently inherited. Let me know your thoughts. More progress soon.

Popcorn Maker 0.1 - For The Developers

2011-11-24T10:30:00-05:00

After my recent stint in Europe for the Mozilla Festival and MozCamp I had the opportunity to reflect on the development environment of Popcorn Maker, courtesy of the 8 hour flight from Berlin to New York. The result is a version of Popcorn Maker which can be proudly labeled as 0.1, and very friendly to developers.

History Lesson: Butter

Butter was an initial exploration of making a GUI for producers hoping to use popcorn.js for their projects. The app enjoyed some success, but it was eclipsed by the need to expand the app in different directions, which I have written about previously. Like most promising experiments, it needed to be rewritten to be the powerful app we hoped it would become.

Consequently, there was a lot of discussion about how to engineer the experience which was to become Popcorn Maker, which Brett and Ben detailed in their post, Designing the Popcorn Maker. In short, the current goal is to create a web app which encompasses the same ideas as Butter, but makes traditional content producers (usually of the film persuasion) comfortable by drawing memes from other popular pieces of software. However, a functioning web app is only a piece of the puzzle, since harnessing community involvement is a very powerful concept. Eventually, users will be able to use Popcorn Maker like they use WordPress – by downloading and installing the app on a machine of their choice – but with unique sharing capabilities and editing opportunities, like publishing your popcornized^{^tm} template which might be a remix of something like a youtube video page.

Require Saves The Day

Popcorn Maker"> The subsequent struggle to design an acceptable user experience and its contention with pace of development and fluctuating feature requests hasn’t offered the opportunity for Popcorn Maker’s codebase to settle. As a result, Popcorn Maker’s developers have had to suffer through a poor debugging experience and a sub-optimal development environment.

Now, that has all changed. Some geniune thinking time and my recent exploits inside Paladin with James Burke, the creator of require.js, has changed Popcorn Maker for the better – for developer and users alike.

Using require.js, I was able to split up Butter’s core and constituent modules into several files respectively. require’s module system makes this process natural and provides a method for producing dependency chains. So, Butter’s core relies on the existence of several smaller modules like Media, Track, and TrackEvent, while Butter’s Timeline module depends on some of the same (plus some others). In other words, dependency chains let a module rely on other modules which may be already shared amongst others still. Of course, this is nothing new to traditional developers who get to use #import to solve a similar problem.

Now, developers are able to isolate and fix bugs without involving a compilation process every time they make a change (since Butter is best suited as a library inside of Popcorn Maker).

Onward: To 0.1

Since Popcorn Maker is an app that wraps Butter, developers can contribute from several perspectives:

Popcorn Maker: developing the app itself (UI, storage, exporting, etc.)
Templates: building general-purpose Butter templates to let users jump into editing a powerful popcorn experience quickly
Editors: providing html-based editors for popcorn plugins which would benefit from specific UI (e.g. Google Maps)
Butter: enhancing the core of Butter to fix bugs or provide better functionality or compatibility (standards like to change)

The beauty of the approach we’ve taken in 0.1 is that these elements are as decoupled as possible because they are actually separate require projects. Then, it’s no problem at all to have them share components, and they can depend on each other to provide a nice unidirectional development flow.

Part of the magic is a pattern which James has worked on that circumvents the data-main attribute traditionally attached to a require.js script tag. Instead, a script like this is referenced (in this case, butter.previewer.js):

(function () {
    // Stub for has function.
    function has() {
        return true;
    }

    if ( has( 'source-config' ) ) {
        // Get the location of the butter source.
        // The last script tag should be the butter source
        // tag since in dev, it will be a blocking script tag,
        // so latest tag is the one for this script.
        var scripts = document.getElementsByTagName( 'script' ),
        path = scripts[scripts.length - 1].src;
        path = path.split( '/' );
        path.pop();
        path = path.join( '/' ) + '/';

        document.write( '');
    }

    var ButterTemplate = function() {
      if ( !ButterTemplate.__waiting ) {
        ButterTemplate.__waiting = [];
      } //if
      ButterTemplate.__waiting.push( arguments );
    }; //ButterTemplate

    if ( !window.ButterTemplate ) {
      window.ButterTemplate = ButterTemplate;
    } //if

}());

This pattern, which writes its own require.js-specific script tag when necessary (when not compiled), takes advantage of require contexts (returned from a call to require.config()), and is easily generalized. In fact, it’s used to load event editors, and butter itself. In other words, Butter (and Popcorn Maker, by extension) contains several require projects, each of which can be compiled to provide a minified version. Butter is actually a collection of libraries which can be included independently to provide functionality specific to things like Templates or Event Editors.

So, to create a simple Butter Template, all you need is this:

Similarly, to create an Event Editor, you need this:

Finally, Butter can be imported in the same way (in Popcorn Maker, for example):

  
    Popcorn Maker

Notice the extra require inclusion here, which is necessary only because Popcorn Maker is also a require.js project. This way butter.js does not depend on Popcorn Maker to include require.js, and every layer of the project can be nicely compiled.

To reiterate, the very best part is that there is no need to re-compile Butter if a change needs to be made. require.js will let us compile the project only when we want to distribute it.

Enjoy

So, have a look at the app at mozillapopcorn.org/maker(on Monday), or clone the source for yourself to see what you can do locally. As always, reporting bugs is paramount, and is worth almost as much as fixing them. Check out our 0.1 milestone on our lighthouse project, where you’ll almost certainly find a plethora of things to do.

Gladius Gets a Renderer

2011-11-23T23:22:00-05:00

I just finished committing and issuing a pull request for the bones of Gladius’ graphics service. Alan and I have been talking about how to do this properly for at least two months now, so it really feels good to write down some code. Here’s how it looks:

A little explanation first…

Keeping modern game engine design in mind, Gladius implements an Entity/Component system where Components such as Models or Cameras are added to Entities to give them functionality. Components are supported by different systems called Services, like the Graphics service. This approach circumvents the need for a bloated, all-inclusive base class from which plants, paladins, and planets all inherit to receive common functionality. Instead, with a proper dependency system, a Camera or Model component can insist that there is a Transform component on the entity to which it’s being added so it can move through – and interact with – space without spilling remnants of itself onto other game objects through inheritance.

So, making an object appear on the screen is as simple as adding a model component to an entity in a scene. Of course, a model needs data, and there are plans being forged to support all sorts of inputs for meshes. However, that’s another discussion entirely. Right now, all you need to know is that Gladius has the fundamentals in place to build basic procedural meshes (a cube, really).

Rendering

The graphics service can achieve a complete render of the entire engine’s state by following this short, pithy, pseudo-code:

For each scene in the engine, find two lists: cameras and models.
For each active camera in a scene, render all of the scene’s models.

This tiny algorithm translates into this longer, Gladiusized-pseudo-code:

Clear the current drawable object (right now, a
For each scene in the engine, get lists of entities with camera and model components respectively.
For each entry in the camera list, grab the camera component; if its an active camera, render the mesh in each model in the model list.

It’s a great starting point, but there are very obvious inefficiencies and missing features in the above process, including…

The inability to render to anything except a canvas: In the near future, we’ll write code to present the proper abstractions to developers which will allow different renderables to be rendered for a given camera. You may want to render to a buffer, a canvas, or a textbox. We were thinking about naming these Films, but Renderable (which is less cool) might also work.
A lack of culling: Right now, rendering from the perspective of a camera is expensive in a large scene with many objects. We need a way to slice up the scene so that we can discard what’s not in view and only render what’s important. But, don’t worry: my octree.js library is nearing completion and will soon fill the void for a space partition culling mechanism. It’s very similar in spirit to the octree implementation in CubicVR.js which I also wrote (unfortunately, with strong evidence of the job I did porting it from a Java implementation I wrote a couple of years ago).

Also, since the graphics service distributes cameras, there is the potential to avoid storing a list of scenes. Camera entities are parented to a scene, so a list of models (and other important things in the future) relavent to a camera can be extracted from its parent. This approach will be useful when spatial partitioning lands, since a run through an octree with the frustum of a specific camera will yield a list of visible models to render.

CubicVR.js & Graphics Services

And where would we be without CubicVR? Well, there would be a significant gap in the Dreamcast homebrew 3D engine community. But, more importantly, we wouldn’t have the talent and engineering behind the engine that drove Flight of the Navigator, No Comply, and Rescuefox.

CubicVR.js is Gladius’ first rendering core, and we’re glad to have it. It gracefully handles a complex and feature-rich shader pipeline, and supports all sorts of neat mesh building and deformation techniques (now with dynamic vbo’s). Check out the samples directory for a bunch of hacker’s delight and eye candy.

Currently, we’re wrapping CubicVR’s Camera and Mesh objects with responsibly named components: Camera and Mesh. Since the graphics service currently in Gladius is specific to CubicVR.js, it knows about the native CubicVR.js objects which live on components and how to make them cooperate.

Our hope, of course, is to support a wide variety of rendering engines in the future – even 2D ones. Each would require an implementation of the graphics service and the core grpahics components to comply with the particulars of the renderer.

Try it!

The most important part of all of this is that you can grab the branch and try it out right now. It won’t do much yet, but you can tell it’s working because gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT); is doing its job by blacking out the canvas on the test page. You can change that by contributing :).

For now though, I’m just happy all the supporting infrastrcture is finally in working order to hack on this renderer. I can almost feel the games we’ll create with this engine.

Transition

2011-11-21T23:22:00-05:00

Here I am, burning the midnight oil (literally), setting up a new server. So much work to do and so much catch-up to play. Since I’m not a sysadmin by trade, my working knowledge of what’s hip in the server world is relatively scant.

But, as I slowly copy over important files from my old setup, spin up new versions of old configs ( debian etch is … quite old now :( ), and learn what programs are coolest to serve static content now, it’s coming together again.

Good bye Apache, old friend. Good bye wordpress. Good bye PHP. I won’t miss you guys very much.

Hello modern web-development world. Should I be scared?