Misadventures writing free software

Nearing the final release

2008-04-16T18:33:00.003-04:00

I got disabling updates working for Firefox, though it is terrible hack since I couldn't find a policy to disable IE updates. A google search didn't help, and most of them pointed to third party tools to disable updating to IE7 from Windows Update. So I had to manually create a registry key to disable browser updates (ironically enough, I unconsciously put it in the Internet Explorer branch of the registry).

There are still so many things to do, most of it cosmetic. For example, the code is still a mess. I have my doubts about fixing it entirely, but I can certainly comment.

So here is what is left to do, at minimum, where I stand :

Document on the wiki how to implement a new policy
Use javascript components whenever possible

And then I can finally say that I am done, and anything else is just frosting :)

Overcomplicated

2008-04-06T21:37:00.004-04:00

While doing some test cases for the javascript wrapper around registry keys, I found a major flaw in an assumption I was making about keys that made me rethink my code. While I made some things simplier, it was still a pain to get something. Here is a progression of how things used to be and how things are right now.

Declare the service :

let service = Cc["@shacknet.nu/winregistry;1"]
    .getService(Ci.nsIWindowsRegistryService);

Oldest way :

let key = service.getKey(
    Ci.nsIWindowsRegistryService.ROOT_KEY_LOCAL_MACHINE,
    "Software\\Mozilla\\Firefox\\Crash Reporter\\EmailMe");
let answer = key.getValue();
key.close();

Older way :

let key = service.getKey(
    "Software\\Mozilla\\Firefox\\Crash Reporter",
    "EmailMe");
let answer = key.getValue();
key.close();

New and simplier :

let answer = service.getKey(
    "Software\\Mozilla\\Firefox\\Crash Reporter",
    "EmailMe");

I figure, this will work for 100% of the use cases right now. I was gonna get complicated with making another, more complete, interface for getting attributes of keys (ie. what was the root, path, and key name). But this early in the process, especially without requirements, I don't care about it :)

Mystery unravelled!

2008-04-03T02:33:00.003-04:00

A bit of my last release failed. Mainly because my javascript component was failing in an odd place. I couldn't figure out why, and so xpcshell tests never got complete.

I went back to the problem today. Firing up venkman and the javascript shell, I started to play around with it some more. I manually tested (grumble) both components independently and they both worked. But they didn't work together. I couldn't get one to return an instance of the other, and it was bothering me.

Walking through the javascript in venkman, It got all the way to the return value and throws the exception there.


        if (regkey.hasValue(value) == false)
            return null;
       
        key.init(regkey, relPath, value);
       
        return key;    /* Always failing here */
    },

Looking at the terminal, I noticed a strange assertion coming up after the code fails. I asked in #developers, and even sdwilsh (a very knowledgeable and promising character that interned last year) took a stab at it and couldn't find anything obvious. I started losing hope of ever getting this blasted thing working. Looking up the assertion didn't help, and neither did a NS_ENSURE_TRUE warning that were both being fired after the code failing.

I finally had it. It couldn't be the .js file. It had to be something more. But what else is there except the idl interface? I looked, examining each line. Then I noticed something very, very, wrong.


void getKey(in unsigned long root, in string path);

The return value was void. I must have just put void there first because I didn't know what I wanted to call the other interface. I promptly change it to :


nsIWindowsRegistryKey getKey(in unsigned long root, in string path);

And it starts to work in the javascript shell, which is absolutely a relief. Now I can start seeing if tests work.

Dynamic Javascript

2008-04-01T01:57:00.002-04:00

For the next release, I want to start cleaning up the code base. I've started with grouppolicy's browser.xul (which overlay's Firefox's browser.xul) because this is a file that confuses me quite a bit. I also found out why it's seemingly so confusing.

The problem is this: Say I have a function called multiply that lives in another file :

function multiply(a,b) {
...
}

Without changing multiply's function internals, how do I change the logic? You might consider making a multiply2(a,b) and copy and pasting code and substituting your own logic, but the catch is you can't change any of the functions that call multiply() either!

It's sorta the same dilemma that we have. There are functions in Firefox's browser.xul that we don't want to change, but we want to add bits of logic jammed into the middle that would, for example, prevent a user from ctrl+click a link to open it in a new tab when tabs are disabled via a policy.

Overlays offer one solution. You can declare your own multiply(a,b) function and it would overwrite the one in Firefox's. However, I don't like this approach because you have to copy and paste code. And when you do that, you're responsible for keeping it current.

Another solution to wrap around the function. Let's expand our multiply function :

function multiply(a, b) {
    var rv = 0, i;
    // If we go over 999, just give up silently because we're evil
    if (b < 1000) { 
        for (i = 0; i < b; ++i) {
            rv += a;
        }
    }
    return rv;
}

And let's say you wanted to change 1000 to 500. One solution is to wrap the function. Something like :

var _multiply = multiply;
function multiply(a, b) {
    if (b < 500) return _multiply(a,b);
    return 0;
}

Now the real multiply doesn't get called unless the second factor is less than 500. This is the approach I'm using to solve my problem. However, there is sometimes duplicate code that you just can't get around.

A third option, which is quite innovative, is something I found someone doing on an addon. It uses eval(), which is risky but gets the job done in this situation.
eval(multiply.toString().replace("1000", "500"));
I'll leave that for you to figure out. But it works!

Yay, bedtime!

Windows Build Prerequisites change

2008-03-31T02:19:00.004-04:00

If your building from source on Windows, then it will come as the most un-welcomed surprise (well, I have been told it's been there for a while) that you now need the Window's Vista SDK to do a build. The exact reason why escapes me at the moment. Something about terrorists winning if we don't.

Makefile.in Madness

2008-03-30T22:45:00.004-04:00

I tend to really hate doing Makefile.in files because they are just copied from somewhere and they sometimes don't have everything you need. And when you don't know what your doing, that can screw you up. I created a generic Makefile.in file that is usable for extensions, idl interfaces, and C++/javascript components. It's also documented as much as I could find information on it. Just uncomment what you don't need and delete the rest. I covered as much as I could, I might not have covered every single option, but I think I got most of them.

You can also use Ted Mielczarek's Javascript Component Wizard. I dislike using wizards, but to each his own.

If you want to repay me, you can add me to the contributor's list (Cesar Oliveira) for the Makefile.in.

Building OpenKomodo

2008-03-24T23:38:00.004-04:00

I use OpenKomodo on windows all the time. It is a nice IDE for javascript and detects formats that you use in Mozilla development like .idl and .xul. The only bad thing about it is that it has a lot of Mozilla 1.8 (Firefox 2) memory problems. You can go over 700 megabytes just by doing nothing.

I wanted to have it here on my Linux box, but they do not offer a 64bit version of OpenKomodo. But because they released it late last year, I can build it. The experience was a little... refreshing. Reminded me of building Firefox for the first time.

If you can build Firefox, you should be able to build Komodo. The steps to build are outlined on their svn's README.txt. I'll have to hand it to them for getting something out in 3 months :)

There are a few things that I stumbled on when doing this that I will note here :

There build searches for your distro. I got an error :
platinfo.InternalError: Could not determine distro & release from first line of
'/etc/bluewhite64-version' (Bluewhite64 12.0.0).
I had to add "Bluewhite64": re.compile("^Bluewhite64 ([\d\.]+)"), to line 583 of util/platinfo.py and later src/release/codeintel/support/platinfo.py.


I had to build python because they only offered the x86 binary. Two mistakes I made were trying to have it find python on my system and symbolic linking python. At one point during the install, something was taking a lot of time and disk was spinning like crazy. I stopped and looked and it was copying everything in /usr to some other directory. A time consuming process indeed.
To solve this, I :

got python 2.5 source
created a folder in my home directory called root: mkdir $HOME/root
untarred and configured python with --prefix $HOME/root and --enable-shared
compiled. Then I went to $HOME/root and zipped up the contents : zip -r linux-x86_64.zip .
Copied linux-x86_64.zip to mozilla/prebuilt/python-2.5/


Undefined reference to X...., was a already patched but not committed bug.

There is something in linux-x86.zip python binary that isn't in the linux-x86_64.zip binary we created. Had to unzip lib/python2.5/site-packages/activestate.py from linux-x86.zip and copy it to build/cvs1.8-ko4.3/mozilla/ko-rel-gtk2-ns-shared-tools/dist/python/lib/python2.5/site-packages/

I had to explicitly say the version : bk configure -V 4.3.0-devel


I hope this is useful for anyone else who wants to build it.



yay, incompetence
2008-03-19T21:07:00.007-04:00
So I have started doing XPCShell tests as part of my next release. This has been giving me a lot of trouble, and not because tests are failing, but because my components and interfaces weren't being registered to XPCShell's liking (the component works fine in Firefox). Today I find, it was because of my incompetence.

When I exit XPCShell, I got the following message (XPCShell prints its error message when you quit on Windows, unless you call the quit() function).

JS Component Loader: ERROR file:///c:/builds/firefox-debug/mozilla/mozobj/dist/bin/modules/XPCOMUtils.jsm:115
                     TypeError: i is undefined

This is particularly tricky because it looks like its an XPCOMUtils.jsm error, when in fact it isn't. The mostly likely reason is that a component is trying to use an interface it cannot find, and therefore the component fails to register. But why can't it find it? I checked XPCShell and got the expected message :

js> print(Components.interfaces.nsIWindowsRegistryService)
undefined

After asking in #developers (twice) thinking that I need to add an EXPORT line to Makefile.in. Waldo pointed me to Javascript HTTP server code in netwerk/test/httpserver/Makefile.in. I compared Makefiles, and noticed one thing that may be causing my problem :

  46 MODULE          = test_necko

I kept everything under one module (grouppolicy). The tests being in the same module as the extension seemed like one rational reason why my code was working in Firefox but not in XPCShell, so I changed by module from grouppolicy to test_grouppolicy.

And holy **** did it work.

js> print(Components.interfaces.nsIWindowsRegistryService)
nsIWindowsRegistryService

I was a little disappointed that I had to spend two days on such a simple mistake. But I was happy enough that I stopped working to start this blog post. But it made me think about how little I know about what I'm doing sometimes, because we copy and paste a lot of things (Makefile.in, install.rdf). And naturally, when something goes wrong we don't know what to do. In fact, I don't know what a lot of lines in a typical Makefile.in do. But it is mistakes like these that help you learn.


Getting this done, one policy at a time
2008-03-12T22:17:00.008-04:00
So while the bug has been posted, I wanted to go into more details about the policies that have been implemented so far. The complete list is on a google spreadsheet, but I will outline some of them :

Apologies to the smaller images. You can click most of them for full resolution.

Enforce Full Screen : Some people have been asking for this, for a kiosk setting. This looks identical to IE7 (it's not full-screen by default). The navigation toolbar, bookmarks toolbar, and menu are all gone along with the status bar.

Disable changing the home page : As you'd expect.

Disable closing the browser and Explorer windows : This is pretty funny, neither File->close nor alt-f4 work. Interestingly, IE7 throws an alert box if you try closing it from the task manager, but your prompted a few seconds later with a dialog box telling you that the application isn't responding and whether you want to force quit. The force quit works. Firefox3 does almost the same thing, but doesn't throw an alert box.

Disable Context Menu : Disables right-click context menu on web pages

Hide Favorites menu : I don't know why you would want to do this, but it's a policy.

Turn off displaying the Internet Explorer Help Menu : Really folks, wtf?

Turn off tabbed browsing : This makes some people sad. If you set the policy after creating tabs and saving the session, the tabs still appear. So still some bugs :)

Proxy Settings : Yay, my part. I guarantee it works or your money back! Here I am connected to a Japanese proxy server and went to google.com


The achilles' heel in my entire project right now is about:config and chrome://. But we can settle that eventually. Right now, the project needs a lot of code polish and tests.


Release
2008-03-09T04:28:00.002-04:00
Yes, at around 4:20 in the morning, bug 267888 got a long awaited response. Well, actually it was the first comment in over a year for that bug, so a toast to that!

This release is about getting community feedback, and hopefully get a dialog going. Best case scenario is that people tell me to keep going and aim for Firefox4 or something. Worst case, I get nothing, and that's bottom of the barrel. But I have hopes that people will say something (probably to disagree verbosely against generic IE policies).

Next for the extension is a lot of code cleanup. There are several areas where it is difficult to read, or it's not apparent what's happening.
There also needs to be tests. Maybe some reftests? A requirement if people want this in the codebase.

Time for sleep.


FUELing the fire
2008-03-04T22:34:00.004-05:00
While doing some work on updating one of the policies to use FUEL, I hit upon a bug in the library. While I wasn't in the mood to patch (it is a trivial fix), I thought this would be a great opportunity to try some Mochitests.

Previously I discussed adding js components to make getting registry values easier, it would nice to have some tests in the process.

I actually rarely do tests actually. I shy away from them for several reasons, but I want to try and turn around this bad habit. I strongly dislike doing manual tests, because if something goes wrong and you get a fix, you have to make sure nothing else broke in the process. Thankfully, Mozilla offers some choice in doing your tests.

Starting Mochitest was acutally a problem. All your tests and utilities are in mozobj/_tests/testing/mochitest/. You must be in that directory when running the mochitests. The tricky bit seems to finding where exactly your tests are. There seems to be a few possiblities:

browser/*
chrome/*
tests/*

The directory structure follows the tree. For example, since FUEL's tests are in mozilla/browser/fuel/test/, the path to fuel's chrome tests (if it had any) would be in chrome/browser/fuel/test/.
The script running the tests is a perl script called runtests.pl. Depending on the parameters, it looks in different places

--chrome is the chrome/* directory
--browser-chrome is the browser/* directory
nothing is the tests/* directory

You can pass the --test-path=browser/fuel/test/some_test.html to do one unit test, or test the entire directory by doing --test-path=browser/fuel/test/ --auto-run. If you leave --test-path=... out, it will test everything. Since my code is in browser/, I did the following command :

perl runtests.pl --browser-chrome --test-path=browser/fuel/test/ --autorun

Predictably, it passed. That's because it wasn't testing a certain condition. I submitted a test case with the bug. I still know very little about how to create a Mochitest, but how to run the tests was just as important as the testcase itself. So in that sense, I am happy with the results.

Inspirational link


Using Venkman to combat terrorism and save freedom
2008-02-27T17:00:00.009-05:00
Venkman has been a big help during this release, and it something that I want to share. I have been avoiding using a javascript for a while, and that's mainly because I opted to use alert() and dump() instead. But after a while, they both suck and can make debugging incredibly frustrating especially if you have to restart firefox each time to go one step ahead.

Introduction
It's been out since 0.9 folks!


Install
Grab the extension

Setting it up
Under Debug menu, make sure "Exclude Browser Files" is not checked.
Under the File menu, check "Save Break/Watch Settings on Exit".

Setting Breakpoints
This is reasonably simple. On the left are your files. Here I am debugging ProxyEnable.js. 

Below it are the functions within that file. Double-clicking on the filename brings up the source code on the right.
Place your breakpoints. The dashes near the left side of the source are where you can put your breakpoints. Here I am putting a breakpoint just before the if-statement.
Exit venkman (this will close firefox)
Relaunch firefox with -venkman argument. For example :
/path/to/firefox -venkman --no-remote -profile /path/to/profile

Once you restart, venkman starts first and catches any breakpoints.

This is ideal when you can't use browser chrome to initiate the javascript, for example, js code that is executed on startup.

Fun
Once the debugger breaks, you have access to all the variables. So you can start investigating the problem much more effectively than with dump. You use the black window at the bottom as an interactive javascript shell.


Navigation
The buttons at the top let you go to the next line, go deeper into js functions, or continue. Try them out.


Now Code
No more excuses.


x86_64 woes
2008-02-23T21:31:00.005-05:00
I now have an amd64 (Sempron) desktop and an emt64 (Core 2 Duo) laptop and fully 64-bit and Y2K38 ready, which I am quickly finding out that it doesn't get it's share of love.

For example, Firefox/Thunderbird nightlies and OpenOffice don't have x86_64 builds. Now, if you have a 32-bit OS on a x86_64 architecture, it's ok and you can emulate. But if your running x86_64 without 32-bit libraries, your in trouble.

Fortunately, my desktop is equipped with Ubuntu, and the greatest package manager, ever.
All you should need to do is :

sudo apt-get install ia32-libs
sudo apt-get install ia32-gtk


This runs prism with a few warnings about an unsupported locale or something. Which is a lot better than compiling it from source.

My laptop however uses BlueWhite64, and I by slackwarian code (this is a complete lie), must compile everything from source. Unfortunately, here I don't have any of the 32-bit libs. So that's a lot of compiling. I wonder if it's even worth it. I'm fairly tempted to just skip it and download the i386 packages. But that would be taking the ubuntu way out, and I'm likely to get lynched by people like me.


Dynamic Proxy Settings
2008-02-18T21:51:00.009-05:00
There were three things I was planning on doing for the previous release that didn't get finished, and something extra that I didn't plan on doing and probably won't be done (priorities and such) but interesting to know that it exists.

Proxy Settings using group policy

Disable changing the proxy settings

Create some Javascript Wrappers around the XPCOM components for locating/reading registry keys


Proxy Settings using Group Policy
This was something that almost everyone that commented said was an important policy to them. I think I only once had to ever use a proxy (I wanted to watch a John Stewart clip from a website then wouldn't let me in, so I had to proxy to an American server to view it). Regardless, if it's the most popular request, I'll do what I can.
I have it working for http:// sites. Still need to make changes for the other protocols. Mossop also taught me a lesson that <preferences> requires a <prefwindow>. That's going to be a problem when you have to use <overlay>'s instead of a prefWindow, so I had to use an nsIPrefBranch instead. On the lighter side, I can refactor and finally have an excuse to use FUEL.

Disable changing the proxy settings
nsIPrefBranch supports locking preferences so that the user cannot change them in about:config or in the UI (it will grey out the settings).

Unfortunately, you can only lock the default value. Which kinda sucks at the moment. The other two alternatives is to make the option of changing your proxy greyed out or hidden. IE7 has it's own tab for connection which completely disappears with the right policy enabled. Since Firefox's proxy settings is located in the same UI tab as offline storage, it is probably best for us to disable the button. To avoid allowing people to change the settings via about:config, we would have a policy to disable about:config access (get back to me on this, I don't have an answer on how we can do that yet).

Javascript Wrappers around the XPCOM components for locating/reading registry keys
I started doing some work on doing some JS wrappers around registry keys, since I dislike the current implementation to a certain degree. I won't go into details, because it's more of a preference then anything else.

I'll end with an amusing quotation.
And _please_ file bugs when you hit problems like this.  I flunked long-distance widebeam telepathy in college, so I'm no good at just guessing that you're running into problems...
-- Boris Zbarsky



Generic IE policies in Firefox
2008-02-06T14:39:00.002-05:00
This is something neat that I cannot take credit for. I just put it in a tree-friendly extension. I got the source onto an svn repository for people to play around with :

svn://cdot.senecac.on.ca/ff-ad

I put up some slightly more detailed steps into the wiki on how to set up Firefox. In the meantime, some explanation of what IE settings means for Firefox.

IE has several policies that Firefox can use, for example you can disable tabs and the help menu. Below we have the example of turning off the help menu

Ignore the bit about IE for a moment.

If you open up Firefox, you get the predicted results

Which is ok. If you look elsewhere, you'll see that not all the help functionality is disabled.

Which is something that still needs to be fixed. I also found some javascript errors that I would like to fix as well.

But the problem now is the overwhelming amount of policies in place. Mike's blog should help me out a fair bit with this, as he has got some response. More updates as this goes along.

==UPDATE==
So I was a little bit wrong. If you read the entire IE policy, it doesn't explicitly say that all help functionality will be turned off. To make a liar out of me, you can go into Options and click on the question mark at the top-right corner and help will popup. Just the help menu is disabled it seems (and F1).


Back on the yellow brick road
2008-01-30T20:13:00.000-05:00
After a series of unfortunate events, things seem getting better. I got enough back that I can start working again.

I started looking through some of FirefoxADM and WetDog code a while back, and I have been grossly misjudged these programs. Neither of them actually turned out to be extensions. This isn't a bad thing actually. FirefoxADM seems to be a fancy visual basic script that reads the registry and makes pref changes. WetDog is some C++ code (I haven't analyzed it too closely).

One of the things that I was worried about early on was redoing the work that has already been done. That doesn't look like it's going to happen. However, now the thing is trying to get enough work done for the next release.

Mike Kaply has started doing some work on using overlays to disable certain functionality, and loading those overlays dynamically. I need another copy of that program, put it into an extension if it's not already, and lubricate the firefox tree and jam it in. I also want to be able to set those overlays based on some registry values (there is a big fuss in bugzilla on where these key/value pairs should be. But that seems to have gotten in the way of getting actual work done).

This will probably be a giant hack, but it will be a good start. And it can be refined later as the semester goes.

Also, Mike made a useful blog post getting comments from people who have enterprise experience trying to figure out what enterprises would like to see. This will do until I can find some enterprises to collaberate with.


A leap of faith
2008-01-14T01:05:00.000-05:00
For the next three and a half months, I will be working on integrating Active Directory into Firefox. This may come as a shock (or not), but after working on Linux for anything remotely productive in the past 3 years, my Windows knowledge could use a fair bit of upgrading.

Why is Active Directory special?
It isn't useful for the general public, but rather for Enterprise administrators who want to control the preferences of the browser. For example, let's say you want to have a default homepage for Firefox. How can you do this without going to each machine and setting the preference? Without re-imaging (Seneca's solution)?

Setting homepage for IE7 (don't ask why it says IE6)

Well, you can do it Firefox, sorta. You need to download an extension, at the very least. I haven't actually tried it. I may be wrong. And it is probably more complicated than that.

The solution seems to be Active Directory and Group Policies. Or it seems to be anyways. The concept is still new to me, but this is  the next few months of work. It is not only required piecing this stuff together (and maybe some upgrading), but also working at it until it gets to the tree. I will hopefully understand Andrew's pain.

My first release will be learning about GPO, and starting to take apart extensions like CCK and FirefoxADM. I will have some documentation, definitely up on the wiki


An introduction
2008-01-14T00:46:00.000-05:00
Firstly, an introduction.

My name is Cesar Oliveira. I am my 4th year of a Bachelors of Software Development at Seneca College. That is really all you need to know about me.

I have decided to make this blog, separate from my wordpress blog, mainly because I can more freely post here without spamming planet@mozilla. I will still post in wordpress, if the announcement I feel needs to reach more people.

This blog will mainly deal with my experience working with Mozilla. I have been working with Mozilla (both as a intern and as a volunteer). I am now doing a project as part of a course.