Update: I got a lot of comments on my post. I am trying to answer them as they come in, so check back after you leave a comment.
Brendan Eich and Mike Shaver have posted an update on our progress on TraceMonkey. There has been a lot of buzz around Google’s new Chrome browsers and its V8 JavaScript VM. Some voices have claimed that V8 is several times faster than TraceMonkey. We did some head to head comparisons and these claims don’t match our observations.
We used Apple’s SunSpider benchmarks for our tests. Depending on the OS and machine configuration we are 1.18x to 1.28x faster than V8. Since V8 is only available for Windows, we didn’t perform any tests on MacOSX and Linux, both of which we support already. Our latest builds also work on ARM, by the way.
I am sure you can derive different results by tweaking the benchmarks or designing entirely new custom benchmarks alltogether, but since SunSpider has been used fairly intensively in the past two years to measure the evolution of JavaScript performance in Safari, Firefox, Opera, and IE, I think SunSpider is probably the most reliable cross-platform benchmarking tool at this point (which doesn’t say that its a particularly good one, its just the best we have right now.)
Talking about IE, our tests also indicate that we are about 15 times faster than IE 7, and about 5 times faster than IE 8 beta on the SunSpider aggregate scores.
If you want to give TraceMonkey a try, take a look at our nightly builds. You can enable the JIT in the about:config settings. The nightly builds are certainly not ready yet for wide-spread use, but we have improved stability significantly since our initial preview release. Firefox with TraceMonkey enabled is now my default browser, and I am writing this post with it.
[...] details at Brendan’s blog post. UPDATE: Andreas Gal and Mike Shaver also share comments on [...]
Pingback by Mozilla in Asia » Blog Archive » TraceMonkey vs. V8 — September 3, 2008 @ 3:37 am
You should append data from the five JS tests G. is delivering with Chrome. Matt Cuts is already throwing out blog links to charts created with the G. Chrome JS test set.
Comment by Markus Merz — September 3, 2008 @ 3:39 am
I made some tests on Dromaeo yesterday, and Chrome’s result is really impressive: 5.12x faster than Firefox 3.0.1 on the same virtual machine (WinXP on Parallels), still 3.37x faster than a nightly build with JIT enabled run on Leopard (so not inside a VM).
The post is in italian, but you can find all the links to the Dromaeo tests
http://www.pseudotecnico.org/blog/2008/09/02/veloce-e-veloce/
Comment by flod — September 3, 2008 @ 3:49 am
It does crash for me on a lot of sites (I’d say about 10% of sites using the nightlies). I’ve only enabled it in the Chrome now; although I do get odd results using some extensions. I am submitting my crash reports!
Comment by Jon Breen — September 3, 2008 @ 5:35 am
This benchmark doesnt reflect a multi tab / long term /real browsing environment doesn’t it?
Comment by IpNextGen — September 3, 2008 @ 7:15 am
What about http://dromaeo.com/ ? :)
Comment by zbraniecki — September 3, 2008 @ 7:50 am
What kind of speed improvement magnitude do you expect in a near / further future with TraceMonkey ?
How far TraceMonkey is different from V8 ?
Comment by David MARTIN — September 3, 2008 @ 8:54 am
With Sunspider, the latest Firefox nightly seems to be ~1.2-1.3x faster than Chrome on my machine. But if I try the benchmark on http://dromaeo.com/, Chrome is faster than Fx on every single test. With Fx I got 1624ms, while with Chrome 718ms. Why is there such a huge difference between the results of the two benchmarks?
Comment by Manny — September 3, 2008 @ 9:50 am
Well my bench on Windows Xp (P4 2.8GHz, 512MiB) were more like:
V8: 3317.0ms
TraceMonkey: 5827.0ms
Trace monkey did better in
bitops,
V8: 94.6ms TM: 78.2ms
access:nsieve,
V8: 27.4ms TM: 20.8ms
crypto:sha1,
V8: 27.8ms TM: 19.6ms
date:format-xparb,
V8: 642.4ms TM: 244.2ms
math,
V8: 209.0ms TM: 113.6ms
regex,
V8: 547.8ms TM: 303.0ms
string:base64,
V8: 101.0ms TM: 27.0ms
string:tagcloud:
V8: 243.0ms TM:236.8ms
TraceMonkey did noticeably worse in
access,
V8: 122.2ms TM: 484.4ms
controlflow,
V8: 4.0ms TM: 142.6ms
Comment by Matthew Swank — September 3, 2008 @ 9:58 am
I don’t have precise data at hand for google’s own benchmarks, but I have no doubt chrome does great on the benchmarks it was tuned for/developed with. We ran the SunSpider microbenchmarks yesterday and we beat chrome 4-5x on those. We didn’t publish those numbers because people are less familiar with the microbenchmarks then the actual SunSpider suite.
Different JITs with different compilation approaches deliver different performance results for different benchmarks. What I was trying to say with my blog post is that we are not “guaranteed many times slower than V8″, as someone suggested.
While Dromaeo contains very similar benchmarks, we don’t perform nearly as well on them as on the original SunSpider benchmarks. I have not deeply looked into this, but I will definitively do so over the next few days. Dromaeo looks more interactive than SunSpider. If it talks to the DOM during the benchmark, we currently don’t compile across such calls (we plan to for Beta2 though).
As for stability, I am sorry if Minefield crashes for you that much. Please post bugs for those failures. Feel free to CC me on the bugs directly, that way they go into my queue right away (bugzilla ID is gal at uci dot edu).
As for “how real are these benchmarks”, I think none of them really are. We definitively need better benchmarks that capture web applications, which is what people really want. Not tests of recursive function calls in different variations.
As for our future, I think it is bright. TraceMonkey exists for a little over 10 weeks now. For that we are running already quite fast for some code. We are working on tracing more code. Keep in mind that for some SunSpider tests we don’t speed up over interpretation at all because we don’t compile anything (most notably we don’t compile recursion at the moment). Once we have improved coverage for such cases, you will see pretty drastic improvements for cases we didn’t compile before (5-10x). For really computationally-heavy code (like the imagine manipulation demo schrep showed), which we already do compile, I expect that we can get at least another 2x speedup over time, probably more. But talk is cheap. Give us some time to speed it up, and I promise to blog about it :)
As for the numbers Matthew posted, we seem to beat V8 in almost every category but we come out 2000ms slower in the total? That looks weird. Maybe just a pasting error. But there is definitively some variance in the benchmark with different machines, and V8 and TM are very close in performance. I wouldn’t be surprised if in some configurations they beat us by a bit, just as we beat them by merely 20% or so in the configurations we tested with.
Comment by Andreas — September 3, 2008 @ 10:24 am
[...] I arrived late last night to the news that Google has entered the browser wars. First impressions for me – something needs to be done about the slowdown/freezing that js-heavy webapps such as gmail create on IE and Firefox. Having multiple flash based sites open causes similar non-responsive issues. Chrome’s js engine, known as V8, responds favourably in benchmarks - although Firefox minions are countering this. [...]
Pingback by Chromed at aleatory — September 3, 2008 @ 11:02 am
@Andreas I just posted the SunSpider suite benchmarks where TraceMonkey did better (with the exception of the very slow marks at the bottom) to show there were some things TM did well. There were plenty of slower, unlisted results for TM as well. I had weird problems copying the urls for the results, otherwise I would have just posted links to the two test runs.
Comment by Matthew Swank — September 3, 2008 @ 11:05 am
The V8 docs say they don’t have an interpreter; they compile straight to machine code at execution time. It’ll be interesting to see how that works out on real-world pages, and whether it limits the amount of low-level optimization they can do.
Comment by Jason — September 3, 2008 @ 12:10 pm
Andreas, is there an agreed-upon “threshold” whereby TM will be enabled by default in the nightlies?
Comment by RyanVM — September 3, 2008 @ 12:24 pm
[...] done anything with it as a user where I really noticed a difference. TraceMonkey vs. V8 has been covered elsewhere, so I’m not going to bother with that. Plus all these VMs are pretty early, [...]
Pingback by Chrome: First Look :: David Mandelin’s blog — September 3, 2008 @ 7:11 pm
[...] multe detalii si un grafic cu rezultatele comparatiei gasiti aici. Etichete: Firefox, Google Chrome, Mozilla, TraceMonkey, [...]
Pingback by Weasel, the Geek - TraceMonkey vs. V8 — September 4, 2008 @ 4:11 am
It’s a great post, thanks…
But the there is a flaw in the link to the nightly builds. You are linking to the mozilla and not the firefox builds.
If anyone would like to try out the nightly firefox build the right URL is
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/
Have a nice day :)
Comment by Brunsgaard — September 4, 2008 @ 5:40 am
Fixed the link, thanks Brunsgaard.
Comment by Andreas — September 4, 2008 @ 7:29 am
I’m also curious about #14 :)
Comment by JimmyR — September 4, 2008 @ 3:03 pm
#14: I think Mike Shaver is the better person to ask this, but I think the main show stopper at the moment is stability, and thats what we are focusing on. The goal is certainly to be as stable for standard web use as with jit off, and thats where we are trying to get to for Beta1. If you want to help, grab a nightly and file bugs :)
Comment by Andreas — September 4, 2008 @ 3:18 pm
[...] 8b2，其實效能並不遜Chrome太多，甚至FireFox 3.1跑JS的速度比Chrome還要快（這裡也有一份TraceMonkey vs [...]
Pingback by Chrome讓我們期待更多了 « 小白的窩 — September 5, 2008 @ 5:19 am
[...] Chrome? TraceMonkey vs V8 Andreas Gal Mozilla Fights Back With New Firefox Benchmarks __________________ CodeCall Blog | CodeCall Wiki [...]
Pingback by Chrome? - Page 2 - CodeCall Programming Forum — September 5, 2008 @ 9:20 am
Is TraceMonkey optimize only for SunSpider?
As people have mentioned, on dromaeo Chrome still beats Tracemonkey. In google V8 test
http://code.google.com/apis/v8/run.html
it beats Tracemonkey as well. Am I missing something here?
Comment by AnonymousCoward — September 5, 2008 @ 11:06 am
I think the entire discussion over who has the best VM is nonsense. Both VMs are open source, and both VMs are new. Over time, we’ll probably see tracing making it’s way into V8, as we’ll probably see general compilation making it’s way into TraceMonkey. Both teams have done an amazing job, and I say congratulations to both.
Comment by Josiah Carlson — September 5, 2008 @ 5:09 pm
[...] A friend pointed me to a benchmark comparing V8 to TraceMonkey, Mozilla/Firefox’s upcoming Javascript engine. Tracemonkey [...]
Pingback by Google Chrome « NPFet.com — September 7, 2008 @ 1:04 am
The new javascript engine seems to choke on scripts from redfin.com
It works fine if I disable the JIT.
Comment by w0wy — September 8, 2008 @ 12:58 pm
Sachliches zum Chrome…
Wie ich schon beschrieben habe gibt es ja einen guten Grund für google einen eigenen Browser zu verteilen:Die optimale Unterstützung ihrer JS-WebapplikationenDas ist ja auch verständlich. Schließlich ist das einer ihrer Einnahmezweige. Das und di…
Trackback by Wurstsalat — September 9, 2008 @ 12:35 am
What’s the status about the powerpc port?
Comment by lu_zero — September 12, 2008 @ 5:05 am
ulu의 생각…
SpiderMonkey의 JIT인 TraceMonkey 때문에 빨라졌다 하는건데 뭐 이런 벤치마크 결과 도 있구나….
Trackback by ulu's me2DAY — September 16, 2008 @ 10:30 pm
Andreas, Any chance you can write a new blog post on what’s happening in TraceMonkey land in the last few weeks in particular how is tracing for recursion coming along
Comment by Yusuf Goolamabbas — September 17, 2008 @ 3:03 pm
[...] de ambos se demostró que los resultados de uno y de otro son prácticamente similares y que mejoran sustancialmente lo que actualmente procesa el Javascript en los navegadores [...]
Pingback by SquirrelFish Extreme entre en el juego de los motores JS | aNieto2K — September 19, 2008 @ 9:56 am
[...] Mozilla may come up with more advanced features in its upcoming version of Firefox. Already, it is said that Javascript VM to be used in Firefox 3.1, TraceMonkey, is much faster than V8, that used by [...]
Pingback by Emerging browser war: Google Chrome vs. Mozilla Firefox « Sriniani’s Weblog — September 20, 2008 @ 10:50 am
I don’t think it really matters whether TraceMonkey/V8/SquirrelFishExtreme are a bit faster at certain (synthetic) benchmarks against each other. What matters, and that’s good, is that we will have really, really fast JavaScript in most major browsers in the future. It’s good competition.
Of course it’s especially nice for Firefox since parts of the browser and extensions are written in JS.
Comment by test — September 20, 2008 @ 11:43 am
[...] they keep fighting it up. Google Chrome got a good PR head-start, but, Mozilla Firefox 3.1 is back in the race with [...]
Pingback by Mozilla Firefox and Google Chrome Harmony | Bits And Buzz — September 23, 2008 @ 7:21 am
[...] out there. However, Mozilla is working on a completely new Javascript engine, TraceMonkey, which at first sight will put a hell of a fight with Chrome’s V8. Who benefits from this? Us, users, of course. [...]
Pingback by A personal view at Firefox vs. Chrome | Tiago's Tech Blog — October 11, 2008 @ 2:57 am