leonardo ([info]leonardo_m) wrote,
@ 2009-07-04 23:57:00
Previous Entry  Add to memories!  Tell a Friend  Next Entry
Entry tags:benchmark, c, d language, dmd, ldc, python

Richards benchmark
This old post of mine is about virtual methods and devirtualizations:
http://leonardo-m.livejournal.com/76547.html

I have performed more benchmarks about virtual methods and its costs with LDC. I have used a little well known benchmark, the Richards one. I have used the code from here (on the Web Archive because it seems to be not online anymore):
http://web.archive.org/web/20060715074131/lissett.port5.com/ben/bench1.htm
http://web.archive.org/web/20060715074131/http://lissett.port5.com/ben/bench3.htm

All the code of the following benchmarks:
http://www.fantascienza.net/leonardo/js/richards.zip

Timing results:

Windows, n = 10_000_000:
  C:       1.26
  D ~C:    2.01
  Java:    2.04 (final classes, -server)
  D2 ~C#:  2.36 (final classes)
  D3 ~C#:  2.36 (final classes, no getters/setters)  
  Java:    2.73 (final classes)
  D4 ~C#:  3.07 (no getters/setters)  
  C#:      3.98
  D1 ~C#:  4.23
  
Windows, n = 100_000_000:
  C:      12.16
  Java:   18.73 (final classes, -server)
  D ~C:   18.86
  D2 ~C#: 23.11 (final classes)
  D3 ~C#: 23.12 (final classes, no getters/setters)  
  Java:   25.40 (final classes)
  D4 ~C#: 30.16 (no getters/setters)    
  C#:     38.39
  D1 ~C#: 41.64


Pubuntu, n = 10_000_000:
  D ~C:    1.35
  C:       1.39
  D2 ~C#:  1.98 (final classes)  
  D3 ~C#:  2.00 (final classes, no getters/setters) 
  Java:    2.73 (final classes)  
  D4 ~C#:  2.94 (no getters/setters)      
  C#:      -  
  D1 ~C#:  4.03
  
Pubuntu, n = 100_000_000:
  D ~C:   13.24
  C:      13.77
  D2 ~C#: 19.64 (final classes)
  D3 ~C#: 19.92 (final classes, no getters/setters) 
  Java:   25.16 (final classes)    
  D4 ~C#: 29.16 (no getters/setters)       
  C#:      -
  D1 ~C#:  40.17

Key:
  D ~C# means D code that comes from the C# version.
  D ~C means D code that comes and looks from the C version.

Note that the classes in C# code aren't final. As usual Java shows very good performance.


On WindowsXp:
DMD Digital Mars D Compiler v1.042
gcc version 4.3.3-dw2-tdm-1 (GCC)

dmd used with:
dmd -O -release -inline

ldc used with:
ldc -O5 -release -inline

gcc used with:
gcc -Wall -O3 -s -fomit-frame-pointer -msse3 -march=core2


On Pubuntu:
gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu4)
ldc based on DMD v1.045 and llvm 2.6svn (Thu Jul 2 23:07:48 2009)

ldc used with:
ldc -O5 -release -inline

gcc used with:
gcc -Wall -O3 -s -fomit-frame-pointer -msse3 -march=native

(Read 8 comments) - (Post a new comment)

Re: C# Linux timings
[info]leonardo_m
2009-07-15 10:07 pm UTC (link)
Nice Benchmark, but where are the Linux C# Benchmark results using mono (should be slower since mono is not as optimized as .net yet but it would be also very interesting) ?

C# Mono is usually slower, its GC is much more primitive, etc. At the moment I am not much interested in trying mono on Linux.



and then the c# version should be as plattform independent as the other versions.

I see, thank you for letting me know.
You can also totally remove the timing-related code, as I've done to some other version.


>Have you also tried to make the classes sealed in the c# version or convert them to structs (since you made the classes final in the java version ?) and benchmarked then again ?<

I have now set all classes in the C# code as sealed, but the performance is exactly the same. I don't know why.
Turning those classes into structs may be a bit too much work now. (If you offer me a C# version that uses structs, I can time it).

(Reply to this) (Parent)(Thread)
Re: C# Linux timings
(Anonymous)
2009-07-17 01:17 am UTC (link)
"C# Mono is usually slower"

That's right (and not only usually but nearly always). But they are getting faster and faster with each version. Since they changed the JIT from a tree-based internal representation to the new linear IR things got faster and still will ( http://www.mono-project.com/Linear_IL ).


"... its GC is much more primitive, etc."

Sure, because at the moment they still use the same conservative Boehm-Demers-Wiser Conservative Garbage Collector like the D programming language which is far from optimal for this kind of stuff (like it is for D). But they are since a long time working on a new garbage collector ( http://mono-project.com/Compacting_GC ) to fix this (i would like to see this for the D programming language too).



"At the moment I am not much interested in trying mono on Linux."

Oh that's a really a pity. It would lessen the C# windows-only view (even if a lot of people don't like this). Then i have to do it myself again :-(




"I have now set all classes in the C# code as sealed, but the performance is exactly the same. I don't know why."

I don't know why too. But at least it did not change the amount of inlining which the jit does (methods in sealed classes have a greater chance of getting inlined since they can't be virtual but on the other hand it is the default behaviour for methods in C# anyway, since methods are final by default (which is really better) ).

But you can always use reflector ( http://reflector.red-gate.com/Download.aspx ) and look at the generated bytecode / decompiled c# code (with reflector decompiled code so you can see what the compiler has made out of your code and how / if he has optimized things). It's really interesting to see what the compiler does to your code and how things are translated.



"Turning those classes into structs may be a bit too much work now."


No it isn't (at least if you do it the primitive way). You just have to change the "class" keyword to "struct". That is basically all you have to do (no other changes - structs are instantiated with new like objects, so again no changes here). Now the structs are generated on the stack (which creates no garbage and let's the garbage collector alone which is really nice and can be a lot faster) and not on the heap anymore and they are now passed by value not by reference which can be slower. But to change this you just have to pass them by reference via the ref keyword (function parameters have to be declared as ref too)). Of course to do it optimal you really would have to think about to which classes this behaviour would make sense.

But for a quick and dirty try out by just changing the classes to structs just to see if things change it should be done quickly ...

(Reply to this) (Parent)(Thread)
Re: C# Linux timings
[info]leonardo_m
2009-07-17 09:54 pm UTC (link)
Since they changed the JIT from a tree-based internal representation to the new linear IR things got faster and still will

See also:
http://tirania.org/blog/archive/2009/Jul-16.html


Sure, because at the moment they still use the same conservative Boehm-Demers-Wiser Conservative Garbage Collector like the D programming language

I think D is currently using a GC a bit better than then Bohem one.


No it isn't (at least if you do it the primitive way). You just have to change the "class" keyword to "struct". That is basically all you have to do (no other changes - structs are instantiated with new like objects, so again no changes here).

I have tried and I have failed. I don't know C# yet.

(Reply to this) (Parent)(Thread)
Re: C# Linux timings
(Anonymous)
2009-07-18 12:27 am UTC (link)
"See also:
http://tirania.org/blog/archive/2009/Jul-16.html "

Oh. This really looks nice. Thank you very much for the information.



"I think D is currently using a GC a bit better than then Bohem one."

I don't know if they have changed it, but the last time i looked at it i am quite sure that the original D with the phobos runtime uses boehm. Tango is afaik using another gc (which is slightly better but sadly not much) but i could be totally wrong here. But anyway, i think D would really benefit from a better tailored GC.



"I have tried and I have failed. I don't know C# yet."

Sorry. It's not your fault. Since i have given you the nice pointy-headed super hint how simple it is to use structs in your benchmark without taking a detailed look at your code, i should better admit, that i too, don't seem to know C# yet :-) So again sorry.

Maybe i will take a look at your C# Code if i have time and try to fail too :-)
And if i should really get something to work with structs, i'll let you know.

(Reply to this) (Parent)

(Read 8 comments) - (Post a new comment)

Create an Account
Forgot your login or password?
Login w/ OpenID
English • Español • Deutsch • Русский…