Question 2: Why does Doom 3 involve 33,717 lines of bash? That's a lot of bash. I realise it's used heavily for build scripts, but thirty three thousand lines seems excessive.
If your habit is what I think it is, I have the opposite habit - I never use sudo until it tells me I've tried to do something I'm not allowed to do - even if I know it's not going to let me. Then I use "sudo !!"
I wonder why the depth fail shadow rendering algorithm had been explicitly excluded from the open source release. It's been around for years now, after all.
Check out draw_common.cpp [2] - the patent work-around is contained in the RB_T_Shadow method:
// patent-free work around
if ( !external ) {
// "preload" the stencil buffer with the number of volumes
// that get clipped by the near or far clip plane
qglStencilOp( GL_KEEP, tr.stencilDecr, tr.stencilDecr );
GL_Cull( CT_FRONT_SIDED );
RB_DrawShadowElementsWithCounters( tri, numIndexes );
qglStencilOp( GL_KEEP, tr.stencilIncr, tr.stencilIncr );
GL_Cull( CT_BACK_SIDED );
RB_DrawShadowElementsWithCounters( tri, numIndexes );
}
I was under the impression that most large projects end up having to write a custom memory allocator, but after a few random probes into the source I didn't see anything that stood out. Did I just miss it, or did they use the built in one?
the xbox version was done by another activision studio called vicarious visions. no xbox or playstation code can be released due to developer license agreements
This is a replacement for the compiler heap code (i.e. "C" malloc() and
free() calls). On average 2.5-3.0 times faster than MSVC malloc()/free().
Worst case performance is 1.65 times faster and best case > 70 times.
To quote the review (which I've read more than the source):
The Zone Memory Allocator(common.c: Z_Malloc, Z_Free,
Z_TagMalloc , Z_FreeTags) is still here in Quake2 but
it is pretty much useless:
Tagging is never used and allocation/deallocation is
build on top of malloc and free (I have no idea why id
software decided to trust the C Standard Library at this
point).
The overflow detector (using the Z_MAGIC contant) is
never used either.
So, unless I totally misunderstand things, it is half C stdlib, half Hunk allocator?
That makes very little sense to do - every single project? Don't you think someone would have released a library to implement the "new and improved allocator" if everyone needs to use it?
Most game projects do end up having to write a custom allocator, due to very specific memory budgets, deterministic malloc/free perf requirements, and issues with fragmentation.
I've worked with several projects where a TCMalloc based allocator (threaded) was used to great success. A simple switch to DLMalloc (non-threaded) can also provide a significant performance boost.
Moderate usage of STL can do nasty, nasty things to memory performance. In one case a game I worked on ran at 5-10fps and a simple, fast change to DLMalloc instantly took perf to 60fps.
We used TCMalloc at Opera Community[0], and they still might be using it there for all I know ... it greatly improved performance on our MySQL servers!
No, the point of a custom allocator is writing one that works with your specific requirements. A general allocator will never be more efficient than a custom one designed for the specific task. It becomes just an issue of is the cost worth what the optimization buys you, and games are an area where you need every drop of speed you can get.
IIRC the source for most of the physics engine has been available for years with the SDK. I think it only including rigid body dynamics and the constraint solver though, no collision code, so it's nice to have that part of it.
I'm not sure where I picked up the info, and I haven't looked at the code to confirm it, but I believe the physics engine implementation is tied tightly to both the collision detection system and the entity update, both of which are quite specific to Doom (in particular, each entity steps physics independently, rather than a whole simulation advancing at once as is common in 'pluggable' physics engines like ODE, Havok etc).
Also, and this is subjective I suppose, I don't remember Doom 3 doing anything that 'felt' or looked better than even what the contemporaneous engines did (such as the version of Havok used in Half-Life 2).
So I guess, there probably aren't valuable additions, but I imagine there is much to tinker with.
From looking at the game code, I see odd things like rolling barrels animating their rotation as though the physics engine itself doesn't handle rolling contact.
My favorite technique is to run the code, place breakpoints in interesting locations and manually type out psuedocode descriptions of each function up the callstacks. That's the fastest way I know to get an understanding of the structure of a new code base.
Barring running the code... I prefer to start with a quick scan through the foundation code. What does the math library look like? How about the file and network I/O? Next, locate the top-level main loop, then the top-level Update Everything/Render Everything functions. At that point, you'll understand the general theme of the code and you'll have spotted lots of interesting avenues to explore.
I've heard the best way to tackle the Quake codebase was to pick a 'component', such as the renderer, and start to understand how that works. Work your way out, one component at a time. Eventually the bigger picture comes together. The same would probably work for this, though unless you're well versed in game and graphics programming nomenclature some of it will go over your head, which is expected - when you see something like that, think of it as an opportunity to learn a new concept and look it up.
It's more a legal thing which you will see with any commercially released codebase. The iD hackers have to jump through innumerable hoops with their publishers' legal department to get this code out to us at all, so it kinda sounds ungrateful to moan about the licenses.
Also, different parts of the code are covered by many different licenses and the copyright on them belongs to many different people, so it is important to be very explicit about everything.
When it's an OSS project owned and operated by a small group of people, there's little motivation to worry about this stuff. Also when you choose a license like the MIT one which basically says "I really don't give a fuck what you do unless you sue me," you have a lot less incentive to plaster it over everything, because you're one step away from being in the public domain anyway.
When you're releasing the doom3 code, things are very different.
What I meant (probably poorly phrased) is that for readability for personal study (like gcv was asking about), I think it's worth stripping huge license preambles from source files. There's obviously good legal reason for them being present to start with.
Only took 8 minutes and 22 seconds to compile this, I was quite surprised. Going to have some fun over the next few days that's for sure! Thanks Carmack, heh.
Can you give me some advice how to add the gamedata to the code?I got compile errors for I had no doom3 on my computer(windows 7),I'm downloading it now.
Open doom.sln
Go to View -> Property Manager
Expand Debug | Win32
Right click "Microsoft.Cpp.Win32.user" and choose Properties
Select VC++ Directories
You need to add the following directories
C:\Program Files (x86)\Microsoft DirectX SDK June 2010\Include
C:\Program Files (x86)\Microsoft DirectX SDK June 2010\Lib\x86
Do the same for Release | Win32.
Doom3 should now compile. Note that 2 projects might fail and 1 could be skipped.
You should find Doom3.exe under doom3.gpl\build\Win32\Release
From your Doom3 installation copy the "base" folder, if you have the expansion copy that too.
hi I got his error, on ubuntu 11.04. I moved the base folder to the neo folder but it still didn't work. Is it because it's missing the game data?
DOOM 1.3.1.1304-debug linux-x86 Nov 24 2011 00:27:04
found interface lo - loopback
found interface eth0 - 10.0.0.19/255.255.255.0
------ Initializing File System ------
Current search path:
/home/xx/.doom3/base
/home/xx/doom3.gpl/neo/base
game DLL: 0x0 in pak: 0x0
Addon pk4s:
file system initialized.
--------------------------------------
----- Initializing Decls -----
------------------------------
------- Initializing renderSystem --------
idRenderSystem::Shutdown()
Sys_Error: _default material not found
A bit off-topic, but I'm curious: why do you think that's impressive? I thought it was pretty commonly accepted these days that readable code saves more time than it costs over the life of a project...?
Game code specifically is usually the exception to this rule, because it tends to be written to very strict deadlines, ad-hoc requirements, short project lifetime (no need to maintain or support it for years after), and not much expectation of re-use.
This does not all hold for Doom3 as it was also an engine sold to other companies for other games.
A lot of things are generally accepted as "good ideas", but how often are they practiced? Most software management would throw their own mother under a bus if it got the product out the door on anything close to schedule or budget.
Which is to say, code cleanliness is very much not the norm anywhere. And if anything the game industry seems to be a good 10-20 years behind the rest of the software industry in terms of hewing to good practices (clean code, automated tests, etc.)
This seems to be pretty hard wired to build 32 bit binaries only, I'm having a little trouble build on Ubuntu w/ amd64. First you need to install multilibs and 32 bit gcc/g++. But I got stuck when I'd need some other libs in 32 bit versions (zlib, curl and probably openal and others). Ubuntu doesn't seem to ship with 32 bit versions of some libraries as some other distros do (like Arch, for example). Or am I wrong, what packages and what repositories I need to get e.g. zlib 32 bit binaries?
Has anyone had success building this on a 64 bit machine?
Is my only option to set up a virtual machine with a 32 bit distro?
I probably would have figured it out by myself, I solve this kinds of things for living. But I was trying to quickly build this before going to work and ran out of time.
Thank you for pointing this out, once I get the time to sit down on this for a while, I'll have to take a look at all the action on Doom3 in GitHub. Things have really gone crazy, there's literally hundreds of commits on Doom3 in the first 48 hours after the open source release. It's great that they put it on GitHub and not just on an ftp server. This way we can see all the action going on with it.
make sure you install g++-multilib so it pulls in the correct version. lib32z1-dev will work for zlib (need to create a symlink since it doesn't look in /usr/lib32). I don't see anything for 32-bit curl though :(
I did install g++-multilib and a few other packages in order to get the build forward. My build finally failed when trying to link the "doom3" executable due to the lack of 32 bit libs for zlib and curl (maybe others too).
I tried to use a very unofficial Ubuntu utility called getlibs (http://frozenfox.freehostia.com/cappy) but that didn't really help, the library search path was wrong and the correct libs were not found.
I can't seem to find references to this anymore, but I seem to remember that the doom3 source code was stolen and released in like '03 or '04. So if that's true then my response to this article is, "Again?".
Oh yea. Anyone still has the old code? Would be interesting to make a diff between the alpha and this one. Not to see specific line-by-line fixes but more just to get a general impression of how feature complete it was back then.
DOOM 1.3.1.1304-debug linux-x86 Nov 24 2011 00:27:04
found interface lo - loopback
found interface eth0 - 10.0.0.19/255.255.255.0
------ Initializing File System ------
Current search path:
/home/xx/.doom3/base
/home/xx/doom3.gpl/neo/base
game DLL: 0x0 in pak: 0x0
Addon pk4s:
file system initialized.
--------------------------------------
----- Initializing Decls -----
------------------------------
------- Initializing renderSystem --------
idRenderSystem::Shutdown()
Sys_Error: _default material not found
Hmm... Looks like this is an Xcode 3 project file. I can still get Xcode 3 from Apple's developer downloads, but it looks like it only includes the older iOS SDKs, not the OS X 10.5 SDK required to build this.
In my experience, it is usually an environmental assumption that is mismanaged rather than the "source code" itself being broken.
Whether that is failure to check for a dependency, failure to document a dependency or failure to clearly specify which specific version of a dependency is required. And the pain of autotools is a completely separate chapter in describing how "my" environment can differ from "theirs."
All of that just goes to show that state management is _the_ hardest part of software.
True. Generally the failure of the packager is that they never do a clean install and build. And yes that is tedious but it finds so many issues, and isn't so bad with VMs.
I am getting the error, "scons: * [build/debug/core/sys/scons/doom] Source `/usr/lib/libz.a' not found, needed by target `build/debug/core/sys/scons/doom'.
scons: building terminated because of errors." Anyone knows what I should install?
Anyone else having trouble trying to run it? got the assets in place, get to the menu fine, hit new game then get a black screen with an empty box in the middle. Suspect its the copy protection?
Nope, it's only the engine. If it's anything like the other iD software code releases, you will be able to use the content from the game disc with the engine built from source, or you will be able to extract the content from the demo if you dont have the game.
Installed Doom3 using the CDs and updated to the latest release. Then opened the sln file in VS 2010 and changed the command line args for debugging to point to the data of the game installed from the cd (instead of steam).
Now what is the matter with default.cfg and what about "Unknown command 'vid_restart'"? I am not familiar with the structure of idgames.
------ Initializing File System ------
Current search path:
C:\Program Files (x86)\DOOM 3\base/base
game DLL: 0x0 in pak: 0x0
Addon pk4s:
file system initialized.
--------------------------------------
Unknown command 'vid_restart'
idRenderSystem::Shutdown()
Shutting down OpenGL subsystem
...shutting down QGL
Couldn't load default.cfg
Any gambling types fancy a little wager? Which will come first, a stable JVM port of doom 3, or a stable build of Textmate 2? The only problem is, I really can't think which side I'd come down on.
NOTE: For those primed to accuse me of hating: this is as much a compliment to certain hackers and their fixated attitude towards porting iD code dumps, as it is a snide reference to textmate2's status as vaporware.
