An idea I got from the amazing 3D NES...

3D NES is a really impressive experimental emulator that turns your NES games into 3D! It's not perfected yet, but the beta is public and it shows us, it's real and it works. Well, the performance is not perfect and while I'm not sure how it generates the 3D world, but it somehow uses the pixel colors maybe. Games that's heavily using dithering runs a lot slower since it generates more complex geometry. I wonder how it'll be improved in overall, it couldn't open all my ROMs, probably not all the mappers are implemented yet.
One of the thing which made me surprised about this project is that it was made in Unity! The accessible beta is WebGL only, but since it's Unity based it means it'll be easy to port to other platforms. Personally I am familiar with Unity and I never ever dreamed about not even a working average NES emulator in Unity. But a 3D one? Just wow!



This project really made me think and gave an idea for a different approach of. Since it generates the meshes in real time it's kinda demanding and the code is obviously not optimized yet, the I'm sure the developer will improve it. It's just a beta so it's totally acceptable in this stage of development.

My idea is. What if we don't generate the meshes? What if we make them? In a 3D modelling software for example? So my way of thinking is:

NES stores things in its tiny little RAM like tiles, placement of tiles, placement of sprites. To be exact, it has a PPU (Picture Processing Unit). There are two pattern tables and the pattern tables store sprites and background tiles separately. Tiles in pattern table can be changed dynamically.
It's a bit more complicated than this, but I don't go in details. So assume that we can read the pattern table in real time. Well, emulators like FCEUX can visualise them in real time, so it's absolutely not an issue.

Other things is the nametable. Basically it has what we see on the screen. Because many games can scroll they have to be mirrored for smooth scrolling. So there is not only one of them at the same time. Also accessible within FCEUX with memory addresses and visual representation.

NES has a very limited palette choice of 56 individual colours, 3 colours per palettes and 4 palettes for sprites and backgrounds. That means, at most there can be 25 colors on the screen. Not very important for my idea actually. The palette indexes can be. For background, after the nametable there is an attribute table which contains which palette to use in a 2x2 area of background tiles. This and the nametable contains exactly 1 KiB of data (960+64 bytes).

So my idea is:
Run the game in a custom built emulator and instead of rendering the data from CHR-ROM / CHR-RAM, render something else! Render 3D models based on the data we read from nametable / pattern table / CHR-ROM / RAM.
My concept is not automatic and procedurally generated. Each games should have a package of 3D models that replaces (more like overrides) the data of the CHR-ROM. So a normal emulation would run but without rendering the game, we render the models based on all the data.

Take Castlevania for example. There are many wall tiles, floor tiles, and other things in that game. Each tiles should be represented by 3D models. There are those brownish floor blocks. Let's create something like that in 3D in 4 parts since it's actually 4 tiles or since there is no exception of having half or quarter of those floor tile, just create the full one with custom textures and look and depth and if we find that in our nametable, let's render that 3D model of that floor. Walls should work the same way, let's consider our floor model's depth and place the wall tile to the same z position so they would be connected and would give a 3D feel. There are those columns in the first stage. Create them, place them the same way. It should be closer to the camera than the wall.
Starcases are simple too, it needs some tricks though. Staircases have railing and if we create a railing model in 3D we can't place a wall behind it, since NES has one background layer. But we can model the same wall on behind the stair railing.
It may sound really bad to model all tiles in 3D since they're 8 by 8 tiles and there are many of them, but there are lots of patterns. There are no half floor block, there are not half columns. It's because the developers wanted to conserve ROM space so they probably created patterns and in the program itself they only need to refer to the pattern not to all the individual tile. Here is an example for better understanding from Metroid: Metroid leveldata explained. Also if you worked with Castlevania III level editor (ReVamp), you can see the predefine structures.

A few types of walls, big columns, and other things with clear patterns

So many times the asset creators for this kind of 3D NES emulator could "cheat" like the developers did and instead of creating all the tiles one by one, in many cases they could be combined into one.

You won't read about sprites too much here, but I am a little bit confused by them and needs to think about the possibilities. We need to render 3D models here too obviously. Sprites are also in CHR-ROM. All sprites are 8 by 8 pixels or 16 by 8.
Now my first headache here: how you do it (tiles feel more straightforward). If you can do it, you need to model all the frames one by one. Some animations share tiles. I can imagine this causing gaps and bad alignment. Also since it works like this the animations would never be smooth. If it was possible to interpolate somehow between sprite changes...

About the sounds. I don't see too much possibility to change them. I was thinking about a few things though. Let's stay at Castlevania. Yeah I know, I love Castlevania too much.
So what if we could use our own music? I can only think about a few crazy things here again. How can we get the music from that RAM? I have no idea! How should we know what music to play? Hmmm... since I said Castlevania, let's see what we have.

For example, we know which stage we are at! We may be able to look up the tile ID in the HUD where the STAGE 01 text is! 01 is at $D0 and $D1 positions in our pattern table. That may tell our custom program what music to play! But uh oh... there are more things here! What if we die... Well, this is kinda complicated for me. Well in case of Castlevania we can look up tile at X:7 Y:3 which is our last part of the lifebar. If it refers to tile $DF it would mean, we died!

Back to graphics for a little. Since it was 3D, we should improve the look. Lightsources can be important. But how could we know where to place the light sources within the 3D engine that runs the game with the background emulation? Castlevania has candles and other things that emits light. Again, we can turn to the RAM, looks where is that specific sprite or tile and just put a lightsource there.

All these things can be complicated and each game should need its own asset package and scripting to do the things right. It's a lot of work but think about it. With a method like this it was possible to remake almost all NES games (especially side scrollers and top-down games) with new graphics and maybe sound while the program would work exactly the same if our emulation is precise.

I'm not even sure if it was possible to do the way I am thinking, but 3DNES is a proof that it's possible to run emulator in Unity and do 3D stuffs with it.