usb support

[redguy]

I changed my mind and decided to implement the context engine to save snes *RAM contents to sd2snes's RAM.  It wasn't as bad as previously thought and there's now a working prototype.  It saves off WRAM, VRAM, APURAM, OAM, CGRAM, and many PPU and CPU registers.  I made a basic c# memory viewer, but realized the viewers that came with bsnes were far better and ended up hacking it to poll the USB contents without a loaded cartridge.  It's pretty cool to see the backgrounds load up in the tilemap viewer and the contents of VRAM in (somewhat) real-time.  Here's an example:

https://www.youtube.com/watch?v=iC-HPhJf_Og

I'd like to make a little DMA engine to copy contents of the RAM around to speed up saving in savestates.  It would also be cool to have a way to set HW breakpoints, but that's going to take a lot more thought.

Another thing I've been trying to do is load a game and immediately load a savestate.  The basics of this have been working for a while, but without having a way to restore the APU it would have no sound and usually freeze after a few seconds.  The context engine stuff makes a basic attempt at saving all the state pre-execution that is written to the APU.  Now on game init this APU state can be restored and the sound works, although, it needs some more work to fix the music.  Here's an example where it transitions from contra to castlevania.  I have to manually trigger the load of castlevania and the associated state right now which is not ideal.  The music works only because I picked a savestate right before the engine loads the music.

https://www.youtube.com/watch?v=ZEzl5HVppvU

At this point there are way too many hacked up c# and c++ utilities which are using my messy USB interface.  A driver that provides a simpler (socket based?) interface to the sd2snes over USB is the next step.  That way I don't have to fix the USB related bugs in 6 different places and it would also allow multiple applications to use the USB port at the same time.

[iwasaperson]

I changed my mind and decided to implement the context engine to save snes *RAM contents to sd2snes's RAM.  It wasn't as bad as previously thought and there's now a working prototype.  It saves off WRAM, VRAM, APURAM, OAM, CGRAM, and many PPU and CPU registers.  I made a basic c# memory viewer, but realized the viewers that came with bsnes were far better and ended up hacking it to poll the USB contents without a loaded cartridge.  It's pretty cool to see the backgrounds load up in the tilemap viewer and the contents of VRAM in (somewhat) real-time.  Here's an example:

https://www.youtube.com/watch?v=iC-HPhJf_Og

I'd like to make a little DMA engine to copy contents of the RAM around to speed up saving in savestates.  It would also be cool to have a way to set HW breakpoints, but that's going to take a lot more thought.

Another thing I've been trying to do is load a game and immediately load a savestate.  The basics of this have been working for a while, but without having a way to restore the APU it would have no sound and usually freeze after a few seconds.  The context engine stuff makes a basic attempt at saving all the state pre-execution that is written to the APU.  Now on game init this APU state can be restored and the sound works, although, it needs some more work to fix the music.  Here's an example where it transitions from contra to castlevania.  I have to manually trigger the load of castlevania and the associated state right now which is not ideal.  The music works only because I picked a savestate right before the engine loads the music.

https://www.youtube.com/watch?v=ZEzl5HVppvU

That looks sweet! Didn't even think something like that would even be possible. Would be insanely helpful to homebrewers out there who want to test on real hardware.

At this point there are way too many hacked up c# and c++ utilities which are using my messy USB interface.  A driver that provides a simpler (socket based?) interface to the sd2snes over USB is the next step.  That way I don't have to fix the USB related bugs in 6 different places and it would also allow multiple applications to use the USB port at the same time.

I mean... if you're redoing the USB interface anyway, would it be reasonable to request Linux support?

[saturnu]

i wrote a linux program for just loading and starting games with the new usb protocol.
but actually this is pretty boring, the savestate dumping and restore feature is something that could be fun to play with.
i could imagine to program something like cheatengine to find offsets to alter health or items in games. ^^

a special driver is no big deal under linux but for windows the driver-signing is expensive and the dev mode for unsinged drivers is inconvenient.
that was the main reason i started with the std usb serial port, because all you need is an inf file. :>
i did some experiments with libusb but meh...

maybe just a little library between a program and the signed microsoft serial driver would do the trick, too.

[Enmet]

I mean... if you're redoing the USB interface anyway, would it be reasonable to request Linux support?

+1 for Linux support.

Also minor note, while I got usb2snes to run, the Zelda item tracker did not, so I guess the work-around wasn't added for it.

Would it be possible to do a soft reset through this interface?

[redguy]

https://github.com/RedGuyyyy/sd2snes/blob/master/rel/usb2snes_v0.5.zip

Here's a new version, but it's mostly for people who want to check out the (simple, non-bsnes-plus) RAM viewer and associated contents.  Save states are going to be slower for the time being because I'm also copying APURAM which is inefficient.  I need to add the hardware DMA engine.  I also worked around some races that I found when compiling everything under windows10.

By driver I was really just thinking of a system tray app in user space and not an actual driver.  I shouldn't have called it that.  The USB stuff has too many races and stalls that I need to work through to make all the apps deal with that code.  It would be better to have one app that deals with those problems and interleaves commands onto the sd2snes from other apps.

I'm not sure why the ZeldaHUD isn't working for you.  I didn't update it to use the latest USB code, but it still works for me.  Eventually I'm going to remove it as it was just for testing.  Yes, it's possible to add a soft reset button.  For now you can just double click/boot the file again to reload it.

EDIT: Actually, I think I forgot to include the fixed HUD in the last release.  I'll do that next time around.  I fixed it a while ago and just didn't include it.

[redguy]

https://github.com/RedGuyyyy/sd2snes/blob/master/rel/usb2snes_v0.6.zip

usb2snes v0.6

- Fixed several timeouts.
- Fixed some graphics corruption/lockups in save state.
- Added hardware RAM DMA engine with spin loop support.
- Minor cleanup in the CTX engine.
- Updated ZeldaHUD so it hopefully doesn't crash on startup for some people.

The save states should be fast again.  The DMA engine is used to copy the current state to a location that stores the save state.

<Technical Stuff>

The DMA engine currently supports a 24b source address, destination address and length.  It has 3 opcodes: copy, reset, and set.  It works similar to MVN (memmove) and will handle overlap assuming it is bug free.  You can put the snes into a BRA $FE spin loop as part of its configuration.  It feeds the instructions directly in order to leave RAM bus cycles for the DMA operation.

[saturnu]

oh cool,
currently i'm writing a c pendant to the c# app, just for fun. ^^
it can already dump and write the savestates, list files..., but it's still lacking the ips patch feature.
maybe i can finish it tonight...

[redguy]

Thanks for working on it.  Now I have to be careful not to change/break the protocol.  I have to figure out a way to add support for a smaller packet size in case someone wants to poll on a small set of data.  Or maybe a scatter-gather style operation since there are probably a lot of random bytes all over memory that certain tools would want.

USB full-speed bandwidth is a little low for some of things I'd like to try.  For example, it would be nice to have a data stream operation where the app can open up a connection and the firmware keeps streaming data from some hardware queue.  The RAM bandwidth may also be a limiting factor.  I have to check if it supports a burst mode.

[saturnu]

oh, please change the protocol if you want, i don't think i'm going to release it anyway, until it's development is stalled a bit more.
i just wanted to play around a little bit with the savestate feature.
yesterday i successfully dumped, compared, edited and reuploaded a savestate under linux, with the usb_v0.6 firmware. ^^

the thing is, searching for cheats could be easily done with bsnes, too.
but that's something, that's often the case.

maybe features that are useful for developers/translators, who need to test on real hardware are the way to go.
- savestates are great for translators, who need to test certain dialogs, this is a big step anyway - even if the music is stopping.

[redguy]

https://www.youtube.com/watch?v=nhNm-3XOrNc
(rgb+ossc output on the left, snes9x/usb output on the right)

I'm trying to implement a digital out from the sd2snes to an emulator (uses the software PPU) over USB.  It still needs a lot of bug fixes, a HDMA model, and some USB protocol optimizations.  The missing HDMA code is the cause of most of the background, item display, and special effect bugs.  It's pretty cool to have all the snes9x scaling and filter options available.

It probably won't work in all games and may not even work at all in the end, but it's a fun project.

[TrueJournals]

I love the work that's been going on here.  The added USB support is incredible and seems like a really awesome utility to add all kinds of supplementary functions to SNES games.

I've been wanting another little side project, so I've also started playing around with extra utilities on top of the USB support.  I've created a little Python HTTP server that provides an API to interact with RAM and do IPS patching (to allow your save state patch to be uploaded ).  I've also been working on a LttP item tracker (because why not) that can read everything in real-time thanks to the WRAM read support in your latest version.  As saturnu said, feel free to change the interface however you want -- I don't mind adapting my code!

As I was working on this, I came to a point where I wanted to try to write WRAM, not just read it.  Based on your usb2snesviewer memory viewer utility code, I got the impression that this should be possible -- but I just can't make it work for the life of me.  I tried building and uploading the usb_test and usb_write patches... but that didn't seem to do anything.

Am I missing something, or are writes to WRAM not possible?

[redguy]

Glad to hear you are finding it useful.

It is possible to write data into WRAM, VRAM, and the other on-snes data stores, but it currently requires patching the NMI hook.  There are two patches:
- usb_test : This is a work in progress that is meant to be a shared "USB"-specific memory address space that will be kept mostly coherent through broadcast operations.  It has an inbound and outbound queue where you can send and receive operations (compare and swap, xor, increment, etc) that operate on a small region of memory.  The operation and data is currently defined to be 4B.
- usb_write : This is a more general purpose patch that re-uses the inbound queue code to support full address space byte writes packed into the same 4B queue element.

I haven't mentioned them before since they are just hacks and aren't well tested.  You'll want the usb_write patch.  Hopefully it still works.  I just checked in an assembled version of it:

https://github.com/RedGuyyyy/sd2snes/blob/master/patch/usb_write/usb_write_v0.ips

Eventually I should merge these patches with the existing sd2snes hook code.  For things like USB address space support, save states, and anything that the hardware can detect it would be nice to have a conditional NMI hook that only runs when the trigger is hit.  But that's going to take some time to figure out and implement.

[TrueJournals]

Well... I don't know what I was doing wrong before, but I've got it working now!  Thanks for the help!

[redguy]

https://github.com/RedGuyyyy/sd2snes/blob/master/rel/usb2snes_v0.65.zip

usb2snes 0.65

Fixed save state to not enable PPU until after PPU alignment.  Fixes some momentary graphics glitches on load.
Fixed many timing and questionable SNES bus loading statements in the FPGA.
Added support for 64B data block sizes and no responses based on flag.
Added support for 64B vector VGET/VPUT commands.
Removed MSU support from base firmware.
Removed ZeldaHUD.
Changed save to SD code to compute CRC across multiple iterations to leave time for USB on large SaveRAM images.

This is mostly bug fixes so you should definitely get it if you have an earlier version.  I temporarily removed MSU support to make room for other things, but I will fit it back in for the next release without all of the USB support.  I had to change how saving to SD works so remember to back up your SD card before trying this.  It's a little slower now for large SaveRAM files (mostly rom hacks) for it to back the save state up to the SD card.

I decided to remove the ZeldaHUD because I wasn't updating it anymore.  The previous version should still work with the 0.65 firmware.  Once there's a better USB interface to sd2snes I'm hoping someone will come along and make one or modify one of the existing ones that are popular.

<Technical stuff below>

EDIT: Looks like there are some bugs with NORESP and the GET operations.  I'm working on a fix.

I was ignoring a lot of timing problems in the verilog which I think were causing weird hardware glitches for people.  The address decode logic in several units is in desperate need of some flops, but I need to understand it better to make sure I don't introduce logic bugs.  This release fixes that and the context state is still working as far as I can tell.

The save state fix is pretty simple and cleans up the first few frames after save/load so it doesn't show garbage.

The 512B minimum packet size was artificial because full speed USB 2.0 bulk has a maximum packet size of 64B.  I'm not ready to redesign the whole USB interface protocol so instead I added flags to: (a) enable a 64B data payload and (b) disable the need for responses on certain transactions.  This is coupled with 2 new 64B commands VGET/VPUT that support 8 vector lanes of 0-255 byte each.  The vector data is packed over USB.  All of the old commands (including responses) are still 512B.  So you can now use VGET/VPUT with responses disabled and 64B payload to do small fetches of somewhat sparse data with lower latency and lower overhead.  The vector operations are reasonably well tested, but may have some bugs.

I haven't started on the app that allow sharing of the USB, but it is very high on the list of things to do.

[redguy]

https://github.com/RedGuyyyy/sd2snes/blob/master/rel/usb2snes_v0.66.zip

usb2snes 0.66

Fixed several bugs with NORESP and vector operations.
Reverted CX4 and MSU enabled FPGA file to 0.1.7e.

I reverted the mmx2 and mmx3 special FPGA programming to use the 0.1.7e files.  That and when you load a MSU enabled file it will load the original 0.1.7e.  The USB stuff and MSU does not coexist nicely, though, as they both want to use interrupts and do FPGA accesses.  And the MSU has real time requirements which I don't want to work around right now.  So if you really want to use MSU stuff then please stick to the normal menu and don't use any of the USB stuff at the same time or you strange things will happen.

<Technical stuff below>

Now the NORESP and vector operations should work in all cases.  The data you need to send/get has to be a multiple of the data size (64 or 512 bytes).  Vector operations will have packed data with no gaps between each lane.