EverDrive Forum

General => FXPAK (SD2SNES) => Topic started by: redguy on June 17, 2017, 11:10 PM

Title: usb support
Post by: redguy on June 17, 2017, 11:10 PM
I took saturnu's usb firmware mods and made the following changes:
- Ported it to 0.1.7e.
- Rewrote most of the packet handling code to provide an interface to the file system and some common FW commands.

It's all a bit of hack right now while I figure out what I ultimately want to do with it.  The USB version supported by the sd2snes HW is 2.0 @ full speed (12 Mb/s theoretical data rate).  Reasonably fast for transferring single digit number of files, but not anywhere close to accessing the SD card directly from your computer.

Please back up your SD card if you decide to try it out.  The error/reset logic isn't the greatest so it's possible to get the usb state machine in the firmware and/or the app accessing it out of sync.  Getting the USB to work again usually involves reopening the app and/or rebooting the snes.

One odd thing I've seen is something related to sd2snes SRAM reads seem to indirectly mask or disable the interrupt that gets called when EP2/BulkIn is available.  I saw some comments from saturnu regarding this, too.  The current code has to poll (hack) in order to push the data out when SRAM is accessed.  When it's a transfer of a file from the SD card the interrupt seems to work fine.  Maybe it's caused by the FPGA accesses?

https://github.com/RedGuyyyy/sd2snes/releases/latest

This includes:
- The firmware with instructions on how to update the firmware and install a suitable windows driver.
- A c# app to access the file system of the SD card over USB with a variety of commands to modify, boot, etc.
- A hacked version of an alttp item randomizer display that reads SRAM over USB to update.

Source here:
https://github.com/RedGuyyyy/sd2snes
https://github.com/RedGuyyyy/usb2snesw
https://github.com/RedGuyyyy/ZeldaHud
Title: Re: usb support
Post by: redguy on June 24, 2017, 03:10 AM
I have real-time IPS patching working over USB which can use the NMI hook to call any new code.  It cheats (pun intended) by using the cheat/snescmd address range that is typically used for WRAM writes to call into arbitrary code.  There's still a race with modifying running code (I may add a spin loop + status to work around this), but it works for applying the patch over other parts of the sd2snes SRAM.

So I thought it would be interesting to use this to write a generic save/load state IPS patch with the help of $21XX and $42XX snooping for write only registers.  I tried mirroring a set of them from $2B00-$2B60.  Then using the new save state trigger in the game they are written to a the SRM file which I can use as a poor man's logic analyzer.  The initial values are read out correctly, but I can't figure out the timing for the ~SNES_PAWR, SNES_PA, and SNES_DATA signals.   I tried collecting a sample write for -5 and +59 cycles of data from the assertion of SNES_PAWR and the data doesn't make sense for the address I think it's writing (e.g. background vertical scroll in a particular part of the game vs bsnes).  I may have to try a different game as the one I have patched isn't making sense. 

Is PA write timing similar to the 24b address timing for writes?  It looks like it's 12 master cycles as opposed to 8 based on SNES_PAWR assertion.

EDIT: I should mention that after looking through the verilog and it looks like the bus is properly tristated from the sd2snes during writes.  The pin on the board also looks to go somewhere... where I can't say for certain, but the samples look sensible for the write assertion and associated address.

EDIT2: I figured it out.  I wasn't driving the OE signal properly.  There are still some bugs to work through, but now I'm seeing mostly correct mirrored values.
Title: Re: usb support
Post by: saturnu on June 27, 2017, 06:36 AM
oh nice!
i have to check this out. ^^
Title: Re: usb support
Post by: redguy on June 27, 2017, 08:22 PM
The usb port has the potential to enable some interesting debug features and runtime game changes.  Both pushed to the snes as well as a usb address space that is accessible by the game.

Im struggling a little with getting a generalized save state patch working. I can patch the game specific ips at runtime and it works great, but when i try to make it more general it fails. One problem is the snooping of 21xx registers sometimes get corrupted.  I think ill need to locate a logic analyzer to help understand the bus transactions.   The other is a problem with relocating the code to unused rom space results in some weird behavior i only see on console. Random controller input is detected by the game for a frame or two.
Title: Re: usb support
Post by: redguy on July 22, 2017, 09:38 PM
https://drive.google.com/file/d/0Bx9gzQZCkH8qampxbTk0UWdUU2M/view?usp=sharing

EDIT: If you downloaded the zip and Chrono Trigger or other 4MB games stopped working then try downloading again.  I fixed a bug.  Make sure to copy the new files in the sd2snes directory to your SD card to fix the bug.

usb2snes v0.3

- Fixed more timeouts due to collisions with other interrupts in FW.  MSU is still buggy.
- Added Write-Only register snooping to enable save-stating.
- Added basic IPS patch and state managing support.  Included save_state IPS patch.

See the readme file for more details on how to setup and use it.  The save state patch works reasonably well.  I haven't tested it with every game so changes are it will lock up or not work.  Despite this, I thought people might like to try it out.

<Technical stuff below>

The way the patch works is hijacking the cheat area, $2A90+, to jump to a region called the PATCH region $F0-$FF.  This region is mapper independent, supports read/write, and only accessible in the NMI hook.  The save state code does a one time init to setup some basic state and then uses SMC (self modifying code) to skip the init after the call.  It monitors the controller button state which is conveniently saved off to $2BF0.  When a save state is triggered it DMAs a lot of stuff to $F0-$F3 (inclusive) including the write-only PPU and CPU registers that are now being snooped by the SD2SNES firmware.  The load state transfers the data in reverse.  Several games (mostly Capcom) have fixes to make the audio port state coherent with the cached copy in WRAM.  I was running into a lot of problems with nested NMIs that were worked around by enabling and waiting for the next NMI at the end of save/load, consuming the associated stack frame, and finally exiting to the game's NMI handler.  This more closely lines up the NMI timing with what the game expects.  I'm sure some race conditions remain, but it works reasonably well.  A logic analyzer hooked up to the cart bus and UART->USB have been extremely helpful for debugging it.

I still consider this a collection of hacks as I'm not sure where to go with it, yet.  One could really go off the deep end and add OS-like functionality for dynamically loaded patches, RAM memory management, and other things.  But even now the patch support is a pretty powerful tool coupled with tool that can read the RAM contents over USB.  I'm considering hijacking the MSU-1 data buffer to provide a writeable address space that can be mapped to USB for communication among different SNESs.
Title: Re: usb support
Post by: iwasaperson on July 22, 2017, 10:22 PM
Can you update the tool for Linux by any chance?

I tried this one, but I can't get it to work: https://github.com/RedGuyyyy/usb2snes

Works fine in a Windows VM, but it's such a pain for me to go in and out of Windows all the time.
Title: Re: usb support
Post by: redguy on July 23, 2017, 03:52 AM
That version of usb2snes on github was made by saturnu a few years ago and the protocol is no longer supported.  My changes aren't stable enough to maintain multiple apps to access the sd2snes so it will be windows only for the time being.  I don't have a linux or macos desktop, but if someone wants to take a look at the c# code and suggest changes to get it to work on those platforms (assuming they have c# compilers available) I'd be happy to make them assuming they are simple.

I also fixed a bug that broke Chrono Trigger so please redownload the zip again and update the sd2snes directory on your flash card.
Title: Re: usb support
Post by: ikari_01 on July 23, 2017, 06:36 AM
I recall there were stalling and/or data corruption issues when transferring data from sd2snes to PC using the Windows build of usb2snes. Did you encounter anything like it?
Title: Re: usb support
Post by: redguy on July 23, 2017, 07:34 AM
Yes.  I had problems with transfers to and from sd2snes with the original code and the windows utility.  It was random and infrequent when I could successfully upload a file to RAM and boot it.  I was getting both data corruption and stalls plus the downloads were slow when they worked.  If I remember correctly, the code was using an interrupt to deliver packets to a buffer which were then serviced by a handler in the menu loop for both command and data.  In the download code path there was some comments about problems with interrupts and it polled the CDC directly to send data every time it entered the loop handler.

The new code retains the loop handler in the menu and snes loop for operating on various command types (get, put, boot, ls, etc) which sets up state for receiving data or performing the operation.  For put operations the data is then directly written into RAM or a file when a full block is received by the interrupt handler rather than waiting for the loop handler to poll.  Uploads are a lot faster now, although, they are still slower than downloads.  The stalls and data corruption with downloads seemed related to other code in the loop.  SD writes for saveram and potentially other things were causing the interrupt to stop asserting even if data was still available.  I never figured out the exact cause but the solution was to put the code in a spin loop on get and ls operations while the interrupt pulled all the data and finally updated the state to exit the spin loop in the loop handler.  Maybe there are DMA operations in the other code that share that change interrupt state?  Downloads are much faster and I haven't encountered data corruption since making the changes.  Timeouts are pretty rare.  The only one I've seen recently is when I hooked up 2 SNES/SD2SNES with USB and switched between them with the app.

The original USB code was a great reference as it setup the PLL and various other things that would have otherwise taken a long time to figure out.
Title: Re: usb support
Post by: OsteHovel on July 27, 2017, 08:04 AM
I was pludering around with your code yesterday and i was able to communicate with no problem with my SNES, but i have a question, can i read/write ram directly from the console or can i only read/write to rom+sram using it?

OsteHovel
Title: Re: usb support
Post by: redguy on July 27, 2017, 04:56 PM
Without the usb changes, the SD2SNES has an address map that includes an IS_WRITABLE flag based on a 3-bit mapper field and some other state.  It basically provides the same address space mapping and attributes (read-only, etc) as the typical mapping modes usually referred to as lorom, hirom, exthirom, etc.  One exception is a region referred to as SNESCMD ($00:2A00-$00:2BFF) which is used to store the NMI hook that enables WRAM cheats and in-game buttons.  This region is protected from access until a NMI (or IRQ, dependent on configuration state) is taken in which case the address map dynamically changes to allow read (execute)/write to it.

With the latest USB changes, the address map is also dynamically modified inside the hook to make $F0:0000-$FF:FFFF available as whats called the PATCH region with flat addressing.  This the 15-16MB memory range on the 16MB SD2SNES RAM that the menu also uses.  So while you are in the NMI hook you can read and write this.  Right now it's used by the save_state patch to hold the code, variables, and save state data.  The address map is changed back to the original mapper when you exit the hook as it would break some games to leave it.

It's possible to extend the address map changes to make any address writable by the SNES so that you can write the full SD2SNES RAM which contains the ROM image (0-? MB), SRAM (14-15? MB), MENU/PATCH (15-16 MB), plus some other regions for MENU and BS-X.  But whether you only want this to be true in the NMI hook or all the time, which will break some games, is not clear.  It would be cool to have a programmable address map where the SNES code chooses the read/write flags and mapping functions in the hook and decide whether they persist outside of it.  But I'm not really sure what this interface should look like, yet, so I kept the address map hacks to a minimum for now.  Allowing all addresses to be writable inside the hook by default is pretty easy to do so I might just add it in while I think of a better solution.

The short answer is not currently, but I'll try adding it in so all addresses are modifiable inside the hook.
Title: Re: usb support
Post by: Enmet on August 10, 2017, 11:33 PM
I have been trying to use this. I have installed the driver and all that correctly according to the readme, but the usb2snes program gives me this on startup:

Quote
************** Exception Text **************
System.Exception: Can not read Bus provided device description device 4d36e978-e325-11ce-bfc1-08002be10318
   at usb2snes.utils.Win32DeviceMgmt.GetDeviceBusDescription(IntPtr hDeviceInfoSet, SP_DEVINFO_DATA deviceInfoData)
   at usb2snes.utils.Win32DeviceMgmt.GetAllCOMPorts()
   at WindowsFormsApplication1.usb2snes.buttonRefresh_Click(Object sender, EventArgs e)
   at System.Windows.Forms.Control.OnClick(EventArgs e)
   at System.Windows.Forms.Button.OnClick(EventArgs e)
   at System.Windows.Forms.Button.PerformClick()
   at WindowsFormsApplication1.usb2snes.usb2snes_Load(Object sender, EventArgs e)
   at System.Windows.Forms.Form.OnLoad(EventArgs e)
   at System.Windows.Forms.Form.OnCreateControl()
   at System.Windows.Forms.Control.CreateControl(Boolean fIgnoreVisible)
   at System.Windows.Forms.Control.CreateControl()
   at System.Windows.Forms.Control.WmShowWindow(Message& m)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
   at System.Windows.Forms.Form.WmShowWindow(Message& m)
   at System.Windows.Forms.Form.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Probably something with .NET but I tried to reinstall it up to 4.5.2 with no results. Note that the usb2snes program still runs if I press continue but there will be no ports listed. I do know it's not the USB port or the cable as I've been using that with an Arduino before. Firmware version shows up correctly in the SD2SNES menu.
Title: Re: usb support
Post by: redguy on August 10, 2017, 11:42 PM
Someone else had the same problem and I have a workaround.  It's interesting that they also had an Arduino.  Let me clean up a few of the things I'm working on and I'll post up another version in a few hours.
Title: Re: usb support
Post by: redguy on August 11, 2017, 02:32 AM
https://github.com/RedGuyyyy/sd2snes/blob/master/rel/usb2snes_v0.4.zip
(click download button in upper right)

usb2snes v0.4

- Fixed more timeouts due to disconnect problems.  There are still more left to fix.  E.g. pulling the cable out typically requires a reboot of the snes and app.
- Added preliminary USB/NET memory region support.  Work in progress.
- Fixed (workaround) problem where querying certain COM devices results in an exception.  The exception is now discarded.

See the readme file for more details on how to setup and use it.  This is mostly bug fixes as the majority of the changes aren't accessible from the app.  Don't click the 'Test' button as it is for testing purposes only.  :)

<Technical stuff below>

This should fix the COM device exception on startup.  I'm not sure why it's happening, but if I catch, discard, and ignore that port it looks to work.  I can't reproduce it, but someone else said this fixed their problem.

The major thing I've been working on is USB/network play support.  It's a bit of a hack right now (much like everything else I've done).  What it provides is a (currently unused) register region at $2010-$2017 and a scratch region which a USB enabled app can access at $1E-$1F:5000-5FFF.  In a test IPS patch I've set up $1000 of that region for scratch and some state variables.  In the other $1000 region it holds an input and output queue.  The patch which currently runs on a NMI inits the scratch and queue state and can generate queue operations (write, add, compare and swap, etc) to regions in scratch memory.  Those queue operations are read by an application polling that memory space over USB which then broadcasts the operation to all networked clients.  The client app pushes these writes down to the other queue which the NMI reads and operates on scratch memory.  This way one game running on hardware can affect the state of another game.

Here's an example where one link gives another link (still in bed) the moon pearl in the lttp randomizer:

https://www.youtube.com/watch?v=q70PdvOG70A

The other thing I tried was allowing save states to be initiated on game start.  It works with one major flaw: I'm cheating and not providing saved APU state.  This results in no sound and eventually it locks up.  I'd like to add a generic HW context engine which saves all APU, WRAM, VRAM, etc writes to a fixed location in sd2snes RAM, but that project needs to wait.  This would provide much simpler (and efficient) save state code while also allowing for snooping of various snes RAM regions for debugging.

I would also like to have a unified driver that multiplexes multiple apps onto the USB connection so you can interleave use of usb2snes, HUDs, input monitors, networked games, etc.

Needless to say this is all going to take a lot of work and long time.  There are some really cool things that can be done with the USB connection.

EDIT: I also made it so the ROM addresses in RAM are writeable while you're in the PATCH region.  It's not very well tested so it may break some games.  This lets you rewrite the game ROM on the fly (code or data... doesn't matter).
Title: Re: usb support
Post by: Enmet on August 11, 2017, 03:01 AM
The new version works for me, great job and thank you!
Title: Re: usb support
Post by: redguy on August 20, 2017, 09:49 PM
I changed my mind and decided to implement the context engine to save snes *RAM contents to sd2snes's RAM.  It wasn't as bad as previously thought and there's now a working prototype.  It saves off WRAM, VRAM, APURAM, OAM, CGRAM, and many PPU and CPU registers.  I made a basic c# memory viewer, but realized the viewers that came with bsnes were far better and ended up hacking it to poll the USB contents without a loaded cartridge.  It's pretty cool to see the backgrounds load up in the tilemap viewer and the contents of VRAM in (somewhat) real-time.  Here's an example:

https://www.youtube.com/watch?v=iC-HPhJf_Og

I'd like to make a little DMA engine to copy contents of the RAM around to speed up saving in savestates.  It would also be cool to have a way to set HW breakpoints, but that's going to take a lot more thought.

Another thing I've been trying to do is load a game and immediately load a savestate.  The basics of this have been working for a while, but without having a way to restore the APU it would have no sound and usually freeze after a few seconds.  The context engine stuff makes a basic attempt at saving all the state pre-execution that is written to the APU.  Now on game init this APU state can be restored and the sound works, although, it needs some more work to fix the music.  Here's an example where it transitions from contra to castlevania.  I have to manually trigger the load of castlevania and the associated state right now which is not ideal.  The music works only because I picked a savestate right before the engine loads the music.

https://www.youtube.com/watch?v=ZEzl5HVppvU

At this point there are way too many hacked up c# and c++ utilities which are using my messy USB interface.  A driver that provides a simpler (socket based?) interface to the sd2snes over USB is the next step.  That way I don't have to fix the USB related bugs in 6 different places and it would also allow multiple applications to use the USB port at the same time.
Title: Re: usb support
Post by: iwasaperson on August 20, 2017, 10:32 PM
I changed my mind and decided to implement the context engine to save snes *RAM contents to sd2snes's RAM.  It wasn't as bad as previously thought and there's now a working prototype.  It saves off WRAM, VRAM, APURAM, OAM, CGRAM, and many PPU and CPU registers.  I made a basic c# memory viewer, but realized the viewers that came with bsnes were far better and ended up hacking it to poll the USB contents without a loaded cartridge.  It's pretty cool to see the backgrounds load up in the tilemap viewer and the contents of VRAM in (somewhat) real-time.  Here's an example:

https://www.youtube.com/watch?v=iC-HPhJf_Og

I'd like to make a little DMA engine to copy contents of the RAM around to speed up saving in savestates.  It would also be cool to have a way to set HW breakpoints, but that's going to take a lot more thought.

Another thing I've been trying to do is load a game and immediately load a savestate.  The basics of this have been working for a while, but without having a way to restore the APU it would have no sound and usually freeze after a few seconds.  The context engine stuff makes a basic attempt at saving all the state pre-execution that is written to the APU.  Now on game init this APU state can be restored and the sound works, although, it needs some more work to fix the music.  Here's an example where it transitions from contra to castlevania.  I have to manually trigger the load of castlevania and the associated state right now which is not ideal.  The music works only because I picked a savestate right before the engine loads the music.

https://www.youtube.com/watch?v=ZEzl5HVppvU
That looks sweet! Didn't even think something like that would even be possible. Would be insanely helpful to homebrewers out there who want to test on real hardware.

Quote
At this point there are way too many hacked up c# and c++ utilities which are using my messy USB interface.  A driver that provides a simpler (socket based?) interface to the sd2snes over USB is the next step.  That way I don't have to fix the USB related bugs in 6 different places and it would also allow multiple applications to use the USB port at the same time.
I mean... if you're redoing the USB interface anyway, would it be reasonable to request Linux support?
Title: Re: usb support
Post by: saturnu on August 21, 2017, 07:58 PM
i wrote a linux program for just loading and starting games with the new usb protocol.
but actually this is pretty boring, the savestate dumping and restore feature is something that could be fun to play with.
i could imagine to program something like cheatengine to find offsets to alter health or items in games. ^^

a special driver is no big deal under linux but for windows the driver-signing is expensive and the dev mode for unsinged drivers is inconvenient.
that was the main reason i started with the std usb serial port, because all you need is an inf file. :>
i did some experiments with libusb but meh...

maybe just a little library between a program and the signed microsoft serial driver would do the trick, too.
Title: Re: usb support
Post by: Enmet on August 22, 2017, 04:18 AM
Quote
I mean... if you're redoing the USB interface anyway, would it be reasonable to request Linux support?

+1 for Linux support.

Also minor note, while I got usb2snes to run, the Zelda item tracker did not, so I guess the work-around wasn't added for it.

Would it be possible to do a soft reset through this interface?
Title: Re: usb support
Post by: redguy on August 22, 2017, 07:49 AM
https://github.com/RedGuyyyy/sd2snes/blob/master/rel/usb2snes_v0.5.zip

Here's a new version, but it's mostly for people who want to check out the (simple, non-bsnes-plus) RAM viewer and associated contents.  Save states are going to be slower for the time being because I'm also copying APURAM which is inefficient.  I need to add the hardware DMA engine.  I also worked around some races that I found when compiling everything under windows10.

By driver I was really just thinking of a system tray app in user space and not an actual driver.  I shouldn't have called it that.  The USB stuff has too many races and stalls that I need to work through to make all the apps deal with that code.  It would be better to have one app that deals with those problems and interleaves commands onto the sd2snes from other apps.

I'm not sure why the ZeldaHUD isn't working for you.  I didn't update it to use the latest USB code, but it still works for me.  Eventually I'm going to remove it as it was just for testing.  Yes, it's possible to add a soft reset button.  For now you can just double click/boot the file again to reload it.

EDIT: Actually, I think I forgot to include the fixed HUD in the last release.  I'll do that next time around.  I fixed it a while ago and just didn't include it.
Title: Re: usb support
Post by: redguy on August 27, 2017, 10:46 PM
https://github.com/RedGuyyyy/sd2snes/blob/master/rel/usb2snes_v0.6.zip

usb2snes v0.6

- Fixed several timeouts.
- Fixed some graphics corruption/lockups in save state.
- Added hardware RAM DMA engine with spin loop support.
- Minor cleanup in the CTX engine.
- Updated ZeldaHUD so it hopefully doesn't crash on startup for some people.

The save states should be fast again.  The DMA engine is used to copy the current state to a location that stores the save state.

<Technical Stuff>

The DMA engine currently supports a 24b source address, destination address and length.  It has 3 opcodes: copy, reset, and set.  It works similar to MVN (memmove) and will handle overlap assuming it is bug free.  You can put the snes into a BRA $FE spin loop as part of its configuration.  It feeds the instructions directly in order to leave RAM bus cycles for the DMA operation.
Title: Re: usb support
Post by: saturnu on August 27, 2017, 10:59 PM
oh cool,
currently i'm writing a c pendant to the c# app, just for fun. ^^
it can already dump and write the savestates, list files..., but it's still lacking the ips patch feature. :D
maybe i can finish it tonight...
Title: Re: usb support
Post by: redguy on August 29, 2017, 06:50 AM
Thanks for working on it.  Now I have to be careful not to change/break the protocol.  I have to figure out a way to add support for a smaller packet size in case someone wants to poll on a small set of data.  Or maybe a scatter-gather style operation since there are probably a lot of random bytes all over memory that certain tools would want.

USB full-speed bandwidth is a little low for some of things I'd like to try.  For example, it would be nice to have a data stream operation where the app can open up a connection and the firmware keeps streaming data from some hardware queue.  The RAM bandwidth may also be a limiting factor.  I have to check if it supports a burst mode.
Title: Re: usb support
Post by: saturnu on August 29, 2017, 08:45 AM
oh, please change the protocol if you want, i don't think i'm going to release it anyway, until it's development is stalled a bit more. :D
i just wanted to play around a little bit with the savestate feature.
yesterday i successfully dumped, compared, edited and reuploaded a savestate under linux, with the usb_v0.6 firmware. ^^

the thing is, searching for cheats could be easily done with bsnes, too.
but that's something, that's often the case.

maybe features that are useful for developers/translators, who need to test on real hardware are the way to go.
- savestates are great for translators, who need to test certain dialogs, this is a big step anyway - even if the music is stopping.
Title: Re: usb support
Post by: redguy on September 13, 2017, 06:39 AM
https://www.youtube.com/watch?v=nhNm-3XOrNc
(rgb+ossc output on the left, snes9x/usb output on the right)

I'm trying to implement a digital out from the sd2snes to an emulator (uses the software PPU) over USB.  It still needs a lot of bug fixes, a HDMA model, and some USB protocol optimizations.  The missing HDMA code is the cause of most of the background, item display, and special effect bugs.  It's pretty cool to have all the snes9x scaling and filter options available.

It probably won't work in all games and may not even work at all in the end, but it's a fun project.
Title: Re: usb support
Post by: TrueJournals on September 16, 2017, 02:25 AM
I love the work that's been going on here.  The added USB support is incredible and seems like a really awesome utility to add all kinds of supplementary functions to SNES games.

I've been wanting another little side project, so I've also started playing around with extra utilities on top of the USB support.  I've created a little Python HTTP server that provides an API to interact with RAM and do IPS patching (to allow your save state patch to be uploaded :) ).  I've also been working on a LttP item tracker (because why not) that can read everything in real-time thanks to the WRAM read support in your latest version.  As saturnu said, feel free to change the interface however you want -- I don't mind adapting my code!

As I was working on this, I came to a point where I wanted to try to write WRAM, not just read it.  Based on your usb2snesviewer memory viewer utility code (https://github.com/RedGuyyyy/usb2snesw/blob/master/usb2snesviewer/usb2snesviewer.cs), I got the impression that this should be possible -- but I just can't make it work for the life of me.  I tried building and uploading the usb_test and usb_write patches (https://github.com/RedGuyyyy/sd2snes/tree/master/patch)... but that didn't seem to do anything.

Am I missing something, or are writes to WRAM not possible?
Title: Re: usb support
Post by: redguy on September 16, 2017, 03:49 AM
Glad to hear you are finding it useful.

It is possible to write data into WRAM, VRAM, and the other on-snes data stores, but it currently requires patching the NMI hook.  There are two patches:
- usb_test : This is a work in progress that is meant to be a shared "USB"-specific memory address space that will be kept mostly coherent through broadcast operations.  It has an inbound and outbound queue where you can send and receive operations (compare and swap, xor, increment, etc) that operate on a small region of memory.  The operation and data is currently defined to be 4B.
- usb_write : This is a more general purpose patch that re-uses the inbound queue code to support full address space byte writes packed into the same 4B queue element.

I haven't mentioned them before since they are just hacks and aren't well tested.  You'll want the usb_write patch.  Hopefully it still works.  I just checked in an assembled version of it:

https://github.com/RedGuyyyy/sd2snes/blob/master/patch/usb_write/usb_write_v0.ips

Eventually I should merge these patches with the existing sd2snes hook code.  For things like USB address space support, save states, and anything that the hardware can detect it would be nice to have a conditional NMI hook that only runs when the trigger is hit.  But that's going to take some time to figure out and implement.
Title: Re: usb support
Post by: TrueJournals on September 16, 2017, 03:57 AM
Well... I don't know what I was doing wrong before, but I've got it working now!  Thanks for the help!
Title: Re: usb support
Post by: redguy on September 24, 2017, 06:28 AM
https://github.com/RedGuyyyy/sd2snes/blob/master/rel/usb2snes_v0.65.zip

usb2snes 0.65

Fixed save state to not enable PPU until after PPU alignment.  Fixes some momentary graphics glitches on load.
Fixed many timing and questionable SNES bus loading statements in the FPGA.
Added support for 64B data block sizes and no responses based on flag.
Added support for 64B vector VGET/VPUT commands.
Removed MSU support from base firmware.
Removed ZeldaHUD.
Changed save to SD code to compute CRC across multiple iterations to leave time for USB on large SaveRAM images.

This is mostly bug fixes so you should definitely get it if you have an earlier version.  I temporarily removed MSU support to make room for other things, but I will fit it back in for the next release without all of the USB support.  I had to change how saving to SD works so remember to back up your SD card before trying this.  It's a little slower now for large SaveRAM files (mostly rom hacks) for it to back the save state up to the SD card.

I decided to remove the ZeldaHUD because I wasn't updating it anymore.  The previous version should still work with the 0.65 firmware.  Once there's a better USB interface to sd2snes I'm hoping someone will come along and make one or modify one of the existing ones that are popular.

<Technical stuff below>

EDIT: Looks like there are some bugs with NORESP and the GET operations.  I'm working on a fix.

I was ignoring a lot of timing problems in the verilog which I think were causing weird hardware glitches for people.  The address decode logic in several units is in desperate need of some flops, but I need to understand it better to make sure I don't introduce logic bugs.  This release fixes that and the context state is still working as far as I can tell.

The save state fix is pretty simple and cleans up the first few frames after save/load so it doesn't show garbage.

The 512B minimum packet size was artificial because full speed USB 2.0 bulk has a maximum packet size of 64B.  I'm not ready to redesign the whole USB interface protocol so instead I added flags to: (a) enable a 64B data payload and (b) disable the need for responses on certain transactions.  This is coupled with 2 new 64B commands VGET/VPUT that support 8 vector lanes of 0-255 byte each.  The vector data is packed over USB.  All of the old commands (including responses) are still 512B.  So you can now use VGET/VPUT with responses disabled and 64B payload to do small fetches of somewhat sparse data with lower latency and lower overhead.  The vector operations are reasonably well tested, but may have some bugs.

I haven't started on the app that allow sharing of the USB, but it is very high on the list of things to do.
Title: Re: usb support
Post by: redguy on September 30, 2017, 09:14 PM
https://github.com/RedGuyyyy/sd2snes/blob/master/rel/usb2snes_v0.66.zip

usb2snes 0.66

Fixed several bugs with NORESP and vector operations.
Reverted CX4 and MSU enabled FPGA file to 0.1.7e.

I reverted the mmx2 and mmx3 special FPGA programming to use the 0.1.7e files.  That and when you load a MSU enabled file it will load the original 0.1.7e.  The USB stuff and MSU does not coexist nicely, though, as they both want to use interrupts and do FPGA accesses.  And the MSU has real time requirements which I don't want to work around right now.  So if you really want to use MSU stuff then please stick to the normal menu and don't use any of the USB stuff at the same time or you strange things will happen.

<Technical stuff below>

Now the NORESP and vector operations should work in all cases.  The data you need to send/get has to be a multiple of the data size (64 or 512 bytes).  Vector operations will have packed data with no gaps between each lane.
Title: Re: usb support
Post by: redguy on October 22, 2017, 01:39 AM
Haven't posted in a while, but I've been working on a few things including the tool tray app, a bus trace capture engine, and various bug fixes.  Still working out bugs before I'm ready to post it up.

<Technical stuff below>

I have observed a random data corruption bug when reading the RAM for a while now.  It started happening a little more frequently when vector reads (VGET) were added.  I noticed that the second and later vector lane would vary rarely (4+ hours of doing random writes + reads with data checks) return shifted/rotated data or a corrupted first byte.  I believe the problem is how the MCU performs an address increment, sends the next read operation, and waits for the response from the RAM scheduler.  After a block of MCU operations are done it will still advance the address pointer and try to perform a read of the address directly after the block.  If that read is delayed and another MCU operation (address set + read) arrives it's possible the stalled read will show up and look like the first read, but with the wrong data.  As a workaround I added a FPGA_WAIT_RDY() call to the fpga_spi.c:set_mcu_addr function and it seems to have solved the problem.  I need to run longer tests to verify.

void set_mcu_addr(uint32_t address) {
  FPGA_SELECT();
->  // wait for prior operations to clear out
->  FPGA_WAIT_RDY();
  FPGA_TX_BYTE(FPGA_CMD_SETADDR | FPGA_TGT_MEM);

The USB firmware and FPGA changes make this more likely due to the context saving engine which has higher priority than the MCU, but it may still be possible to hit in unmodified 0.1.7e.

It's also interesting to note that the following print in memory.c:sram_reliable() hasn't printed since I made the change, either.  It used to show up at random times exhibiting the same first byte corruption/shifting I saw on USB reads.
printf("i=%d val=%08lX\n", i, val)
Title: Re: usb support
Post by: redguy on November 05, 2017, 12:55 AM
EDIT: I fixed a bug with a hardcoded COM port in the InputViewer and also made minor changes to the apps/ directory structure and parsing code.

https://github.com/RedGuyyyy/sd2snes/releases/download/v3/usb2snes_v3.1.zip

usb2snes v3.1

Added websocket/tray app.
Added input viewer based on NintendoSpy.
Fixed lots of bugs and added lots more.

The main update is the tray app.  You can launch the file, input, and memory viewer from there.  They can all work at the same time including multiple instances, but things start to get slow if you are reading a lot of data with something like the memory viewer.  The input viewer will work with only a few games.  If it doesn't work and you know the memory address where it stores the controller buttons you can type that (in hex) into the address field and try that.  The memory viewer can help to find the location in memory for the buttons if you know what to look for and the game writes them out.

<Technical stuff below>
The tray app listens on localhost:8080 (likely IPv6 depending on OS) and supports the websocket protocol with JSON commands and binary data.  It's a pretty simple protocol and everything available across USB has a simplified command over the websocket interface.  The websocket API I used has some issues including not exposing partial messages.

There's also more general configuration register accesses (on the FPGA) with masking support and a trace buffer which uses that config.  The trace buffer needs some testing and a tool to interface with it.  It lets you set a start and stop trigger event on the bus and captures the events in that window.  It re-uses the MSU buffer which is now expanded to 30 KB.

To free up space, the BSX, DSP, and MSU code was commented out from the default FPGA file.  The firmware loads up the appropriate default 0.1.7e when it detects these files.  So those games will lose several of the features of the USB changes.
Title: Re: usb support
Post by: redguy on November 18, 2017, 10:35 PM
https://github.com/RedGuyyyy/sd2snes/releases/download/v4/usb2snes_v4.1.zip

usb2snes v4.1

Added persistant patch support that can be applied anywhere including the menu.
Fixed save_state patch code so it doesn't conflict with the menu.
Added memory trigger for save and load into save_state patch.
Added (experimental) firmware update feature + power cycle.

The save state patch now lasts as long as you keep it powered.  You can change games and go back to the menu and it won't be disabled.  The patch still requires NMI hooks to be enabled to work.

Future updates will support updating at start of the usb2snes application and support uploading the various firmware files.  Unfortunately, that doesn't work until after you update to this version because of a bug in v3.x.
Title: Re: usb support
Post by: redguy on December 03, 2017, 07:58 PM
https://github.com/RedGuyyyy/sd2snes/releases/download/v5/usb2snes_v5.1.zip

usb2snes v5.1

Added hookless NMI patching support.  The MemoryViewer uses this to now allow updates to regions like WRAM and VRAM.
Added auto tracking Zelda HUD.
Adding websocket message logging support.
Fixed (workaround) another USB timeout I can't figure out.
Title: Re: usb support
Post by: Rockslam on December 17, 2017, 09:53 PM
https://github.com/RedGuyyyy/sd2snes/releases/download/v5/usb2snes_v5.1.zip

usb2snes v5.1

Added hookless NMI patching support.  The MemoryViewer uses this to now allow updates to regions like WRAM and VRAM.
Added auto tracking Zelda HUD.
Adding websocket message logging support.
Fixed (workaround) another USB timeout I can't figure out.

Congratulations for your work. i am testing now in my sd2snes.
Title: Re: usb support
Post by: V0lrat on March 04, 2018, 03:25 AM
First of all thank you for this!

I have been using the Save State functionality and it is quite impressive. I have been facing compatibility issues with some games and was wondering if there's anything that can be done to make them work or if it's something that's currently being looked at.

The main game I am looking to make it work with is Claymates (USA).
Title: Re: usb support
Post by: redguy on March 04, 2018, 05:31 AM
Unfortunately, Claymates doesn't appear to use NMIs (Non Maskable Interrupt) to generate a frame and the save state relies on that.

The good news is I was able to make a really simple change to the US version (attached) that enables an empty NMI.  It seems to work, but I only tested it for a few minutes.  Make sure to create a backup of the original.  Use something like lunar IPS to patch it.

You also need the attached save_state patch.  The patch will only work if you apply it while the game is started.  A bit of a hack, but that's to avoid it running more code.
Title: Re: usb support
Post by: V0lrat on March 05, 2018, 01:55 AM
Unfortunately, Claymates doesn't appear to use NMIs (Non Maskable Interrupt) to generate a frame and the save state relies on that.

The good news is I was able to make a really simple change to the US version (attached) that enables an empty NMI.  It seems to work, but I only tested it for a few minutes.  Make sure to create a backup of the original.  Use something like lunar IPS to patch it.

You also need the attached save_state patch.  The patch will only work if you apply it while the game is started.  A bit of a hack, but that's to avoid it running more code.

Your sir are a  L E G E N D! Thank you so much for that.
Title: Re: usb support
Post by: redguy on March 06, 2018, 05:03 AM
I should mention that the latest version (currently v7) can be found here: https://github.com/RedGuyyyy/sd2snes/releases/latest

I haven't had time to work on it lately so it will stay at v7 for a while.
Title: Re: usb support
Post by: ErivandoXP on April 03, 2018, 06:35 AM
Awesome features!

I used savestate on the SD2SNES for the First time!

Easy to use. Thanks a lot and good Lucky.
Title: Re: usb support
Post by: FartPuff on April 03, 2018, 05:33 PM
besides save-states, super nerdy developer stuff like memory view, and HUDs, what else could this kind of gear-chain provide?  Netplay perhaps?

edit:  I just did notice Redguy mention he was tinkering with netplay.  wow.  if USB/netplay becomes a thing it's definitely interesting to me.
Title: Re: usb support
Post by: Monkey on April 06, 2018, 01:16 PM
This is really intresting stuff.  Does this mean that theoretically with 2 SNESes and 2 SD2SNES carts something like this could be hacked to work on real hardware?

https://www.romhacking.net/forum/index.php?topic=7912.0 (https://www.romhacking.net/forum/index.php?topic=7912.0)

A 4 player SMK would be really cool.  Or just 2 player battle mode without seeing the others screen.

Or a really bad way to play doom deathmatch!
Title: Re: usb support
Post by: ErivandoXP on April 06, 2018, 03:23 PM
The usb2snes savestate function is possible to integrate with the official firmware without a computer?
Title: Re: usb support
Post by: Enmet on April 06, 2018, 10:39 PM
This is really intresting stuff.  Does this mean that theoretically with 2 SNESes and 2 SD2SNES carts something like this could be hacked to work on real hardware?

Maybe, the closest thing we have so far is Multitroid (http://multitroid.com/), a co-op client for Super Metroid that syncs game events and items between players.

In the case of Multitroid, you will have two different instances of Super Metroid running on two different consoles (SD2SNES and Snes9x supported) but ammo, health, items, map, events (bosses down, special doors opened, game triggers) are all shared between players. Here's a video of 100% from the release event. (https://www.twitch.tv/videos/183873886?t=02h02m58s)

Perhaps there would be too much latency for a vanilla 2-player mode.
Title: Re: usb support
Post by: Monkey on April 08, 2018, 04:55 PM
So in theory you could have a raspberry pi connected to the usb port running software to make a modern day BSX feature?  There could be a ROM hacked version of games like F-Zero, Super Mario Kart or SMW etc. that coud run on the SD2SNES that you could select and download custom levels from online.  Upload scores and lap times/ host data. The pi could handle you account/ username and ghost is patches for the latest games to be hacked and features.
Title: Re: usb support
Post by: Atse on April 12, 2018, 02:09 PM
Really intresting!

Could the USB port read the controller input?
Title: Re: usb support
Post by: Enmet on April 12, 2018, 07:13 PM
Really intresting!

Could the USB port read the controller input?
Yes, RedGuy made a port of NintendoSpy, but it doesn't seem to work with all games. It's in clients > InputViewer.

I've used it myself for Super Metroid.
Title: Re: usb support
Post by: Atse on April 13, 2018, 01:39 PM
Thanks will check that out.
How cool is that? :)

Cool with streaming.
Title: Re: usb support
Post by: iwasaperson on May 19, 2018, 12:04 AM
Added GSU support to the USB firmware (with caveats): https://www66.zippyshare.com/v/nZv76DNW/file.html

USB functionality is extremely limited during GSU gameplay (and probably Cx4 as well).

This is USB2SNES v7 and GSU v10 combined, nothing else.

Savestate2SNES will NOT function unless it is updated to support this firmware version number (0x1337BEEF). Savestates are very buggy even when they do work and will likely always be on real hardware.

As of now, this does NOT include ikari_01's 1CHIP brightness fix. If you require that, please use my unofficial GSU firmware v10+ found in that thread instead.
Title: Re: usb support
Post by: Arnold101 on May 21, 2018, 02:14 AM
Added GSU support to the USB firmware (with caveats): https://www66.zippyshare.com/v/nZv76DNW/file.html

USB functionality is extremely limited during GSU gameplay (and probably Cx4 as well).

This is USB2SNES v7 and GSU v10 combined, nothing else.

Savestate2SNES will NOT function unless it is updated to support this firmware version number (0x1337BEEF). Savestates are very buggy even when they do work and will likely always be on real hardware.

As of now, this does NOT include ikari_01's 1CHIP brightness fix. If you require that, please use my unofficial GSU firmware v10+ found in that thread instead.
Savestate are not in usb2snes already? why not work?
Title: Re: usb support
Post by: Skarsnik on May 22, 2018, 11:20 AM
Added GSU support to the USB firmware (with caveats): https://www66.zippyshare.com/v/nZv76DNW/file.html

USB functionality is extremely limited during GSU gameplay (and probably Cx4 as well).

This is USB2SNES v7 and GSU v10 combined, nothing else.

Savestate2SNES will NOT function unless it is updated to support this firmware version number (0x1337BEEF). Savestates are very buggy even when they do work and will likely always be on real hardware.

As of now, this does NOT include ikari_01's 1CHIP brightness fix. If you require that, please use my unofficial GSU firmware v10+ found in that thread instead.

Actually I already added a fix for a Redguy usb2snes version with gsu. Basicly if the firmware version string end with gsu it assumes it's v7. Maybe you could do the same?
Title: Re: usb support
Post by: iwasaperson on May 22, 2018, 06:28 PM
Actually I already added a fix for a Redguy usb2snes version with gsu. Basicly if the firmware version string end with gsu it assumes it's v7. Maybe you could do the same?

https://www48.zippyshare.com/v/TrL84SqI/file.html

Does this work? (Copy over the other download).

Changed FW version string to "0.1.7e-v7-usb-v10-gsu"
Title: Re: usb support
Post by: iwasaperson on July 07, 2018, 09:13 PM
Link died. Here is a more permanent one: https://mega.nz/#!HrJGQIaI!j7Sj9arTjfBHZGfiTQZ-ENq0p72TUpSlbgLk9wTbs6M
Title: Re: usb support
Post by: Spriten on August 08, 2018, 10:01 AM
Hey there, is it possible to make this work with the latest firmware?