"Delete This Image"--it should be right at the top after clicking "Image Tools" then "Edit Image." Above the thumbnail that appears for editing.
I don't recommend attempting to use the "Category Admin" option on your gallery mainpage; I tried renaming my albums from there, and it seems to have permanently f**ed all my links up. It only seems to provide the option of creating a new 'sub-album' within your existing ones (as opposed to renaming existing albums, or even just creating a new album)--the 'sub-album' title
does replace the original title on the gallery mainpage, but the link to it doesn't work, and if you look at the location breakdown over the individual pictures, it doesn't seem like either the new 'sub-album' or the rename actually happened from in there. Not consistently, anyway.
In light of those link mixups plus the incorrect image coding on the 'red x' pictures I mentioned earlier (though I went ahead and deleted my 'red x' pic anyway), I'm inclined to suspect that everyone's 'red x' pics are still in there somewhere, it's just that the links to them are somehow messed up.
Also, there no longer seems to be any reasonably easy way to organize the pics in the order you want. You used to be able to number them and then have it automatically order them that way.