Some comments:
Code:
150 .version EQUS " 0.01"
ROM version string must not start with any padding before it's content as you have done, it should be "a.bc" or "a.bc (de Fgh ijkl)", ie "0.01" or "0.01 (01 Jan 2012)". Also, you've got an extra zero byte after the version string. The zero byte at the start of the copyright string is the terminator of the version string. However, you /can/ have additional strings after the version string and before the copyright string, but the copyright string's zero byte terminates the previous string.
Code:
180 EQUS "(C) pstnotpd 2012"
If there is no date in the version string, the copyright string should be
"(C)year ....." not
"(C) ... year" ie
"(C)2012 ...." It is better to put a date in the version string, so:
Code:
130 .title :EQUS "HEXAROM Hex Sprites":EQUB 0
150 .version :EQUS "0.01 (15 Nov 2012)"
170 .copyright :EQUB 0:EQUS "(C) pstnotpd":EQUB 0
The service handler does not need to save/restore X as X holds the ROM number and is immediately restored by the MOS on exit. Also, you must claim claimed calls by returning A set to zero, you must not restore it. Also, as OSBYTE and OSWORD pass their parameters in &EF/&F0/&F1 you don't even need to save/restore Y either.
Code:
330 .hexosb
340 LDA &EF:CMP#&64:BEQosb64:LDA #7:RTS
350 :
351 .osb64 :LDA&F0:LDX&F1:LDY#0:JSRhexer:LDA #0:RTS
As you've realised, OSBYTE less than 128 can only pass X. If you want to pass more than one byte you need to use OSWORD as all high OSBYTEs are allocated. The only spare OSBYTE is 163, but you pass an ID in X, and a parameter in Y, so that still only allows you to pass one byte of data.
On return from OSWORD you should set something in the returned control block so the calling code knows something has happened.
Code:
380 .hexer ASL A:STA &84
You MUST NOT corrupt external memory. You are trampling over language workspace here. &A8-&AF are reserved for this purpose, but you should preserve and restore them as you may be being called by some other code that is also using them.
Code:
.Service
CMP #7:BEQ ServOSByte
CMP #8:BEQ ServOSWord
CMP #9:BEQ ServHelp
RTS
:
.ServOSWord
LDA &EF:CMP #oswNumHexer:BEQ OswHexer
LDA #8:RTS
:
.OswHexer
LDX #7:.SaveZP
LDA &A8,X:PHA:DEX:BPL SaveZP :\ Save zero page workspace
LDY #1:LDA (&F0),Y:TAX
DEY:LDA (&F0),Y :\ Get parameters from control block
JSR hexer :\ Perform action
LDX #0:.RestZP
PLA:STA &A8,X:INX:CPX #8:BNE RestZP :\ Restore zero page workspace
LDY #2:LDA #0:STA (&F0),Y :\ Return 'ok'
RTS :\ Return A=0, call claimed
The example code above passes the 16-bit parameter in XY+0 and XY+1 and returns a result in XY+2. This saves and restores zero page in the OSWORD handler, but if you have more service routines that use zero page workspace it's more efficient to do the save/restore in the service handler itself.
As for the actual OSWORD number, use
http://beebwiki.jonripley.com/OSWORDs as a reference. OSWORDS under 128 all use a 16-byte control block, OSWORDs 128 and above use a varying length control block. It looks like for your needs a short OSWORD will be sufficient. I'd recommend &48 (72) onwards. In fact I'd recommend only a single OSWORD number, if you need additional subfunctions you should do it all through the same OSWORD, you have plenty of space in the control block to indicate the subfunction. The convention is:
XY+0 command
XY+1 result
XY+2... parameters